--- title: Categorizing Data with Large Language Models in Rust description: LibreQoS is an open-source project for monitoring and ensuring quality-of-experience for ISPs by tracking individual data flows. To make the data... canonical: https://daily.dev/posts/categorizing-data-with-large-language-models-in-rust-emyci8jlw twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Categorizing Data with Large Language Models in Rust | daily.dev og:description: LibreQoS is an open-source project for monitoring and ensuring quality-of-experience for ISPs by tracking individual data flows. To make the data... og:url: https://daily.dev/posts/categorizing-data-with-large-language-models-in-rust-emyci8jlw og:image: https://api.daily.dev/og/posts/emyCi8jlw.png og:image:alt: Categorizing Data with Large Language Models in Rust og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Categorizing Data with Large Language Models in Rust **[Ardan Labs](https://daily.dev/sources/ardlbs)** · 18 min read · 20 upvotes · 1 comments ## Summary LibreQoS is an open-source project for monitoring and ensuring quality-of-experience for ISPs by tracking individual data flows. To make the data understandable, ASN (Autonomous System Numbers) needs categorization, which is automated using Rust and large language models (LLMs). The post explains how to obtain ASN data, load and deduplicate it, set up a local LLM, and categorize data using context scraped from associated websites. The process involves using crates like Serde, CSV, Itertools, Reqwest, and Scraper for efficient data handling, and leveraging Tokio for parallel processing to speed up the categorization task. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community discussion Top comments from developers on daily.dev. **@joshuamo** · 0 upvotes > i'm a simple man > > i see rust, i upvote --- Tags: [#crawling](https://daily.dev/tags/crawling), [#machine-learning](https://daily.dev/tags/machine-learning), [#nlp](https://daily.dev/tags/nlp), [#rust](https://daily.dev/tags/rust) [View this post on daily.dev](https://daily.dev/posts/categorizing-data-with-large-language-models-in-rust-emyci8jlw) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Categorizing Data with Large Language Models in Rust","url":"https://daily.dev/posts/categorizing-data-with-large-language-models-in-rust-emyci8jlw","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/categorizing-data-with-large-language-models-in-rust-emyci8jlw"},"datePublished":"2024-08-06T23:30:54.593Z","dateModified":"2024-11-16T02:20:10.907Z","description":"LibreQoS is an open-source project for monitoring and ensuring quality-of-experience for ISPs by tracking individual data flows. To make the data...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fdb6de7e06056ccd5a2f042ceeec3ad1?_a=AQAEuiZ","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fdb6de7e06056ccd5a2f042ceeec3ad1?_a=AQAEuiZ","isAccessibleForFree":true,"articleSection":"Ardan Labs","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Ardan Labs","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/22ce9de869f24137bce6d1694dbe5f6e","url":"https://daily.dev/sources/ardlbs"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/categorizing-data-with-large-language-models-in-rust-emyci8jlw","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":20},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"crawling,machine-learning,nlp,rust","timeRequired":"PT18M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Ardan Labs","item":"https://daily.dev/sources/ardlbs"},{"@type":"ListItem","position":3,"name":"Categorizing Data with Large Language Models in Rust"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/categorizing-data-with-large-language-models-in-rust-emyci8jlw","comment":[{"@type":"Comment","text":"i’m a simple man\ni see rust, i upvote","datePublished":"2024-08-24T08:40:38.903Z","url":"https://daily.dev/posts/emyCi8jlw#c-Iznpl6rFX","author":{"@type":"Person","name":"Joshua Mo","url":"https://daily.dev/joshuamo","image":"https://avatars.githubusercontent.com/u/102877324?v=4"}}]} ```