Part 2 of a series on building a product classification API, focusing on data preparation for product titles. Covers measuring data purity to identify mislabeled products, and step-by-step text cleaning: ASCII encoding of non-ASCII characters, lowercasing, custom regex tokenization (with configurable excluded punctuation), stop word removal, numeric token removal, short-word filtering, and deduplication. Each step is illustrated with Python code using list comprehensions and regex. Cleaned titles are used to identify and exclude impure (conflicting category) products from model training.
Table of contents
Measuring data purityPreparing the title (and short descriptions)Encoding titles as asciiLowercasing titlesTokenizing titlesRemoving words that are solely numericRemoving words with too few charactersRemoving duplicated wordsRemoving empty titlesExcluding titles that are impureConclusion3 Impressions