pg_tiktoken is a new Postgres extension that wraps OpenAI's tiktoken tokenizer to bring fast BPE (Byte Pair Encoding) tokenization directly into PostgreSQL. It provides two main functions: tiktoken_encode for tokenizing text and tiktoken_count for counting tokens. The extension supports multiple OpenAI model encodings (cl100k_base, p50k_base, etc.) and is particularly useful for managing ChatGPT conversation history within token limits. A practical example shows how to store message history in Postgres and query messages that fit within the 4096-token limit of gpt-3.5-turbo using a formula: MAX_HISTORY_TOKENS = MODEL_MAX_TOKENS – NUM_SYSTEM_TOKENS – NUM_COMPLETION_TOKENS.
