---
title: "Announcing pg_tiktoken: A Postgres Extension for Fast BPE Tokenization"
url: https://daily.dev/posts/announcing-pg-tiktoken-a-postgres-extension-for-fast-bpe-tokenization-x7sxccucu
source_url: https://neon.com/blog/announcing-pg-tiktoken-a-postgres-extension-for-fast-bpe-tokenization
type: article
source: "Neon"
published: 2026-07-09T06:57:50.028Z
updated: 2026-07-09T07:03:50.742Z
tags: ["nlp", "openai", "postgresql"]
reading_time: 4
upvotes: 0
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Announcing pg_tiktoken: A Postgres Extension for Fast BPE Tokenization

**[Neon](https://daily.dev/sources/neontech)** · 4 min read · 0 upvotes · 0 comments

## Summary

pg_tiktoken is a new Postgres extension that wraps OpenAI's tiktoken tokenizer to bring fast BPE (Byte Pair Encoding) tokenization directly into PostgreSQL. It provides two main functions: tiktoken_encode for tokenizing text and tiktoken_count for counting tokens. The extension supports multiple OpenAI model encodings (cl100k_base, p50k_base, etc.) and is particularly useful for managing ChatGPT conversation history within token limits. A practical example shows how to store message history in Postgres and query messages that fit within the 4096-token limit of gpt-3.5-turbo using a formula: MAX_HISTORY_TOKENS = MODEL_MAX_TOKENS – NUM_SYSTEM_TOKENS – NUM_COMPLETION_TOKENS.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://neon.com/blog/announcing-pg-tiktoken-a-postgres-extension-for-fast-bpe-tokenization>

## Similar posts on daily.dev

- [GitHub - marcelroed/gigatoken: Language model tokenization at GB/s](https://daily.dev/posts/github---marcelroed-gigatoken-language-model-tokenization-at-gb-s-eobew1umo) · Hacker News · 23 upvotes · 3 comments
- [Watch Out Elasticsearch\! Tiger Data's PostgreSQL BM25 Search Extension Goes Open Source](https://daily.dev/posts/watch-out-elasticsearch-tiger-data-s-postgresql-bm25-search-extension-goes-open-source-i9m2ohtmp) · It's Foss · 32 upvotes · 0 comments
- [Better Relevance for AI Apps With BM25 Algorithm in PostgreSQL](https://daily.dev/posts/better-relevance-for-ai-apps-with-bm25-algorithm-in-postgresql-wrgphmk6m) · The New Stack · 1 upvotes · 0 comments
- [GitHub - timescale/pg\_textsearch: PostgreSQL extension for BM25 relevance-ranked full-text search. Postgres OSS licensed.](https://daily.dev/posts/github---timescale-pg-textsearch-postgresql-extension-for-bm25-relevance-ranked-full-text-search-p-xeiapecnq) · Hacker News · 11 upvotes · 0 comments
- [Let’s Build the GPT Tokenizer: A Complete Guide to Tokenization in LLMs – fast.ai](https://daily.dev/posts/let-s-build-the-gpt-tokenizer-a-complete-guide-to-tokenization-in-llms-fast-ai-enm7elmas) · fast.ai · 1 upvotes · 0 comments

---

Tags: [#nlp](https://daily.dev/tags/nlp), [#openai](https://daily.dev/tags/openai), [#postgresql](https://daily.dev/tags/postgresql)

[View this post on daily.dev](https://daily.dev/posts/announcing-pg-tiktoken-a-postgres-extension-for-fast-bpe-tokenization-x7sxccucu)