--- title: AI or ain't: LLMs | daily.dev description: A hands-on deep dive into building a GPT-2 model from scratch in Go, under 300 lines of code. Covers tokenization, word embeddings (WTE/WPE), layer... canonical: https://daily.dev/posts/ai-or-ain-t-llms-jff0d4qzb twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: AI or ain't: LLMs | daily.dev og:description: A hands-on deep dive into building a GPT-2 model from scratch in Go, under 300 lines of code. Covers tokenization, word embeddings (WTE/WPE), layer... og:url: https://daily.dev/posts/ai-or-ain-t-llms-jff0d4qzb og:image: https://api.daily.dev/og/posts/JFF0d4Qzb.png og:image:alt: AI or ain't: LLMs og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # AI or ain't: LLMs **[zserge's blog](https://daily.dev/sources/zserge)** · 16 min read · 0 upvotes · 0 comments ## Summary A hands-on deep dive into building a GPT-2 model from scratch in Go, under 300 lines of code. Covers tokenization, word embeddings (WTE/WPE), layer normalization, GeLU activation, softmax, and the mechanics of self-attention with query/key/value vectors. Walks through the full forward-propagation pipeline, from loading raw model weight files to running multi-head attention across layers, and finally decoding output vectors back into text. Ends with sample completions from the 124M-parameter model, showing both surprisingly coherent and hallucinated outputs, and a reflection on whether such models count as 'intelligent'. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does self-attention work inside a GPT-2 model? Self-attention uses a query vector representing the current word, key vectors representing other words in context, and value vectors added to the current word's data when a key is relevant. Multiplying the query by each key gives relevance scores, and multiplying values by those scores and summing produces the attention output. GPT-2's smallest model runs this across 12 heads simultaneously, concatenating results before projecting them into the next layer. _daily.dev surfaces deep dives like this for developers building intuition around transformer internals._ ### Why does GPT-2 use word vectors instead of raw token indices for training? Token indices are arbitrary numbers that don't reflect meaning, so a network trained directly on them can't learn that 'Paris' and 'capital' are related despite having token IDs 40313 and 3139 apart. GPT-2's word token embedding (WTE) matrix converts each token into a vector where semantically similar words end up numerically closer, similar to the older Word2Vec approach, enabling arithmetic-like relationships between word meanings. _following practical breakdowns of embeddings helps when you're choosing how to represent data for your own models._ ### What activation function does GPT-2 use instead of ReLU and why? GPT-2 uses the GeLU (Gaussian Error Linear Unit) activation function rather than ReLU. It has a similar overall shape to ReLU but a smoother, more complex curve, computed as 0.5*x*(1+tanh(0.7978845676080871*(x+0.044715*x^3))), and is applied within the dense feed-forward sub-layers after the self-attention projection step. _developers implementing transformer layers by hand can track these implementation details on daily.dev._ ## Similar posts on daily.dev - [GitHub - angelos-p/llm-from-scratch](https://daily.dev/posts/github---angelos-p-llm-from-scratch-5sw4itutx) · Hacker News · 1 upvotes · 0 comments - [LLM From Scratch is a hands-on workshop where you write every piece of an AI from nothing](https://daily.dev/posts/llm-from-scratch-is-a-hands-on-workshop-where-you-write-every-piece-of-an-ai-from-nothing-ntyfh6hef) · XDA Developers · 0 upvotes · 0 comments - [OpenAI’s new LLM exposes the secrets of how AI really works](https://daily.dev/posts/openai-s-new-llm-exposes-the-secrets-of-how-ai-really-works-iepupajzg) · MIT Technology Review · 0 upvotes · 0 comments - [From Autocomplete to Intelligence: How Language Became the Operating System of AI](https://daily.dev/posts/from-autocomplete-to-intelligence-how-language-became-the-operating-system-of-ai-ivetrqcjn) · Wix Engineering · 1 upvotes · 0 comments --- Tags: [#golang](https://daily.dev/tags/golang), [#nlp](https://daily.dev/tags/nlp), [#neural-networks](https://daily.dev/tags/neural-networks), [#transformers](https://daily.dev/tags/transformers) [View this post on daily.dev](https://daily.dev/posts/ai-or-ain-t-llms-jff0d4qzb) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"AI or ain't: LLMs","url":"https://daily.dev/posts/ai-or-ain-t-llms-jff0d4qzb","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/ai-or-ain-t-llms-jff0d4qzb"},"datePublished":"2026-08-23T10:20:15.864Z","dateModified":"2026-08-23T10:21:23.130Z","description":"A hands-on deep dive into building a GPT-2 model from scratch in Go, under 300 lines of code. Covers tokenization, word embeddings (WTE/WPE), layer...","image":"https://media.daily.dev/image/upload/s--foaA6JGU--/f_auto/v1722860399/public/Placeholder%2004","thumbnailUrl":"https://media.daily.dev/image/upload/s--foaA6JGU--/f_auto/v1722860399/public/Placeholder%2004","isAccessibleForFree":true,"articleSection":"zserge's blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"zserge's blog","logo":"https://media.daily.dev/image/upload/s--KQnFBOIT--/f_auto,q_auto/v1787480407/logos/zserge","url":"https://daily.dev/sources/zserge"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/ai-or-ain-t-llms-jff0d4qzb","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"golang,nlp,neural-networks,transformers","timeRequired":"PT16M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"zserge's blog","item":"https://daily.dev/sources/zserge"},{"@type":"ListItem","position":3,"name":"AI or ain't: LLMs"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/ai-or-ain-t-llms-jff0d4qzb#faq","mainEntity":[{"@type":"Question","name":"How does self-attention work inside a GPT-2 model?","acceptedAnswer":{"@type":"Answer","text":"Self-attention uses a query vector representing the current word, key vectors representing other words in context, and value vectors added to the current word's data when a key is relevant. Multiplying the query by each key gives relevance scores, and multiplying values by those scores and summing produces the attention output. GPT-2's smallest model runs this across 12 heads simultaneously, concatenating results before projecting them into the next layer. daily.dev surfaces deep dives like this for developers building intuition around transformer internals."}},{"@type":"Question","name":"Why does GPT-2 use word vectors instead of raw token indices for training?","acceptedAnswer":{"@type":"Answer","text":"Token indices are arbitrary numbers that don't reflect meaning, so a network trained directly on them can't learn that 'Paris' and 'capital' are related despite having token IDs 40313 and 3139 apart. GPT-2's word token embedding (WTE) matrix converts each token into a vector where semantically similar words end up numerically closer, similar to the older Word2Vec approach, enabling arithmetic-like relationships between word meanings. following practical breakdowns of embeddings helps when you're choosing how to represent data for your own models."}},{"@type":"Question","name":"What activation function does GPT-2 use instead of ReLU and why?","acceptedAnswer":{"@type":"Answer","text":"GPT-2 uses the GeLU (Gaussian Error Linear Unit) activation function rather than ReLU. It has a similar overall shape to ReLU but a smoother, more complex curve, computed as 0.5*x*(1+tanh(0.7978845676080871*(x+0.044715*x^3))), and is applied within the dense feed-forward sub-layers after the self-attention projection step. developers implementing transformer layers by hand can track these implementation details on daily.dev."}}]} ```