---
title: "Pre-Training BERT with Hugging Face Transformers and Habana Gaudi"
url: https://daily.dev/posts/pre-training-bert-with-hugging-face-transformers-and-habana-gaudi-v5qyytfiy
source_url: https://www.philschmid.de/pre-training-bert-habana
type: article
source: "Philipp Schmid"
published: 2026-05-31T07:40:58.286Z
updated: 2026-05-31T08:23:19.769Z
tags: ["bert"]
reading_time: 9
upvotes: 0
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Pre-Training BERT with Hugging Face Transformers and Habana Gaudi

**[Philipp Schmid](https://daily.dev/sources/philschmid)** · 9 min read · 0 upvotes · 0 comments

## Summary

A step-by-step tutorial on pre-training BERT-base from scratch using Hugging Face Transformers, Optimum Habana, and AWS DL1 (Habana Gaudi) instances. Covers dataset preparation using Wikipedia and BookCorpus, training a custom BertTokenizerFast tokenizer, preprocessing and tokenizing the dataset, and running distributed masked-language modeling pre-training across 8 HPU cores via GaudiTrainer. Includes a cost comparison showing Habana Gaudi delivers ~25% cost savings (~$1,650 for full pre-training) versus NVIDIA V100-based GPU setups (~$2,075), and notes that a full 1M-step pre-training would take roughly 125 hours.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://www.philschmid.de/pre-training-bert-habana>

---

Tags: [#bert](https://daily.dev/tags/bert)

[View this post on daily.dev](https://daily.dev/posts/pre-training-bert-with-hugging-face-transformers-and-habana-gaudi-v5qyytfiy)