End-to-end tutorial for optimizing and deploying a BERT model (FinBERT) on AWS Inferentia2 using the optimum-neuron library. Covers converting BERT to Neuron format with static shapes, writing a custom inference.py script for text classification, packaging and uploading the model to S3, deploying a real-time endpoint on Amazon SageMaker, and benchmarking performance. Results show ~4ms average latency for BERT-base with sequence length 128, enabling ~500 inferences/second, at a cost of roughly $0.55 per million requests on an inf2.xlarge instance.
Table of contents
Quick intro: AWS Inferentia 21. Convert BERT to AWS Neuron (Inferentia2) with optimum-neuron2. Create a custom inference.py script for text-classification3. Upload the neuron model and inference script to Amazon S34. Deploy a Real-time Inference Endpoint on Amazon SageMaker5. Run and evaluate Inference performance of BERT on Inferentia2Conclusion3 Impressions