A million records a minute
A lakehouse on Apache Iceberg with Kafka and Flink stream processing, handling 1M+ records per minute in near real-time — plus a no-code builder so clients could wire their own ETL pipelines.
- Kafka
- Flink
- Iceberg
- ETL
Machine Learning Engineer · Johannesburg, South Africa · UTC+2
Sholto Armstrong
I build ML that survives contact with production.
Machine Learning Engineer with 7+ years building and deploying production ML, RAG/LLM and real-time data systems across banking, IoT and retail. Led a 10-person AI team, built streaming data platforms processing 1M+ records per minute, and shipped production RAG systems on Kubernetes. MEng in Computer Engineering (cum laude).
- Python
- PyTorch
- FastAPI
- Kafka
- Flink
- Docker
- Kubernetes
- AWS/GCP
- 7M+transactions / day scored by rules I shipped
- 1M+records / minute through streaming platforms
- 10engineers led as AI product manager
- 3xcum laude degrees in computer engineering
Selected work
01 / things I shippedDecider
An open-source rules engine I built at Capitec, now productionised across business credit, retail credit and fraud detection — executing on 7M+ transactions a day. Presented at PyCon.
- Python
- Rules engine
- Open source
- Fraud
capitec/dsp-decision-engine ↗
RAG that cites its sources
A production chatbot platform on LLMs, embeddings and vector search. Moved it off a lone EC2 box onto Kubernetes, rebuilt ingestion serverless, and made every answer traceable to a downloadable source.
- RAG
- LLMs
- Kubernetes
- Serverless
Eyes on the physical world
Computer vision in places it has to actually work: planogram compliance at 97% accuracy with detection cut from 15 minutes to 5 seconds, and real-time person detection on bank branch CCTV.
- Computer vision
- PyTorch
- Edge
- Retail
Eight roles, seven years, one long thread of shipping models into production —the full timeline is here.
Toolbox
02 / what I reach forMachine Learning
- PyTorch
- TensorFlow
- scikit-learn
- Pandas
- Computer vision
- Anomaly detection
- Time-series forecasting
- Model deployment
- MLOps
LLMs & GenAI
- RAG pipelines
- LLMs
- Embeddings
- Vector databases
- Hallucination reduction
Data Engineering
- Apache Flink
- Kafka
- Spark
- Beam
- Apache Iceberg
- Real-time streaming
- ETL pipelines
- Lakehouse architecture
Languages
- Python
- SQL
- TypeScript
- Java
- C++
- Rust
- Kotlin
Backend
- FastAPI
- Flask
- REST APIs
- OAuth2
Databases & Storage
- PostgreSQL
- Redis
- MongoDB
Infrastructure & DevOps
- Docker
- Kubernetes
- Helm
- AWS
- GCP
- CI/CD
- Serverless
Frontend
- React
- Angular
- Astro
Tools & Practices
- Git
- GitHub
- GitLab
- Azure DevOps
- Agile