Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog. — Kunal

Building resilient ML Engineering Skills

Talk
Stas Bekman
- books on debugging & ML engineering
Religiously take notes
- problems will repeat themselves
- process of discovery & fixing
people
- who you know, not what you know for resiliency
- having access to key people with core components
machine learning engineering
performance, incl. marketing vs reality
- real numbers are very different
- have your own tools to measure
ROI
- speed at which training will finish
tflops
- make sure to check against scarcity
- theoretical numbers are unacheivable
- shape needs to be appropriate
- fn of the clock
- fma = fused multiply add
- main component for compute
- can't boost the clock
- measure it for your particular setup
buying gpus is very scary
- renting is much safer for compute
have to manage all systems: need fast network and disk
gpu buffer, time to replace SLA, etc.
- will end up in trouble otherwise
- lack of engineering
log everything while running
- make everything reproducible
- always bisect
- compare environments
- be very rigorous, like a lab
MFU
- hardware utilization
- efficiency keeps going down over time
have to move data to hbm memory
benchmarking/duplex