Think twice. In serial, not parallel

This is more a note to self than any sort of helpful article.

I've always been somewhat impressed by concurrency. Maybe because (traditional) computers are deterministic, I've always thought there was something ... special(?) ... about asking it to do something and not really knowing when it will complete. Which I realise sounds completely stupid.

The first version of Collabor8Online was slow. It was written by a whole load of different people, in the traditional Rails style, loading up loads of @ variables (and hence database queries) and spitting them out, almost randomly, in various views and partials. So when I joined the company I didn't really know which bits I could safely remove to speed things up.

When I started work on V2, given that Rails had ActiveJob and Hotwire (two tools not available to the writers of Collabor8Online V1), I thought "I'm going to move as much of the heavy lifting into background tasks as possible".

Not async/await style concurrency where you've got a limited number of things and they're not really running at the same time.

But multiple threads on multiple processes across multiple CPUs on multiple boxes.

This was a mistake.

It's so easy to write code that works fine 99.9% of the time. But when it doesn't it's really hard to trace through and figure out where it's going wrong.

It's so easy to flood your queues with unnecessary jobs - each may complete in fractions of a second but if you've just put thousands of them onto the queue you're going to hold everyone else up.

And putting the hard work onto the queue can mean you forget about optimising stuff - so you don't notice those dodgy, heavyweight, queries till your database tables are locking up, your server is stuck at 100% CPU and you're getting calls from SREs complaining at the alerts they're receiving at all hours.

So next time I think to myself "async all the things" I really must give myself a slap.