Zach Tellman - Everything Will Flow

Yeah this is true, typically you want to keep blocking functions outside this. By default, core.async allocates as many threads as cores in your CPU. In addition to this, I personally find core.async to deal poorly with flow control (e.g. slow subscribers slowing down the entire flow for every subscriber), and seems to not have a lot of telemetry / it’s difficult to find out what’s going on in the thread pool behind it. I’ve personally settled on Zach Tellman’s Manifold library, which provides a much more sane abstraction and is fully compatible with core.async. For some addition thought and material on these topics, I highly recommend watching this talk by Zach, “Everything Will Flow” [this video]

I like this essay but I think it is very strange that he doesn't link to the talk that Zach Tellman gave in which Tellman talks about the origins of Factual's Durable Queues: [this video] Factual's Durable Queues are a fairly minimal queue effort that is useful in cases where you need some minimal way of handling back pressure, because a service downstream has failed. In his talk, Tellman talks about the fact that he needed to write a lot of data to AWS S3, and sometimes S3 stops accepting writes, so he needed an easy re-try mechanism. I used Durable Queues everywhere in my code and find it very handy. It's useful when you need a very minimal queue that still has the backing of a file system.

Queue theory suggests the more nodes you add to handle your queue then the longer your queue can survive (facing rising traffic) but the faster it will collapse when it finally runs out of capacity. Zach Tellman went over this in one of his talks and he had a nice graphic to demonstrate what this looks like: [this video] I suppose any complex system has dynamics like that, more capacity to absorb load, but then a faster collapse when capacity is surpassed. Hemmingway's Laws Of Motion applies: ”How did you go bankrupt?” Bill asked. ”Two ways,” Mike said. ”Gradually and then suddenly.” Put a bit differently by the economist Rudiger Dornbusch: ”The crisis takes a much longer time coming than you think, and then it happens much faster than you would have thought.” lifted from: [link]