I have a multiple frontends to my service written in Node.js and workers written in Ruby. Now the question is how to make those communicate? I need to maintain dynamic pool of workers to handle load (spawn more workers when load rises) and messages are quite big ~2-3M because I'm sending images to workers uploaded by users through Node.js frontends. Because I want nice scaling I thought about some queuing solution, but I didn't find any existing solutions (or misunderstood guides) that will provide:
- Fallback mechanisms. Solutions I've found so far have single failure point - message broker and there are no ways to provide fallbacks.
- Serialization. So when broker fails tasks are not lost.
- Ability to pass big messages.
- Easy API for Ruby and Node.js
- Some API to track queue size so I could rearrange workers pool.
- Preferrably lightweight.
Maybe my approach is wrong? Maybe I shouldn't use queues but some other way? Or there's some queueing solution that fits requirements above?
