Task Queue

What it is

Many programs have work that should not happen right away, or not on the same machine: send an email, resize a photo, build a report. Instead of doing it on the spot, the program puts a small job on a task queue. Separate workers take jobs off the queue, do them, and say when each one is finished. Amazon SQS, RabbitMQ and many job systems work this way.

The queue itself lives on a server. Your code talks to it over a network connection. In this challenge you build the client side: adding jobs, and the worker loop.

The core idea

A worker does three things for every job, always in this order:

1. Take the next job from the server.
2. Do the work.
3. Tell the server the job is done, so it can forget it.

The server does not trust a worker blindly. When it hands out a job, it keeps it aside for a short time. If no "done" message arrives in that time, it puts the job back so a worker can take it again. People call this time the visibility timeout. It is what makes a crash safe: a worker that dies in the middle of a job never says "done", so the job comes back.

That only works if the worker says "done" after the work, never before. If it says "done" first and then crashes, the server has already forgotten the job, and nobody will ever do it.

A job might also run twice: a worker can finish the work, then lose its connection before the "done" message arrives, and the job comes back. Queues like this promise every job is done at least once, not exactly once. Real job code is written so that doing it twice is harmless.

Example

One worker, three jobs, and the server waits 1 second for a "done" message:

take   -> job 7
work on job 7, finished
done 7 -> ok                 (job 7 is gone for good)
take   -> job 8
work on job 8, crash         (no "done" message)
...1 second later, the server puts job 8 back
take   -> job 8              (a worker tries again)
take   -> nothing left       (wait a moment, ask again)

When the connection breaks

Networks fail. A connection can be cut at any moment, even right after it opens. A worker that treats a broken connection as the end of its life stops for good, and the jobs it never took just sit there. A good worker closes the broken connection, opens a new one, and keeps asking for jobs. It does not need to remember what it was doing: any job it took but did not finish comes back on its own.

When the server stops answering

A connection can also stay open and go quiet. A network split or a stuck server sends nothing, and nothing fails either. At work this looks like a healthy worker that uses no CPU while the queue keeps growing, until someone asks why a job never ran. So give every read of an answer a time limit, about a second. When it runs out, treat the connection as broken: close it and open a new one. A job the server handed out on the quiet connection comes back on its own, because it was never marked done.

Where you meet this

• Amazon SQS: a received message stays in the queue but is hidden (the visibility timeout). If it is not deleted before the timeout ends, it becomes visible again for another worker. (SQS visibility timeout)
• RabbitMQ: with manual acknowledgements, "any delivery (message) that was not acked is automatically requeued" when the channel or connection closes. (RabbitMQ acknowledgements)
• Celery: task_acks_late means the task message is acknowledged after the task has run, not just before. Even then, by default, a task whose worker process dies suddenly is still acknowledged. The separate task_reject_on_worker_lost setting puts it back on the queue, so the work must be safe to run twice. (Celery configuration)

When to choose it: independent jobs that can run later or on another machine, and that must not be lost if a worker crashes.

When not to: the caller needs the answer right now, or many readers need the same messages in order. Because a task can come back after a slow or lost acknowledgement, the same task can also run twice. Make the work safe to repeat, for example with an idempotency key.

Common bugs

• Saying "done" before the work, so a crash in the middle loses the job.
• Never saying "done", so every job keeps coming back and nothing finishes.
• Giving up for good when one connection breaks.
• Waiting for an answer forever, with no time limit on the read.
• Sharing one connection between several workers, so their lines get mixed up.
• Asking for jobs in a tight loop with no pause when the queue is empty.
• Counting "done" when the message is sent, instead of when the server answers that it got it.

Reading a result

The tester reads the server's own record: every job added, handed out, and marked done. "Marked done" should equal the number of jobs added, and no job should be with two workers at the same time. Your own counts should match the server's numbers exactly.

Keep it fast

Adding a job is one short line out and one short line back. Build the line in one step, write it with one call, and read the answer without creating extra buffers or strings for every job.

Learn more

The Amazon SQS guide on visibility timeouts and the RabbitMQ guide on consumer acknowledgements explain how real queues hand out work and take it back when a worker disappears.

Go to the Task Queue challenge