Message Log Reader

What it is

Some systems keep messages in a log: a long list where each new message goes at the end, and nothing already written ever changes. Kafka is the best known example. Many programs can read the same log, each at its own pace.

Each message has a position. The first one is 0, the next is 1, and so on. A reader works through the log one message at a time. It also tells the server how far it got, so that after a crash or a restart it can carry on from that place instead of starting again. People call this saved position the offset.

The core idea

A reader does three things for every message, always in this order:

1. Get the message at its current position.
2. Do the work for that message.
3. Save its place on the server: "next time, start at the position after this one".

The order of steps 2 and 3 is the whole lesson. If the reader saves its place first and then crashes during the work, the saved place already points past that message. Nobody will ever do it. If it does the work first and crashes before saving, the message comes round again after the restart. Doing a message twice is usually harmless. Losing one silently is not.

The saved place should only move forward. Saving a smaller number than before throws away progress.

Example

A log with 6 messages. The reader saves its place after each one.

saved place 0
get 0, work, save 1
get 1, work, save 2
get 2, work, crash before saving    (saved place is still 2)
...restart
ask for the saved place -> 2
get 2, work, save 3                 (message 2 done again, nothing lost)
get 3 ... and so on up to 5, save 6
get 6 -> nothing there, all read

When the connection breaks

Networks fail. The server may end a session, or a connection may be cut the moment it opens. A reader that gives up stops for good, and the rest of the log is never read. A good reader opens a new connection, asks the server for its saved place, and carries on from exactly there.

Exactly there matters both ways. Starting from 0 repeats everything. Starting from a number kept only in memory can skip messages that were never saved, or repeat ones that were.

When the log is taken away

When many readers share the work, the server sometimes moves a log from one reader to another. It tells the current reader that the log is no longer its own. The right move is to save its place, so the next reader knows where to start, and then stop. A reader that keeps going does work that now belongs to someone else.

Where you meet this

• Apache Kafka: the committed position is the last offset stored safely. If the reader fails and restarts, it starts again from there. (KafkaConsumer)
• Kafka's design notes name the two orders. Save the position and then do the work: a crash can skip messages ("at most once"). Do the work and then save: a crash can repeat messages ("at least once"). (Delivery semantics)
• Amazon Kinesis readers that use the Kinesis Client Library keep a checkpoint per shard in a DynamoDB table, and a lease so only one worker reads a shard at a time. (KCL lease table)

When to choose it: many readers need the same messages, each at its own pace, or you want to read old messages again later.

When not to: each message is one job for one worker, and you never need to read it again. A queue that takes back unfinished jobs is simpler. See Task Queue.

Common bugs

• Saving your place before the work is done.
• Giving up after one broken connection.
• Starting again from 0 after a reconnect, or from a number kept in memory instead of the saved one.
• Starting one position too early after a reconnect, so a saved message is done twice.
• Carrying on after the log was taken away.
• Counting a save when you send it, not when the server answers that it got it.

Reading a result

The tester reads the server's own record: every message it sent and every place you saved. Messages should be read with no gaps, the saved place should only move forward and end at the end of the log, and your own counts should match the server's numbers.

Keep it fast

Each message is one short line out, one line back, then one more pair to save your place. Build each line in one step, write it with one call, and read answers with one buffered reader per connection, without creating extra strings for every message.

Learn more

The Apache Kafka documentation on consumer groups shows how real readers save their place and hand logs over to each other. The book Designing Data-Intensive Applications explains why "at least once" is the usual promise for this kind of reading.

Go to the Message Log Reader challenge