Kinesis Consumer

golang, kinesis, consumer

AWS has created an excellent Golang SDK for interacting with Kinesis streams. However, the library is lacking checkpointing functionality which allows consumers to restart and continue processing data where they left off. That’s why I created a Kinesis Consumer library for Golang.

The library is intended to be a lightweight wrapper around the Kinesis API to read records, save checkpoints (with swappable backends), and gracefully recover from service timeouts/errors.

The heart of the consumer library is it’s checkpointing functionality. When a consumer has finished processing a page of records from the stream, it will store the last sequence number into a database to save its progress.