AWS has created an excellent Golang SDK for interacting with Kinesis streams. However, the library is lacking checkpointing functionality which allows consumers to restart and continue processing data where they left off. That's why I created a Kinesis Consumer library for Golang.
The library is intended to be a lightweight wrapper around the Kinesis API to read records, save checkpoints (with swappable backends), and gracefully recover from service timeouts/errors.
The heart of the consumer library is it's checkpointing functionality. When a consumer has finished processing a page of records from the stream, it will store the last sequence number into a database to save its progress.
Hi, I'm Harlow. I live in San Francisco and head up the enginnering team at Clearbit. I've been writing software professionally for the better part of a decade for companies like Bills.com, thoughtbot, and HotelTonight. I like fast APIs, streaming event data, and of course my lovely @tayloresque.