kafka-exactly-once

Usage examples for the Kafka createDirectStream / createRDD api that I contributed to Spark (available since Spark 1.3)

Master corresponds to Spark 2.0 / Kafka 0.10

If you're looking for earlier versions, see the Spark 1.6 branch

For more detail, see the presentation or the blog post or the slides or the jira ticket

If you want to try running this,

schema.sql contains postgres schemas for the tables used

src/main/resources/application.conf contains jdbc and kafka config info

The examples are indifferent to the exact kafka topic or message format used, although IdempotentExample assumes each message body is unique.