Offset Management For Apache Kafka With Apache Spark Streaming

An ingest pattern that we commonly see being adopted at Cloudera customers is Apache Spark Streaming applications which read data from Kafka. Streaming data continuously from Kafka has many benefits such as having the capability to gather insights faster. However, users must take into consideration management of Kafka offsets in order to recover their streaming application from failures. In this post, we will provide an overview of Offset Management and following topics. Storing offsets in external data stores Checkpoints HBase ZooKeeper Kafka Not managing offsets Overview of Offset Management Spark Streaming integration with Kafka allows users to read messages from a single Kafka topic or multiple Kafka topics. Read more The post Offset Management For Apache Kafka With Apache Spark Streaming appeared first on Cloudera Engineering Blog.

✔ Read More...

I guess you came to this post by searching similar kind of issues in any of the search engine and hope that this resolved your problem. If you find this tips useful, just drop a line below and share the link to others and who knows they might find it useful too.

Stay tuned to my blog, twitter or facebook to read more articles, tutorials, news, tips & tricks on various technology fields. Also Subscribe to our Newsletter with your Email ID to keep you updated on latest posts. We will send newsletter to your registered email address. We will not share your email address to anybody as we respect privacy.

This article is related to

CDH,Kafka,Spark,checkpointing,commitasync,data lifecycle,failure,offset management,recover,reliability,spark,spark streaming