Apache Kafka CCDAK Exam Notes Apache Kafka CCDAK Exam Notes

Page content

Dear Readers,

If you are planning or preparing for Apache Kafka Certification then this is the right place for you.There are many Apache Kafka Certifications are available in the market but CCDAK (Confluent Certified Developer for Apache Kafka) is the most known certification as Kafka is now maintained by Confluent.

CCDAK vs CCAAK

CCAAK is for devOps engineers focusing on building and managing a Kafka cluster. CCDAK is for developers and Solution architects focusing on design, producer and consumer. If you are still not sure, I recommend to go for CCDAK as it is a more comprehensive exam compared to CCAAK. These exam notes are very helpful for both CCDAK and CCAAK certifications.

From here onward, we will talk about how to prepare for CCDAK.

Frequently Asked Questions


How much preparation do I need for the CCDAK exam?
At least 40-50 hours of preparation is required, since the exam verifies your theoretical as well as practical understanding of Kafka.

Where do I register and schedule the CCDAK exam?
Register and schedule your proctored exam (delivered via Honorlock) on the Confluent Certification page. I suggest setting a goal of 1 to 2 months for exam preparation before registering.

What does the exam cost, and how soon can I retake it if I fail?
Current exam pricing is listed on the Confluent Certification page. If you fail, you need to wait at least 7 days before purchasing and retaking the exam.

What is the CCDAK exam format?
You need to answer 60 multiple-choice questions in 90 minutes from your laptop (with webcam) under the supervision of an online proctor. There is no negative scoring, so try to answer as many questions as possible.

How is the exam scored, and when do I get my result?
Confluent doesn’t disclose how many questions you need to answer correctly to pass. Your result (PASS or FAIL) is shown immediately at the end of the exam, with no scoring or percentage provided.

What do I get after passing, and how long does it last?
You’ll receive a certificate similar to my CCDAK certificate. What an achievement !!! The certification expires after two years, so you’ll need to recertify to keep it current.

Are there CCDAK exam dumps I can study from?
Don’t waste time searching for CCDAK certification dumps - there ain’t any. Confluent has launched a FREE Fundamentals Accreditation Exam you can signup for free to better understand the CCDAK exam.

Exam Preparation


I have prepared for CCDAK using following:

  1. Apache Kafka Documentation
  2. Confluent Kafka Documentation
  3. Confluent Kafka Definitive Guide PDF
  4. Udemy Apache Kafka Series - Learn Apache Kafka for Beginners v2
  5. Udemy CCDAK 150 Practice Exam Questions

You should prepare well for following topics. It is recommended to study the topics in the same sequence.

# Topic Study & Revise
1 Kafka Architecture Read Confluent Kafka Definitive Guide PDF and Apache Kafka Documentation, then revise using the Kafka Architecture section of this post
2 Kafka CLI Read Confluent Kafka Definitive Guide PDF, then revise using the KAFKA CLI section of this post
3 Kafka Streams Read Confluent Kafka Definitive Guide PDF, then revise using the Kafka Streams section of this post
4 Kafka Security Read Apache Kafka Documentation Security Section
5 Kafka APIs Read Apache Kafka Documentation API section, then revise using the Kafka Client APIs section of this post
6 Kafka Monitoring (Metrics) Read Confluent Kafka Definitive Guide PDF and Apache Kafka Documentation for important metrics, and Confluent Kafka Documentation as well
7 Confluent Schema Registry Read Confluent Kafka Documentation, then revise using the Confluent Schema Registry section of this post
8 ksqlDB (formerly KSQL) Read Confluent ksqlDB Documentation
9 Confluent REST Proxy Read Confluent Kafka Documentation Rest Proxy Section

Sample Exam Questions


Please note that these are not the actual questions from the CCDAK exam but most likelihood to ask in exam.

1. Kafka Theory
Question Answer
Kafka is a …. ? pub-sub system
Mostly Kafka is written in which language? Scala
Which errors are retriable from Kafka Producer? LEADER_NOT_AVAILABLE, NOT_LEADER_FOR_PARTITION, UNKNOWN_TOPIC_OR_PARTITION
What is a generic unique id which can be used for a message received from a consumer? Topic + Partition + Offset

Read Kafka Architecture section of this post for more questions and answers

2. Kafka Streams
Question Answer
To transform data from a Kafka topic to another one, you should use? Kafka Streams
Which of the Kafka Stream operators are stateful? (see Kafka Streams section)
Which of the Kafka Stream operators are stateless? (see Kafka Streams section)
Which window is not having a gap? (see Kafka Streams section)
Which Kafka Stream joins doesn’t require co-partition of data? KStream-to-GlobalKTable
Which Kafka Stream joins is always a windowed join? KStream-to-KStream
What is the output of a KStream-to-KTable join? KStream

Read Kafka Streams section of this post for answers

3. Confluent Schema Registry
Question Answer
Which of the following is not a primitive type of Avro? (see Confluent Schema Registry section)
Which of the following is not a complex type of Avro? (see Confluent Schema Registry section)
Which of the following is not a required field in an Avro Schema? (see Confluent Schema Registry section)
Delete a field without default value in Avro schema is …… compatibility? backward
Adding a field to a record without default value is …… schema evolution? forward
In Avro, removing or adding a field that has a default value is a …… schema evolution? full
What client protocols are supported for the schema registry? HTTP, HTTPS
Where are Avro schemas stored in Confluent Schema Registry? _schemas Kafka topic

Read Confluent Schema Registry section of this post for answers

4. ksqlDB
Question Answer
Is ksqlDB ANSI SQL Compliant? No
What Java library is ksqlDB based on? Kafka Streams
5. Kafka Security
Question Answer
What are the valid authentication mechanisms in Kafka? SSL
SASL/GSSAPI (Kerberos)
SASL/PLAIN
SASL/SCRAM-SHA-256 and SASL/SCRAM-SHA-512
SASL/OAUTHBEARER

Kafka Architecture


☛ I have spent so much time preparing these notes. You can expect most of the questions related to Kafka architecture from these notes. ☛ Moreover, If you are preparing for interviews then you can also expect most of the Kafka interview questions from these notes.

Cluster

  • Cluster typically consist of mutliple Kafka brokers.
  • All Kafka brokers within a Cluster are managed and coordinated by a KRaft controller quorum using the Raft consensus protocol.

Rack

  • Rack is a logical grouping of Kafka brokers within a Cluster.
  • Cluster can have multiple racks. Rack can have multiple brokers.
  • Broker belongs to a rack when property broker.rack=<rack-id> is specified at broker level. This enables the rack awareness feature which spreads replicas of the same partition across different racks.
  • Let’s say you have 6 brokers and 2 racks. Brokers 1, 2, 3 are on the rack_1, and brokers 4, 5, 6 are on rack_2.
    Now when you create a topic with 6 partition, instead of assigning broker to partition in order from 1, 2, 3, 4, 5, 6, each partition is assigned to each rack repeatedly i.e. 1, 4, 2, 5, 3, 6.
    ┌――――――――――――┐                  ┌――――――――――――┐
    |  rack_1    |                  |  rack_2    | 
    |――――――――――――|                  |――――――――――――|
    |  Broker 1  |◁― Partition 1    |  Broker 4  |◁― Partition 2
    |  Broker 2  |◁― Partition 3    |  Broker 5  |◁― Partition 4
    |  Broker 3  |◁― Partition 5    |  Broker 6  |◁― Partition 6
    └――――――――――――┘                  └――――――――――――┘
    

Broker

  • Every broker in Kafka is a bootstrap server which knows about all brokers, topics and partitions (metadata) that means Kafka client (e.g. producer,consumer etc) only need to connect to one broker in order to connect to entire cluster.
  • At all times, only one broker should be the controller, and one broker must always be the controller in the cluster

Topic

  • Kafka takes bytes as input without even loading them into memory (that’s called zero copy)
  • Brokers have defaults for all the topic configuration parameters

Partition

  • Topic can have one or more partition.
  • Each partition can have one or more replica decided by Replication Factor. Replication Factor = 3 means One leader partition and two replicas.
  • It is not possible to delete a partition of topic once created.
  • Order is guaranteed within the partition and once data is written into partition, its immutable!
  • If producer writes at 1 GB/sec and consumer consumes at 250MB/sec then requires 4 partition!

Segment

  • Partitions are made of segments (*.log files)
  • At a time only one segment is active in a partition
  • Segment stores the actual Kafka messages along with offset, timestamp, compression etc.
  • log.segment.bytes = 1 GB (default), Max size in bytes to close the segment and roll over to new segment.
  • log.segment.ms = 1 Week (default), Max time in ms to wait before closing the segment even if it is not full or reached Max Size.
  • Every segment also has two indexes (files):-
    1. An offset to position index (*.index file) - Allows kafka where to read to find a message
    2. A timestamp to offset index (*.timeindex file) - Allows kafka to find a message with a timestamp

Log Retention and Cleanup policies

  • Log cleanup happen on partition segments (.log files). Smaller or more segments means the log cleanup will happen more often!
  • Old segments are cleaned up based on time-based and size-based log retention policies whichever happens first.
  • Time Based Retention: The segment older by configured retention time eligible for Cleanup. You can use any of the three configuration log.retention.ms, log.retention.minutes, or log.retention.hours to specify retention period in ms, minutes or hours. If ms is not set, minutes is used. If minutes is not set, hours is used. Default retention time is 1 Week.
  • Size Based Retention: The older segment are cleaned up when the max configured size of a topic partition (includes all segments), is reached. You can use log.retention.bytes=-1 (default is infinite) to configure max size in bytes.
  • Cleanup process checks any logs to cleanup in every log.cleaner.backoff.ms=15 seconds (default).
  • Cleanup Policy: Logs are deleted, compacted, or both based on log.cleanup.policy. Delete policy discard the old segments when their retention time or size is reached. Compact policy delete the old messages per key and keep the latest copy for that key.
  • Deleted records can still be seen by consumers for a period of delete.retention.ms=24 hours (default)

Offset

  • Each Topic Partition is having its own offset starting from 0.

Topic Replication

  • Replication factor = 3 and partition = 2 means there will be total 6 partition distributed across Kafka cluster. Each partition will be having 1 leader and 2 ISR (in-sync replica).
  • Broker contains leader partition called leader of that partition and only leader can receive and serve data for partition.
  • Replication factor can not be greater then number of broker in the kafka cluster. If topic is having a replication factor of 3 then each partition will live on 3 different brokers.
  • Kafka also tracks Eligible Leader Replicas (ELR) - a subset of the ISR that’s guaranteed to be fully caught up to the high-watermark. This lets the controller safely elect a new leader after an unclean shutdown without risking data loss or requiring unclean.leader.election.enable=true.

Producer

Kafka Producer error handling by type:

Error Type Errors
Retriable (producer automatically recovers) LEADER_NOT_AVAILABLE, NOT_LEADER_FOR_PARTITION
Non-Retriable (producer throws immediately) OFFSET_OUT_OF_RANGE, BROKER_NOT_AVAILABLE, MESSAGE_TOO_LARGE, INVALID_TOPIC_EXCEPTION
  • If you send a message of size 3 MB to a topic with default message size configuration. Then producer will throw MessageSizeTooLarge exception immediately since it is not a retriable exception.
  • When produce to a topic which doesn’t exist and auto.create.topic.enable=true then kafka creates the topic automatically with the broker/topic settings num.partition and default.replication.factor

Producer Acknowledgment

acks Producer… Data Loss
acks=0 do not wait for acknowledgement possible data loss
acks=1 wait for leader acknowledgement limited data loss
acks=all wait for leader+replica acknowledgement no data loss

acks=all must be used in conjunction with min.insync.replicas which can be set at broker or topic level. *assuming that replicas are distributed across 3 brokers for below points

  • min.insync.replica only matters if acks=all
  • acks=all, min.insync.replica=2 implies that at least 2 brokers that are ISR (including leader) must acknowledge
  • acks=all, min.insync.replica=1 implies that at least 1 brokers that is ISR (including leader) must acknowledge
  • A kafka topic with replication.factor=3, acks=all, min.insync.replicas=2 can only tolerate 1 broker going down, otherwise the producer will receive an exception NOT_ENOUGH_REPLICAS on send.
  • A kafka topic with replication.factor=3, acks=all, min.insync.replicas=1, can tolerate maximum number of 2 brokers going down, so that a producer can still produce to the topic.

Producer Configuration

  • Mandatory properties to configure Kafka producer is as follows:
    bootstrap.servers
    key.serializer
    value.serializer

Safe Producer Configuration

  • min.insync.replicas=2 (set at broker or topic level)
  • retries=MAX_INT number of reties by producer in case of transient failure/exception. (default is 0)
  • max.in.flight.per.connection number=5 number of producer request can be made in parallel (default is 5)
  • acks=all
  • enable.idempotence=true producer send producerId with each message to identify for duplicate msg at kafka end. When kafka receives duplicate message with same producerId which it already committed. It do not commit it again and send ack to producer (default is false)

High Throughput Producer using compression and batching

  • compression.type=snappy value can be none(default), gzip, lz4, snappy. Compression is enabled at the producer level and doesn’t require any config change in broker or consumer Compression is more effective in case of bigger batch of messages being sent to kafka
  • linger.ms=20 Number of millisecond a producer is willing to wait before sending a batch out. (default 0). Increase linger.ms value increase the chance of batching.
  • batch.size=32KB or 64KB Maximum number of bytes that will be included in a batch (default 16KB). Any message bigger than the batch size will not be batched

Message Key

  • Producer can choose to send a message with key.
  • If key=null, data is sent in round robin fashion
  • If key!=null and has some value, then all the message for that key always go to the same partition. This can be used to order the messages for a specific key since order is guaranteed in same partition.
  • Adding a partition to the topic will loose the guarantee of same key go to same partition.
  • Keys are hashed using murmur2 algorithm by default.

Message Size

  • Default Message max size is 1MB
  • If you try to send message > 1MB then MessageSizeTooLargeException is thrown

Suppose you want to send a 15MB message from producer to broker to consumer successfully, you need to configure:

Config Level Purpose
message.max.bytes=15728640 broker/topic Max size of message that can be received by the broker from the producer
replica.fetch.max.bytes=15728640 broker Max size of message that can be replicated across brokers
max.partition.fetch.bytes=15728640 consumer Max size of message that can be fetched by the consumer per partition

Consumer

  • Per thread one consumer is the rule. Consumer must not be multi threaded.
  • Each consumer is assigned to different partition in same consumer group.
    • If there are 5 consumers of same consumer group consuming from a topic with 10 partition then 2 partitions will be assigned to each consumer and no consumer will remain idle.
    • If there are 10 consumers of same consumer group consuming from a topic with 5 partition then 5 partition will be assigned to 5 consumers and rest 5 consumers will remain idle.
  • records-lag-max (monitoring metrics) The maximum lag in terms of number of records for any partition in this window. An increasing value over time is your best indication that the consumer group is not keeping up with the producers.

Consumer Group

  • If two applications want to process all the messages independently from a kafka topic having 4 partition, then you should create 2 consumer groups with 4 consumers in each group for optimal performance.

Consumer Rebalance Protocol

  • Kafka 4.0 introduced the next generation consumer rebalance protocol (group.protocol=consumer) as the new default, replacing the older classic protocol.
  • In the classic protocol, all consumers in a group paused during a rebalance (a stop-the-world pause) while clients ran the assignment logic themselves.
  • In the new protocol, the group coordinator (broker) computes and pushes partition assignments to consumers incrementally, so only the consumers actually affected by a change pause - the rest of the group keeps processing.
  • As of Kafka 4.3, the classic protocol is on a deprecation path, with brokers logging warnings that recommend migrating to the new protocol.

Share Groups (Queues for Kafka)

  • A share group lets multiple consumers cooperatively process records from the same partition at the same time, unlike a regular consumer group where each partition is assigned to exactly one consumer.
  • This gives Kafka traditional queue semantics (point-to-point delivery) on top of a topic, so the number of consumers in a share group can exceed the number of partitions.
  • Consume using the new KafkaShareConsumer API instead of KafkaConsumer. Partition assignment is entirely server-side - applications can not manually assign partitions.
  • Records can be acknowledged implicitly (automatically on poll()/commitSync()/commitAsync(), the default) or explicitly per record using acknowledge(record, AcknowledgeType.ACCEPT) or .REJECT.
  • Kafka tracks a delivery count per record to detect and handle poison messages that repeatedly fail processing.

Consumer Offset

When consumer in a group has processed the data received from Kafka, it commits the offset in Kafka topic named _consumer_offset which is used when a consumer dies, it will be able to read back from where it left off.

Delivery Semantics

  • At most once Offset are committed as soon as message batch is received. If the processing goes wrong, the message will be lost (it won’t be read again)
  • At least once (default) Offset are committed after the message is processed. If the processing goes wrong, the message will be read again. This can result in duplicate processing of message.
    • Make sure your processing is idempotent. (i.e. re-processing the message won’t impact your systems). For most of the application, we use this and ensure processing are idempotent.
  • Exactly once Native for Kafka=>Kafka workflows using the Kafka Streams API. For Kafka=>External-system workflows, use a Kafka Connect connector with exactly-once support (available for source connectors since KIP-618) or ensure the consuming side is idempotent.

Consumer Offset commit strategy

  • enable.auto.commit=true & synchronous processing of batches with auto commit, offset will be committed automatically for you at regular interval (auto.commit.interval.ms=5000 by default) every time you call .poll(). If you don’t use synchronous processing, you will be in “at most once” behavior because offsets will be committed before your data is processed.
  • enable.auto.commit=false & manual commit of offsets (recommended)

Consumer Offset reset behavior

  • auto.offset.reset=latest will read from the end of the log
  • auto.offset.reset=earliest will read from the start of the log
  • auto.offset.reset=none will throw exception of no offset is found
  • Consumer offset can be lost if hasn’t read new data in 7 days. This can be controlled by broker setting offset.retention.minutes

Consumer Poll Behavior

  • fetch.min.bytes = 1 (default), Control how much data you want to pull at least on each request. Help improving throughput and decreasing request number. At the cost of latency.
  • max.poll.records = 500 (default), Controls how many records to receive per poll request. Increase if your messages are very small and have a lot of available RAM.
  • max.partition.fetch.bytes = 1MB (default), Maximum data returned by broker per partition. If you read from 100 partition, you will need a lot of memory (RAM)
  • fetch.max.bytes = 50MB (default), Maximum data returned for each fetch request (covers multiple partition). Consumer performs multiple fetches in parallel.

Consumer Heartbeat Thread

  • Heartbeat mechanism is used to detect if consumer application in dead.
  • session.timeout.ms=10s (default), If heartbeat is not sent in 10 second period, the consumer is considered dead. Set lower value to faster consumer rebalances
  • heartbeat.interval.ms=3s (default), Heartbeat is sent in every 3 seconds interval. Usually 1/3rd of session.timeout.ms

Consumer Poll Thread

  • Poll mechanism is also used to detect if consumer application is dead.
  • max.poll.interval.ms = 5min (default), Max amount of time between two .poll() calls before declaring consumer dead. If processing of message batch takes more time in general in application then should increase the interval.

Kafka Guarantees

  • Messages are appended to a topic-partition in the order they are sent
  • Consumer read the messages in the order stored in topic-partition
  • With a replication factor of N, producers and consumers can tolerate upto N-1 brokers being down
  • As long as number of partitions remains constant for a topic ( no new partition), the same key will always go to same partition

Client Bi-Directional Compatibility

  • an Older client (1.1) can talk to Newer broker (2.0)
  • a Newer client (2.0) can talk to Older broker (1.1)

Kafka Connect

  • Source connect Get data from common data source to Kafka for e.g. import data from external database to kafka
  • Sink connect Publish data from Kafka to common data source for e.g. export data from Kafka to external database

KRaft

  • Kafka cluster metadata (topics, partitions, ISR, etc.) is managed by a KRaft controller quorum using the Raft consensus protocol.
  • A Kafka admin typically selects 3 or 5 controllers for the quorum. A majority of controllers must be alive to maintain availability - 3 controllers tolerate 1 failure, 5 controllers tolerate 2 failures.
  • Each node’s role is set with process.roles=broker, controller, or broker,controller (combined mode, handy for dev/test but not recommended for critical deployments).
  • Each node has a unique node.id, and the quorum is configured with controller.quorum.bootstrap.servers=host1:port1,host2:port2,… listing every controller. The default controller listener port is 9093.
  • One controller is elected the active controller (leader) through Raft consensus; the rest act as hot standbys. Cluster metadata itself is stored as a replicated log, not in a separate ensemble.

KAFKA CLI


  1. kafka-topics.sh allows you to create, modify, delete, and list information about topic in the cluster.
  2. kafka-console-producer.sh allows you to write messages into a kafka topic in your cluster.
  3. kafka-console-consumer.sh allows you to consume messages out of one or more topics in your Kafka cluster.
  4. kafka-consumer-groups.sh allows you to list consumer groups, describe, specific groups, delete consumer groups or specific group info, or reset consumer group offset information.
  5. kafka-configs.sh allows you to change the configuration dynamically for --entity-types topics, brokers, users, and clients.

You are required to provide the cluster connection string and port through the --bootstrap-server option.

Following are few examples of Kafka command line:-

① Start a kafka server at default port 9092

$bin/kafka-server-start.sh config/server.properties

② Create a kafka topic ‘my-first-topic’ with 3 partitions and 3 replicas

$bin/kafka-topics.sh \
    --bootstrap-server localhost:9092 \
    --topic my-first-topic \
    --create \
    --replication-factor 3 \
    --partitions 3

③ List all kafka topics in a cluster

$bin/kafka-topics.sh \
    --bootstrap-server localhost:9092 \
    --list

④ Modify kafka topic ‘my-first-topic’

$bin/kafka-topics.sh \
    --bootstrap-server localhost:9092 \
    --topic my-first-topic \
    --partitions 9 \
    --alter

This will add 6 more partition to the topic, which is already having 3 partition

⑤ Delete kafka topic ‘my-first-topic’

$bin/kafka-topics.sh \
    --bootstrap-server localhost:9092 \
    --topic my-first-topic \
    --delete

This will have no impact if delete.topic.enable is not set to true

⑥ Describe kafka topic ‘my-first-topic’ details

$bin/kafka-topics.sh \
    --bootstrap-server localhost:9092 \
    --topic my-first-topic \
    --describe

Find out all the partitions without a leader

$bin/kafka-topics.sh \
--bootstrap-server localhost:9092 \
--describe --unavailable-partitions

⑦ Produce messages to Kafka topic ‘my-first-topic’

$bin/kafka-console-producer.sh \
    --bootstrap-server localhost:9092 \
    --topic my-first-topic \
    --producer-property acks=all 
> message 1  
> message 2  
> ^C

⑧ Start Consuming messages from kafka topic ‘my-first-topic’

$bin/kafka-console-consumer.sh \
    --bootstrap-server localhost:9092 \
    --topic my-first-topic \
    --from-beginning
> message 1  
> message 2

⑨ Start Consuming messages in a consumer group from kafka topic ‘my-first-topic’

$bin/kafka-console-consumer.sh \
    --bootstrap-server localhost:9092 \
    --topic my-first-topic \
    --group my-first-consumer-group \
    --from-beginning

⑩ List all consumer groups

$bin/kafka-consumer-groups.sh \
    --bootstrap-server localhost:9092 \
    --list

⑪ Describe consumer group

$bin/kafka-consumer-groups.sh \
    --bootstrap-server localhost:9092 \
    -group my-first-consumer-group \
    --describe 

⑫ Reset offset of consumer group to replay all messages

$bin/kafka-consumer-groups.sh \
    --bootstrap-server localhost:9092 \
    --group my-first-consumer-group \
    --reset-offsets --to-earliest \
    --topic my-first-topic \
    --execute

⑬ Shift offsets by 2 (forward) of a consumer group

$bin/kafka-consumer-groups \
    --bootstrap-server localhost:9092 \
    --group my-first-consumer-group \
    --reset-offsets --shift-by 2 \
    --topic my-first_topic \
    --execute

You can shift offsets by 2 (backward) using --shift-by -2

⑭ Setting the retention for the topic named ‘my-first-topic’ to 1 hour (3,600,000 ms):

$bin/kafka-configs.sh \
    --bootstrap-server localhost:9092
    --entity-type topics --entity-name my-first-topic
    --add-config retention.ms=3600000
    --alter

Kafka Streams


Kafka Streams is used for building streaming applications which transform data of one Kafka topics and feeds to another Kafka topic.

1. Stateless & Stateful Operators

Stateless Operators Stateful Operators
branch join
filter aggregate
inverseFilter count
flatMap reduce
flatMapValues windowing
foreach
groupByKey
groupBy
map
mapValues

2. Window

Window Characteristics Example
Tumbling window Time based, Fixed Size, Non overlapping, Gap less If window-size=5min and advance-interval=5min, windows look like [0-5min] [5min-10min] [10min-15min]…..
Hopping window Time based, Fixed Size, Overlapping If window-size=5min and advance-interval=3min, windows look like [0-5min] [3min-8min] [6min-11min]…..
Sliding window Fixed size, overlapping, works on the difference between record timestamps Used only for join operations
Session window Session based, Dynamically sized, Non overlapping, Data driven Used to aggregate key based events into a session

For more information, refer Apache Kafka Documentation on windowing

3. SerDes data types

Kafka stream operations require SerDes (Serializer/Deserializer) to identify data type: byte[], ByteBuffer, Double, Integer, Long, String.

4. Streams DSL

Type Abstraction DB Analogy Example Use Case
KStream Record stream from a subset of partitions of a topic INSERT statement In an e-commerce application, transactions like purchase or payment should be modeled as a KStream
KTable Changelog stream from a subset of partitions of a topic UPDATE statement In an e-commerce application, static data like inventory list, customer list, and aggregated data like total sales should be modeled as a KTable
GlobalKTable Changelog stream from all partitions of a topic UPDATE statement Reference/lookup data that every stream task needs a full local copy of

For more information, refer Apache Kafka Documentation on stream DSL

5. Join Operands

Join Operands Output Type co-partition required Join Type
KStream-to-KStream KStream Windowed Yes key and window based
KTable-to-KTable KTable Non-windowed Yes key or foreign-key based
KStream-to-KTable KStream Non-windowed Yes key based
KStream-to-GlobalKTable KStream Non-windowed No key or foreign-key based
co-partition

co-partition means both the left and right join operand topics must have same number of partitions.

A join between a topic A ( 5 parition ) and topic B (3 partition) is possible using KStream-to-GlobalKTrade since it does not require co-partition.

Kafka Client APIs


  • Click here to find out how we can create a Safe and high throughput Kafka Producer using Java.
  • Click here to find out how we can create a Kafka consumer using Java with manual auto commit enabled.

Confluent Schema Registry


1. Avro

Primitive Types Complex Types
null record
boolean enum
int (32 bit) array
long (64 bit) map
float (32 bit) union
double (64 bit) fixed
byte[] (8 bit)
string (char sequence)
Avro Schema Definition
Level Field Required?
Schema namespace Required
Schema type => record, enum, array, map, union, fixed Required
Schema name Required
Schema doc Optional
Schema aliases Optional
Schema fields (array) Required
fields[] name Required
fields[] type Required
fields[] doc Optional
fields[] default Optional
fields[] order Optional
fields[] aliases Optional

2. Confluent Schema Notes

  • Schema Registry stores all schemas in a Kafka topic _schemas defined by kafkastore.config = _schemas (default) which is a single partition topic with log compacted.
  • The default response media type application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json are used in response header.
  • HTTP and HTTPS client protocol are supported for schema registry.
  • Prefix to apply to metric names for the default JMX reporter kafka.schema.registry
  • Default port for listener is 8081
  • Confluent support primitive types of null, Boolean, Integer, Long, Float, Double, String, byte[], and complex type of IndexedRecord. Sending data of other types to KafkaAvroSerializer will cause a SerializationException

3. Confluent Schema Compatibility Types

Each type below compares a schema version X against the immediately prior version X-1, unless you use the _TRANSITIVE variant (e.g. BACKWARD_TRANSITIVE), which extends the same guarantee across all previous versions X, X-1, X-2, and so on.

Type Add a Field Delete a Field Guarantee Upgrade Order
BACKWARD (default) Only with a default value (optional field) Allowed without a default value (required field) - consumer ignores it Consumer on schema X can read data produced with schema X or X-1. No assurance an older-schema consumer can read new-schema data. Upgrade all consumers before you start producing new events
FORWARD Allowed without a default value (required field) - consumer ignores it Only with a default value (optional field) - consumer assigns the default Data produced with schema X can be read by consumers on schema X or X-1. No assurance a new-schema consumer can read old-schema data. Upgrade all producers to the new schema first, then upgrade the consumers
FULL Only with a default value Only with a default value Backward and forward compatible between schema X and X-1 - both directions are guaranteed. Producers and consumers can be upgraded independently
NONE Any change allowed Any change allowed Compatibility checks are disabled entirely. Depends on the change - e.g. changing a field’s type usually needs producers and consumers upgraded at the same time

Default Ports


Service Port
Broker 9092
KRaft Controller 9093
REST Proxy 8082
Schema Registry 8081
ksqlDB 8088

Also Read How to install kafka broker on local machine for development and testing

Also Read Configure Kafka Producer and Consumer in Spring Boot

Also Read How to Configure Multiple Kafka Producer in Spring Boot

Also Read How to Configure Multiple Kafka Consumer in Spring Boot