CCDAK Questions and Answers

To increase consumer throughput, one effective strategy is to increase the amount of data fetched in each poll by raising fetch.max.bytes or max.partition.fetch.bytes. This allows each poll to retrieve more records per request, improving processing efficiency.

From Kafka Consumer Config Docs:

“Increasing fetch size allows consumers to retrieve larger batches of messages, improving throughput and reducing request overhead.”

Removing consumers (A) may reduce parallelism.

Manual commit (C) adds complexity, not throughput.

Decreasing session timeout (D) risks unnecessary rebalances.

The Apache Kafka consumer documentation explains that when a RecordDeserializationException occurs, the consumer cannot continue polling until the problematic offset is skipped. Simply logging the error is insufficient, because the consumer will repeatedly fail on the same record.

The recommended pattern is to log the malformed record, extract its topic, partition, and offset from the exception, and then call consumer.seek() to move the consumer position to the next offset (offset + 1). This allows the application to skip the poison pill and resume processing subsequent records.

Option B is invalid because Kafka does not provide a consumer.skip() API. Option C is unnecessary if the application is designed to tolerate malformed records. Option D results in an infinite failure loop.

Therefore, seeking past the bad record after logging it is the correct and officially documented way to handle poison pill records while maintaining consumer liveness and resilience.

According to the official Apache Kafka Connect REST API documentation, restarting a connector can be done at different granularities: connector-level, task-level, or both. By default, calling POST /connectors/{name}/restart only restarts the connector instance, not its tasks.

To restart the connector and all of its tasks, the request must explicitly include the query parameter includeTasks=true. This instructs Kafka Connect to restart the connector and all associated tasks, regardless of whether they are currently running or failed.

Option B (onlyFailed=true) limits the restart to only failed tasks, which does not meet the requirement to restart all tasks. Option C restarts only the connector instance, leaving tasks untouched. Option D restarts a single specific task and does not restart the connector itself.

Therefore, the correct and officially documented REST request to restart the connector instance and all tasks is:

POST /connectors/s3-connector/restart?includeTasks=true

This ensures a full restart of the connector and its execution context across the Kafka Connect cluster.

The Apache Kafka producer API is asynchronous by default, and proper error handling requires explicit mechanisms. The official documentation recommends using a callback function with producer.send() to determine whether a record was successfully acknowledged or failed due to an exception. The callback provides access to both RecordMetadata and any exception encountered during sending.

Additionally, certain errors—such as serialization errors or configuration issues—are thrown synchronously when calling producer.send(). These must be handled using a try/catch block that catches KafkaException or its subclasses.

Option A is invalid because ProducerRecord has no status() method. Option C is incorrect because producer.send() returns a Future, not a RecordMetadata object directly.

Thus, combining a callback for asynchronous errors and a try/catch block for synchronous errors is the correct and officially documented approach to ensure reliable producer error handling when using acks=all.

=========================

Comprehensive and Detailed Explanation From Exact Extract:

✅ C. Consumers can consume both committed and uncommitted transactions.By default, Kafka consumers only read committed messages if they are configured with isolation.level=read_committed. However, if configured as read_uncommitted, they can also consume uncommitted (potentially aborted) transactional messages.

From Kafka Documentation:

"The isolation.level setting controls whether the consumer will read only committed messages or all messages, including uncommitted messages from ongoing or aborted transactions."

✅ D. Information about producers and their transactions is stored in the _transaction_state topic.Kafka uses an internal topic named __transaction_state to maintain metadata about producer transactions. This topic is essential for tracking the transaction lifecycle, fencing, and recovery.

From Kafka Internals:

“Kafka stores the state of active and completed transactions in an internal topic called __transaction_state.”

If the consumer fails to send heartbeats in time (due to being slow under load), it is considered dead, causing a rebalance. Increasing session.timeout.ms gives the consumer more time to process messages and send heartbeats, preventing premature removal.

From Kafka Docs > Heartbeats and Failures:

“If the consumer does not send a heartbeat in session.timeout.ms, it is considered dead and the group coordinator will trigger a rebalance.”

Reducing max.poll.records might slow consumption further. More consumers won’t help since each partition already has a dedicated consumer.

According to the official Apache Kafka Streams documentation, session windows are specifically designed to model periods of activity separated by gaps of inactivity. A session window groups events that occur close together in time and closes when no new events arrive within a defined inactivity gap.

In this use case, tracking user shopping cart behavior requires detecting when a user becomes inactive (for example, stops adding or removing items). Session windows naturally represent this pattern by creating a new window when activity resumes after a period of silence. This makes them ideal for modeling user sessions, cart abandonment detection, and inactivity analysis.

Hopping and tumbling windows (Options B and C) are fixed-time windows and are not well-suited for detecting inactivity, as they segment data based on clock time rather than behavior. Sliding windows are primarily used for continuous joins and do not explicitly model inactivity gaps.

Therefore, session windows are the correct and officially recommended choice for identifying periods of user inactivity in Kafka Streams applications.

=========================

A GlobalKTable is a replicated, read-only table available in full on all instances. It's best for reference or lookup datasets such as:

Product catalog

Customer profiles

Warehouse inventory

From Kafka Streams Developer Guide:

“Use GlobalKTable when you need to perform joins using non-partition-aligned reference data that's small enough to replicate.”

Purchases and transactions are high-throughput, time-sensitive streams, not static reference data.

The Apache Kafka Streams documentation clearly distinguishes between the Kafka Streams DSL and the Processor API. While the DSL is designed for common stream processing operations such as filtering, mapping, joining, and aggregations, it does not support fully custom processing logic.

For use cases that require custom transformations, fine-grained control over record processing, access to headers, timestamps, state stores, and punctuation, Kafka Streams provides the Processor API. This API allows developers to implement custom Processor, Transformer, or ValueTransformer classes and explicitly define the processing topology.

Option A (TopologyTestDriver) is a testing utility, not a development API. Option C (DSL) is higher-level and not suitable for advanced custom logic. Option D does not exist in Kafka.

Therefore, the correct and officially supported approach for defining custom processors in Kafka Streams is to use the Processor API.

=========================

The default maximum message size that a Kafka broker accepts is 1MB (1,048,576 bytes), controlled by the config property message.max.bytes.

From Kafka Broker Configuration Docs:

“The default maximum message size is 1MB. To accept larger messages, configure message.max.bytes and the producer’s max.request.size.”

Producers also have a matching limit via max.request.size, and consumers via fetch.message.max.bytes.

The Apache Kafka documentation explains that Kafka’s parallelism model is based on topic partitions and consumer groups. Within a single consumer group, each partition can be assigned to at most one consumer at a time. As a result, the maximum level of parallelism for a consumer group is equal to the number of partitions in the topic.

In this scenario, the topic has five partitions, which means that up to five consumers in the same consumer group can actively consume data in parallel. If more than five consumers are added, the extra consumers will remain idle because there are no additional partitions to assign.

Option B is incorrect because Kafka does not enforce equal message consumption across partitions; message distribution depends on producers and partitioning strategy. Option C is false because partitions may be hosted on different brokers, and consumers fetch data from multiple brokers. Option D is incorrect because a single partition cannot be consumed by multiple consumers within the same group simultaneously.

Therefore, the only valid assumption is that a maximum of five consumers can actively consume messages in the same consumer group.

=========================

Increasing partitions increases parallelism, but also means:

Consumers in a group may have to handle more partitions, especially if the number of consumers is lower than the number of partitions.

This can result in increased lag, especially under high load.

From Kafka Topic Management Docs:

“Increasing the number of partitions increases consumer work, and if consumers can’t keep up, lag can accumulate.”

A is false: existing data is not redistributed.

B is false: records with the same key always map to the same partition based on hash.

D is not directly impacted by the partition count.

If the application is CPU-bound and lagging, adding more consumers to the group will allow better parallel processing, especially since the topic has 4 partitions, allowing up to 4 active consumers.

From Kafka Consumer Group Docs:

“Kafka achieves parallelism by distributing partitions across consumers in a group. Adding consumers helps reduce lag if partitions are underutilized.”

B may help but requires repartitioning and coordination.

C or D affects how much data is polled, not how fast it’s processed.

Kafka Connect source connectors use the offset.storage.topic configuration parameter to define where to store offsets that track the connector's position in the source system (e.g., database, file, etc.).

From Kafka Connect Worker Configuration:

“offset.storage.topic specifies the topic in Kafka where the connector stores offsets for each source partition.”

This allows Kafka Connect to resume from the correct position on restart.

The other options (B–D) are not valid Kafka configs.

According to the official Apache Kafka Connect and Confluent JDBC Source Connector documentation, the correct connector class for a JDBC source connector is io.confluent.connect.jdbc.JdbcSourceConnector. This connector is used to read data from relational databases and publish each table as a Kafka topic.

To read all rows from a specific table, the configuration must include table.whitelist (or table.include.list in newer versions) with the table name, and a topic.prefix to determine the Kafka topic name. In this case, using topic.prefix=dbl- with table.whitelist=orders results in records being written to the dbl-orders topic, which matches the requirement.

Options A, B, and C are invalid because they reference a non-existent connector class (DdbcSourceConnector), contain unsupported properties such as topic.whitelist for a source connector, or include malformed/incorrect JDBC parameters. Additionally, blacklisting tables does not ensure that only the orders table is read.

Therefore, option D is the only configuration that is syntactically correct, uses the proper connector class, and aligns with the official Kafka Connect JDBC Source Connector documentation.

According to the official Apache Kafka consumer semantics documentation, at-most-once delivery means that records may be lost but must never be processed more than once. To achieve this guarantee, a consumer must commit offsets before processing records. If the application crashes after committing but before processing, messages are skipped rather than reprocessed, which satisfies at-most-once semantics.

Option B correctly implements this pattern. It disables automatic offset commits (enable.auto.commit=false) and explicitly calls commitAsync() immediately after polling and before processing records. This ensures that offsets are advanced regardless of whether processing succeeds, eliminating the possibility of duplicate processing.

Option C represents at-least-once semantics, because offsets are committed after processing, which can lead to duplicates on failure. Options A and D rely on automatic commits or incorrect ordering and do not provide strict at-most-once guarantees.

The Kafka documentation explicitly states that manual offset management with commits before processing is required for at-most-once delivery.

Therefore, Option B is the best and officially documented configuration and code skeleton to avoid duplicate message processing.

According to the Apache Kafka consumer and scalability documentation, parallelism in message processing is primarily determined by the number of partitions and the number of consumers within a consumer group. Each partition can be assigned to only one consumer at a time. In this scenario, the topic already has four partitions, but only two consumers are processing them, meaning only two partitions are actively consumed in parallel.

Since monitoring indicates that message processing is CPU-bound, the bottleneck is not Kafka I/O but application-side processing. Adding more consumers (up to the number of partitions) allows Kafka to rebalance the partitions and distribute them across more consumer instances, increasing CPU parallelism and reducing lag.

Option A (adding more partitions) can increase parallelism but is more intrusive and may require repartitioning considerations for keys and downstream systems. Option B increases batch size, which can worsen CPU pressure. Option D may reduce per-poll workload but does not increase parallel processing capacity.

Therefore, the correct and recommended action per Kafka documentation is to add more consumers to the consumer group.

=========================

Apache Kafka Connect stores source connector offsets in a Kafka topic defined by the offset.storage.topic configuration. Since this property is explicitly set to connect-offsets, Kafka Connect will commit offsets to that topic, making Option C correct.

The offset.storage.replication.factor controls how many replicas the offsets topic will have. When this value is set to -1, Kafka Connect uses the broker’s default replication factor, as documented in the Kafka Connect worker configuration reference. Therefore, Option A is also correct.

Option B is incorrect because Kafka Connect does not use Kafka’s internal consumer offsets topic (__consumer_offsets). It always uses the configured offsets topic. Option D is incorrect because offset.flush.timeout.ms defines how long Connect will wait for offset commits to complete, not how long it waits before attempting a commit. The commit interval itself is controlled by offset.flush.interval.ms (60 seconds in this case).

Thus, the correct statements that match the expected behavior are A and C.

With acks=all and min.insync.replicas=2, Kafka requires at least two in-sync replicas to acknowledge a write. If only one broker is alive, the condition fails, and NotEnoughReplicasException is thrown by the producer.

From Kafka Producer Exception Docs:

“NotEnoughReplicasException is thrown when the number of in-sync replicas is insufficient to satisfy acks=all with min.insync.replicas.”

NetworkException is generic and not raised here.

NotCoordinatorException is related to consumer group coordination.

NotLeaderForPartitionException is unrelated unless accessing an unassigned leader.

The official CLI utility for consuming messages from Kafka topics is kafka-console-consumer.sh. It connects to the broker, consumes messages, and prints them to standard output.

From Kafka CLI Tools Documentation:

"kafka-console-consumer.sh is used to read data from a Kafka topic and write it to standard output."

The other options are not valid Kafka CLI tools.

The Apache Kafka documentation states that consumer offsets are stored per consumer group and must be explicitly reset if an application needs to reprocess data from an earlier position. The officially supported and safest way to reset offsets for an existing consumer group is by using the kafka-consumer-groups CLI tool with the --reset-offsets option and the --to-earliest flag.

Setting auto.offset.reset=earliest (Option D) only applies when no committed offset exists for a partition. Since this consumer is already running in production, offsets are already committed, making this option ineffective. Option C does not change offsets because Kafka identifies consumers by group ID, not application instance. Option B is extremely dangerous and incorrect, as setting retention.ms=0 deletes data for all consumers and impacts other applications.

Therefore, using the CLI to reset offsets for the specific consumer group ensures controlled, auditable, and targeted offset management without affecting other consumers or the topic itself. This approach is explicitly documented in Kafka’s consumer group administration tools.

=========================

RetriableException is a subclass of KafkaException, indicating transient issues that might succeed if retried. Examples include:

LeaderNotAvailableException: Happens when metadata is not yet propagated or leader election is in progress.

NotEnoughReplicasException: Happens when the number of in-sync replicas is insufficient. This can be transient if replicas come back online.

From the Apache Kafka Java Client documentation:

"These exceptions (like LeaderNotAvailableException, NotEnoughReplicasException) are transient and the client will retry them automatically depending on retry configuration."

RecordTooLargeException and AuthorizationException are non-retriable as they indicate client-side or permission errors.