Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Dumps with Practice Exam Questions Answers

Comprehensive and Detailed Explanation:

To read from a specific Kafka topic using Structured Streaming, the correct syntax is:

python

CopyEdit

option("subscribe","feed")

This is explicitly defined in the Spark documentation:

"subscribe – The Kafka topic to subscribe to. Only one topic can be specified for this option."

(Source:Apache Spark Structured Streaming + Kafka Integration Guide)

B."subscribe.topic" is invalid.

C."kafka.topic" is not a recognized option.

D."topic" is not valid for Kafka source in Spark.

Comprehensive and Detailed Explanation From Exact Extract:

Apache Arrow is used under the hood to optimize conversion between Pandas and PySpark DataFrames. The correct configuration setting is:

spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")

From the official documentation:

“This configuration must be enabled to allow for vectorized execution and efficient conversion between Pandas and PySpark using Arrow.”

Option B is correct.

Options A, C, and D are invalid config keys and not recognized by Spark.

Final Answer: B

dedup_df = iot_bronze_df.dropDuplicates(["event_ts","sensor_id","metric_value"])

dropDuplicates accepts a list of columns to use for deduplication.

This ensures only unique records based on the specified keys are retained.

Temp views likecreateOrReplaceTempVieware session-scoped.

They disappear once the Spark session ends.

To retain data across sessions, it must be persisted:

df.write.saveAsTable("users_vw")

Thus, the view needs to be persisted as a table to survive session termination.

Comprehensive and Detailed Explanation From Exact Extract:

The message"GC overhead limit exceeded"typically indicates that the JVM is spending too much time in garbage collection with little memory recovery. This suggests that the driver or executor is under-provisioned in memory.

The most effective remedy is to increase the driver memory using:

--driver-memory 4g

This is confirmed in Spark's official troubleshooting documentation:

“If you see a lot ofGC overhead limit exceedederrors in the driver logs, it’s a sign that the driver is running out of memory.”

—Spark Tuning Guide

Why others are incorrect:

Amay help but does not directly address the driver memory shortage.

Bis not a valid action; GC cannot be disabled.

Dincreases memory usage, worsening the problem.