DASD

Free DASD practice questions

10 questions from our DASD practice tests, each with the correct answer and a written explanation. Read them with the answers shown, or switch to quiz mode and answer first.

Back to the DASD practice exams

Question 1 of 10, Apache Spark Architecture and Components

In a Spark cluster, what is an executor?

  1. AA physical machine in the cluster that the cluster manager reserves for a single Spark job
  2. BA thread inside the driver that converts each stage into bytecode before it is shipped
  3. CA daemon shared by all applications on a node that keeps shuffle files after jobs finish
  4. DA per-application JVM process on a worker node that runs tasks and caches data Correct answer

Answer and explanation: Executors are processes launched for a specific application on worker nodes; they run tasks in their cores (slots) and hold cached data and shuffle output. Code generation happens in the driver's planning phase, not in a dedicated executor thread. A shared node daemon that serves shuffle files is the external shuffle service, not an executor, and a worker node can host executors of several applications, so an executor is not a reserved machine.

Question 2 of 10, Apache Spark Architecture and Components

Which statement correctly describes lazy evaluation in Apache Spark?

  1. ALazy evaluation only applies to RDDs, because DataFrame transformations are executed eagerly by the Catalyst optimizer
  2. BSpark delays reading the source files until the first transformation, then materializes each intermediate DataFrame
  3. CActions are queued and executed in the background, while transformations run immediately to validate the data
  4. DTransformations only build up a logical plan; nothing executes until an action such as count() or write is called Correct answer

Answer and explanation: Transformations such as select, filter and join return a new DataFrame describing the computation; Spark executes the plan only when an action requests a result. Actions run synchronously and trigger execution rather than being queued. Spark does not materialize intermediate DataFrames after each transformation, and DataFrames are lazy too, which is exactly what allows Catalyst to optimize the whole plan before running it.

Question 3 of 10, Using Spark SQL

A developer needs a persistent table where data is hashed into 16 buckets by user_id and sorted within each bucket by event_ts, to speed up later joins on user_id. Which code is valid?

  1. Adf.write.bucketBy(16, "user_id").sortBy("event_ts").saveAsTable("events_b") Correct answer
  2. Bdf.write.bucketBy(16, "user_id").sortBy("event_ts").parquet("/data/events_b")
  3. Cdf.write.bucketBy("user_id", 16).orderBy("event_ts").saveAsTable("events_b")
  4. Ddf.write.sortBy("event_ts").partitionBy("user_id").saveAsTable("events_b")

Answer and explanation: Bucketing requires the number of buckets first, then the columns, and it only works with saveAsTable because bucket metadata is stored in the metastore; sortBy is valid only together with bucketBy. Writing bucketed output with parquet(path) raises an error. sortBy without bucketBy is not allowed, and partitioning by a high-cardinality user_id would create far too many directories. The writer has no orderBy method and the arguments of bucketBy are reversed.

Question 4 of 10, Using Spark SQL

A developer reads a PostgreSQL table with the JDBC source. The read is slow because only one task pulls all 80 million rows. Which option set lets Spark read the table in parallel?

  1. ApartitionColumn, lowerBound, upperBound and numPartitions on a numeric or date column Correct answer
  2. BnumPartitions on its own, since Spark then splits the table automatically by the primary key
  3. CpushDownPredicate and queryTimeout so the database splits the query across several cursors
  4. Dfetchsize and batchsize set high enough so that each executor opens its own connection

Answer and explanation: Parallel JDBC reads require partitionColumn together with lowerBound, upperBound and numPartitions; Spark generates one range query per partition. fetchsize controls rows per round trip and batchsize affects writes, neither adds parallelism. numPartitions alone does not split a read without a partition column, and pushDownPredicate/queryTimeout control filter pushdown and timeouts, not parallelism.

Question 5 of 10, Developing Apache Spark DataFrame/Dataset API Applications

Two DataFrames, orders (500 million rows) and countries (250 rows), are joined on country_code. The developer wants to make sure the large table is not shuffled. Which code achieves this?

  1. Abroadcast(orders).join(countries, "country_code")
  2. Borders.join(countries.repartition(1), "country_code")
  3. Corders.join(broadcast(countries), "country_code") Correct answer
  4. Dorders.join(countries, "country_code").coalesce(1)

Answer and explanation: Wrapping the small DataFrame in broadcast() ships a copy of it to every executor, so each partition of orders is joined locally with no shuffle of the large side. Broadcasting the large table would send 500 million rows to every executor. Repartitioning the small side to one partition does not stop a sort-merge join from shuffling orders, and coalescing after the join changes only the output partitioning.

Question 6 of 10, Developing Apache Spark DataFrame/Dataset API Applications

A pandas UDF computes a value from two numeric columns. Which definition is correct for Spark 3.x/4.x?

  1. A@pandas_udf("double", PandasUDFType.SCALAR_ITER) def ratio(a, b): return a.values / b.values
  2. B@pandas_udf("double") def ratio(a: pd.Series, b: pd.Series) -> pd.Series: return a / b Correct answer
  3. C@udf("double") def ratio(a: pd.Series, b: pd.Series) -> pd.Series: return a / b
  4. D@pandas_udf def ratio(a: float, b: float) -> float: return a / b

Answer and explanation: Modern pandas UDFs are declared with @pandas_udf(returnType) and Python type hints; pd.Series inputs and output define a Series-to-Series UDF. PandasUDFType is the deprecated pre-3.0 style, and SCALAR_ITER would expect iterator input. @udf creates a row-at-a-time Python UDF that receives scalars, not Series. Scalar float hints are not a valid pandas UDF signature and no return type is given.

Question 7 of 10, Developing Apache Spark DataFrame/Dataset API Applications

A UDF that normalizes phone numbers must be used in DataFrame code and in SQL queries. Which code registers it for both uses?

  1. Anorm = udf(normalize, StringType()).alias("norm_phone")
  2. Bspark.catalog.createFunction("norm_phone", normalize)
  3. Cnorm = udf(normalize, StringType()); spark.sql("CREATE FUNCTION norm_phone AS norm")
  4. Dnorm = spark.udf.register("norm_phone", normalize, StringType()) Correct answer

Answer and explanation: spark.udf.register registers the function for SQL under the given name and returns a UDF object that can also be used in DataFrame expressions. A udf() object is not visible to SQL, and CREATE FUNCTION expects a class or a SQL body rather than a Python variable. Aliasing a UDF names an output column, and the catalog API has no createFunction method for registering Python functions.

Question 8 of 10, Troubleshooting and Tuning Apache Spark DataFrame API Applications. Choose 2.

Which TWO optimizations does Adaptive Query Execution (AQE) perform at runtime using shuffle statistics? (Choose two.)

  1. AChoosing the number of executors for the next application
  2. BRewriting Python UDFs into native JVM expressions
  3. CSplitting skewed partitions in a sort-merge join into smaller tasks Correct answer
  4. DCollecting table statistics before the query starts running
  5. ECoalescing many small shuffle partitions into fewer, larger ones Correct answer

Answer and explanation: AQE re-optimizes the plan at stage boundaries using real shuffle statistics: it coalesces small shuffle partitions, splits skewed join partitions, and can switch a sort-merge join to a broadcast join. It never rewrites Python UDFs, it does not size executors (that is dynamic allocation or the cluster configuration), and it relies on runtime shuffle statistics rather than collecting table statistics in advance like ANALYZE TABLE.

Question 9 of 10, Structured Streaming

A stream may deliver the same order event several times within a few minutes. The query must drop duplicates by order_id while keeping state bounded. Which code is the best fit in Spark 3.5 or later?

  1. Adf.distinct() applied after the sink so the files written are de-duplicated afterwards
  2. Bdf.dropDuplicates(["order_id"]) without any watermark, so every order_id is kept in state
  3. Cdf.dropDuplicates(["order_id", "event_time"]) without a watermark on event_time
  4. Ddf.withWatermark("event_time", "10 minutes").dropDuplicatesWithinWatermark(["order_id"]) Correct answer

Answer and explanation: dropDuplicatesWithinWatermark (Spark 3.5+) removes duplicates by key that arrive within the watermark delay, and state older than the watermark is purged, so state stays bounded even when duplicates have different timestamps. dropDuplicates without a watermark keeps every key forever. distinct cannot be applied to data after it is written by a sink, and including event_time without a watermark both misses duplicates with different timestamps and grows state without limit.

Question 10 of 10, Using Spark Connect to Deploy Applications

Which deployment mode is MOST suitable for unit testing PySpark code on a CI runner without a cluster?

  1. ALocal mode, for example master local[2] Correct answer
  2. BClient mode against a YARN cluster
  3. CCluster mode on a Kubernetes cluster
  4. DStandalone cluster with high availability

Answer and explanation: Local mode runs the driver and executor in one process on the runner, which is ideal for tests with small data and no external dependencies. Cluster and client modes require a running cluster manager and cluster resources, and a highly available standalone cluster adds infrastructure that a unit test does not need.