How To Pass Databricks Certified Associate Developer for Apache Spark 3.0 Exam Easily?

test2022-01-25T03:13:56+00:00

By test Databricks Certified Associate Developer for Apache Spark 3.0 0 Comments

Databricks Certified Associate Developer for Apache Spark 3.0 certification exam assesses an understanding of the basics of the Spark architecture and the ability to apply the Spark DataFrame API to complete individual data manipulation tasks. FreeTestShare provides 100% real and reliable Databricks Certified Associate Developer for Apache Spark 3.0 questions and answers files to help you pass in the first try.

This Databricks Certified Associate Developer for Apache Spark 3.0 free demo aims to make you well-prepared. Test now!

Page 1 of 5

1. Which of the following code blocks can be used to save DataFrame transactionsDf to memory only, recalculating partitions that do not fit in memory when they are needed?

from pyspark import StorageLevel transactionsDf.cache(StorageLevel.MEMORY_ONLY)

transactionsDf.cache()

transactionsDf.storage_level('MEMORY_ONLY')

transactionsDf.persist()

transactionsDf.clear_persist()

from pyspark import StorageLevel transactionsDf.persist(StorageLevel.MEMORY_ONLY)

2. The code block displayed below contains an error. The code block is intended to return all columns of DataFrame transactionsDf except for columns predError, productId, and value. Find the error.

Excerpt of DataFrame transactionsDf:

transactionsDf.select(~col("predError"), ~col("productId"), ~col("value"))

The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value wrapped in the col operator so they should be expressed like drop(col(predError), col(productId), col(value)).

The select operator should be replaced with the deselect operator.

The column names in the select operator should not be strings and wrapped in the col operator, so they should be expressed like select(~col(predError), ~col(productId), ~col(value)).

The select operator should be replaced by the drop operator.

The select operator should be replaced by the drop operator and the arguments to the drop operator should be column names predError, productId and value as strings. (Correct)

3. Which of the following code blocks sorts DataFrame transactionsDf both by column storeId in ascending and by column productId in descending order, in this priority?

transactionsDf.sort("storeId", asc("productId"))

transactionsDf.sort(col(storeId)).desc(col(productId))

transactionsDf.order_by(col(storeId), desc(col(productId)))

transactionsDf.sort("storeId", desc("productId"))

transactionsDf.sort("storeId").sort(desc("productId"))

4. The code block displayed below contains an error. The code block should return a new DataFrame that only contains rows from DataFrame transactionsDf in which the value in column predError is at least 5. Find the error.

Code block:

transactionsDf.where("col(predError) >= 5")

The argument to the where method should be "predError >= 5".

Instead of where(), filter() should be used.

The expression returns the original DataFrame transactionsDf and not a new DataFrame. To avoid this, the code block should be transactionsDf.toNewDataFrame().where("col(predError) >= 5").

The argument to the where method cannot be a string.

Instead of >=, the SQL operator GEQ should be used.

5. Which of the following statements about RDDs is incorrect?

An RDD consists of a single partition.

The high-level DataFrame API is built on top of the low-level RDD AP

RDDs are immutable.

RDD stands for Resilient Distributed Dataset.

RDDs are great for precisely instructing Spark on how to do a query.

6. .sum(col('value'))

4, 1, 2

3, 1, 6

3, 1, 2

3, 5, 2

4, 6

7. Which of the following code blocks generally causes a great amount of network traffic?

DataFrame.select()

DataFrame.coalesce()

DataFrame.collect()

DataFrame.rdd.map()

DataFrame.count()

8. Which of the following statements about garbage collection in Spark is incorrect?

Garbage collection information can be accessed in the Spark UI's stage detail view.

Optimizing garbage collection performance in Spark may limit caching ability.

Manually persisting RDDs in Spark prevents them from being garbage collected.

In Spark, using the G1 garbage collector is an alternative to using the default Parallel garbage collector.

Serialized caching is a strategy to increase the performance of garbage collection.

9. +-------------+---------+-----+-------+---------+----+

transactionsDf.max('value').min('value')

transactionsDf.agg(max('value').alias('highest'), min('value').alias('lowest'))

transactionsDf.groupby(col(productId)).agg(max(col(value)).alias("highest"), min(col(value)).alias("lowest"))

transactionsDf.groupby('productId').agg(max('value').alias('highest'), min('value').alias('lowest'))

transactionsDf.groupby("productId").agg({"highest": max("value"), "lowest": min("value")})

10. +-------+-------------+---------+-----+

Code block:

sc.union([transactionsDfMonday, transactionsDfTuesday])

The DataFrames' RDDs need to be passed into the sc.union method instead of the DataFrame variable names.

Instead of union, the concat method should be used, making sure to not use its default arguments.

Instead of the Spark context, transactionDfMonday should be called with the join method instead of the union method, making sure to use its default arguments.

Instead of the Spark context, transactionDfMonday should be called with the union method.

Instead of the Spark context, transactionDfMonday should be called with the unionByName method instead of the union method, making sure to not use its default arguments.

Page 2 of 5

11. Which of the following statements about lazy evaluation is incorrect?

Predicate pushdown is a feature resulting from lazy evaluation.

Execution is triggered by transformations.

Spark will fail a job only during execution, but not during definition.

Accumulators do not change the lazy evaluation model of Spark.

Lineages allow Spark to coalesce transformations into stages

12. Which of the following describes slots?

Slots are dynamically created and destroyed in accordance with an executor's workload.

To optimize I/O performance, Spark stores data on disk in multiple slots.

A Java Virtual Machine (JVM) working as an executor can be considered as a pool of slots for task execution.

A slot is always limited to a single core. Slots are the communication interface for executors and are used for receiving commands and sending results to the driver.

13. "attributes"

14. Which of the following describes the difference between client and cluster execution modes?

In cluster mode, the driver runs on the worker nodes, while the client mode runs the driver on the client machine.

In cluster mode, the driver runs on the edge node, while the client mode runs the driver in a worker node.

In cluster mode, each node will launch its own executor, while in client mode, executors will exclusively run on the client machine.

In client mode, the cluster manager runs on the same host as the driver, while in cluster mode, the cluster manager runs on a separate node.

In cluster mode, the driver runs on the master node, while in client mode, the driver runs on a virtual machine in the cloud.

15. Which of the following code blocks returns a copy of DataFrame transactionsDf where the column storeId has been converted to string type?

transactionsDf.withColumn("storeId", convert("storeId", "string"))

transactionsDf.withColumn("storeId", col("storeId", "string"))

transactionsDf.withColumn("storeId", col("storeId").convert("string"))

transactionsDf.withColumn("storeId", col("storeId").cast("string"))

transactionsDf.withColumn("storeId", convert("storeId").as("string"))

16. Which of the following statements about broadcast variables is correct?

Broadcast variables are serialized with every single task.

Broadcast variables are commonly used for tables that do not fit into memory.

Broadcast variables are immutable.

Broadcast variables are occasionally dynamically updated on a per-task basis.

Broadcast variables are local to the worker node and not shared across the cluster.

17. +------------- +--------- +----- +------- +--------- +---- +

transactionsDf.schema.print()

transactionsDf.rdd.printSchema()

transactionsDf.rdd.formatSchema()

transactionsDf.printSchema()

print(transactionsDf.schema)

18. Which of the following describes Spark's way of managing memory?

Spark uses a subset of the reserved system memory.

Storage memory is used for caching partitions derived from DataFrames.

As a general rule for garbage collection, Spark performs better on many small objects than few big objects.

Disabling serialization potentially greatly reduces the memory footprint of a Spark application.

Spark's memory usage can be divided into three categories: Execution, transaction, and storage.

19. Which of the following code blocks returns about 150 randomly selected rows from the 1000-row DataFrame transactionsDf, assuming that any row can appear more than once in the returned DataFrame?

transactionsDf.resample(0.15, False, 3142)

transactionsDf.sample(0.15, False, 3142)

transactionsDf.sample(0.15)

transactionsDf.sample(0.85, 8429)

transactionsDf.sample(True, 0.15, 8261)

20. Which of the following describes properties of a shuffle?

Operations involving shuffles are never evaluated lazily.

Shuffles involve only single partitions.

Shuffles belong to a class known as "full transformations".

A shuffle is one of many actions in Spark.

In a shuffle, Spark writes data to disk.

Page 3 of 5

21. Which of the elements in the labeled panels represent the operation performed for broadcast variables?

Larger image

2, 5

2, 3

1, 2

1, 3, 4

22. transactionsDf.withColumnRenamed("predErrorAdded", add_2_if_geq_3_udf(col("predError")))

The operator used to adding the column does not add column predErrorAdded to the DataFrame.

Instead of col("predError"), the actual DataFrame with the column needs to be passed, like so transactionsDf.predError.

The udf() method does not declare a return type.

UDFs are only available through the SQL API, but not in the Python API as shown in the code block.

The Python function is unable to handle null values, resulting in the code block crashing on execution.

23. Which of the following DataFrame operators is never classified as a wide transformation?

DataFrame.sort()

DataFrame.aggregate()

DataFrame.repartition()

DataFrame.select()

DataFrame.join()

24. +------+----------------------------------+-------------------+

Code block:

itemsDf.withColumnRenamed("itemNameElements", split("itemName"))

itemsDf.withColumnRenamed("itemNameElements", split("itemName"))

All column names need to be wrapped in the col() operator.

Operator withColumnRenamed needs to be replaced with operator withColumn and a second argument "," needs to be passed to the split method.

Operator withColumnRenamed needs to be replaced with operator withColumn and the split method needs to be replaced by the splitString method.

Operator withColumnRenamed needs to be replaced with operator withColumn and a second argument " " needs to be passed to the split method.

The expressions "itemNameElements" and split("itemName") need to be swapped.

25. itemsDf.withColumnRenamed("supplier", "feature1")

C. itemsDf.withColumnRenamed(col("attributes"), col("feature0"), col("supplier"), col("feature1"))

D. itemsDf.withColumnRenamed("attributes", "feature0").withColumnRenamed("supplier", "feature1")

E. itemsDf.withColumn("attributes", "feature0").withColumn("supplier", "feature1")

26. itemsDf = spark.read(schema=itemsDfSchema).parquet(filePath)

27. part-00003-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-301-1-

c000.csv.gz

spark.option("header",True).csv(filePath)

spark.read.format("csv").option("header",True).option("compression","zip").load(filePath)

spark.read().option("header",True).load(filePath)

spark.read.format("csv").option("header",True).load(filePath)

spark.read.load(filePath)

28. Which of the following code blocks returns a single-column DataFrame of all entries in Python list throughputRates which contains only float-type values?

spark.createDataFrame((throughputRates), FloatType)

spark.createDataFrame(throughputRates, FloatType)

spark.DataFrame(throughputRates, FloatType)

spark.createDataFrame(throughputRates)

spark.createDataFrame(throughputRates, FloatType())

29. Which of the following code blocks immediately removes the previously cached DataFrame transactionsDf from memory and disk?

array_remove(transactionsDf, "*")

transactionsDf.unpersist() (Correct)

del transactionsDf

transactionsDf.clearCache()

transactionsDf.persist()

30. Which of the following statements about data skew is incorrect?

Spark will not automatically optimize skew joins by default.

Broadcast joins are a viable way to increase join performance for skewed data over sort-merge joins.

In skewed DataFrames, the largest and the smallest partition consume very different amounts of memory.

To mitigate skew, Spark automatically disregards null values in keys when joining.

Salting can resolve data skew.

Page 4 of 5

31. The code block displayed below contains an error. The code block should save DataFrame transactionsDf at path path as a parquet file, appending to any existing parquet file. Find the error.

Code block:

transactionsDf.format("parquet").option("mode", "append").save(path)

The code block is missing a reference to the DataFrameWriter.

save() is evaluated lazily and needs to be followed by an action.

The mode option should be omitted so that the command uses the default mode.

The code block is missing a bucketBy command that takes care of partitions.

Given that the DataFrame should be saved as parquet file, path is being passed to the wrong method.

32. The code block displayed below contains an error. The code block should return a DataFrame where all entries in column supplier contain the letter combination et in this order. Find the error.

Code block:

itemsDf.filter(Column('supplier').isin('et'))

The Column operator should be replaced by the col operator and instead of isin, contains should be used.

The expression inside the filter parenthesis is malformed and should be replaced by isin('et', 'supplier').

Instead of isin, it should be checked whether column supplier contains the letters et, so isin should be replaced with contains. In addition, the column should be accessed using col['supplier'].

The expression only returns a single column and filter should be replaced by select.

33. csv

34. Which of the following code blocks returns a DataFrame that is an inner join of DataFrame itemsDf and DataFrame transactionsDf, on columns itemId and productId, respectively and in which every itemId just appears once?

itemsDf.join(transactionsDf, "itemsDf.itemId==transactionsDf.productId").distinct("itemId")

itemsDf.join(transactionsDf, itemsDf.itemId==transactionsDf.productId).dropDuplicates(["itemId"])

itemsDf.join(transactionsDf, itemsDf.itemId==transactionsDf.productId).dropDuplicates("itemId")

itemsDf.join(transactionsDf, itemsDf.itemId==transactionsDf.productId, how="inner").distinct(["itemId"])

itemsDf.join(transactionsDf, "itemsDf.itemId==transactionsDf.productId", how="inner").dropDuplicates(["itemId"])

35. 3;23;12;green grass

Code block:

transactionsDf = spark.read.load("data/transactions.csv", sep=";", format="csv", header=True)

The DataFrameReader is not accessed correctly.

The transaction is evaluated lazily, so no file will be read.

Spark is unable to understand the file type.

The code block is unable to capture all columns.

The resulting DataFrame will not have the appropriate schema.

36. Which of the following is not a feature of Adaptive Query Execution?

Replace a sort merge join with a broadcast join, where appropriate.

Coalesce partitions to accelerate data processing.

Split skewed partitions into smaller partitions to avoid differences in partition processing time.

Reroute a query in case of an executor failure.

Collect runtime statistics during query execution.

37. alias

38. Which of the following code blocks returns a copy of DataFrame itemsDf where the column supplier has been renamed to manufacturer?

itemsDf.withColumn(["supplier", "manufacturer"])

itemsDf.withColumn("supplier").alias("manufacturer")

itemsDf.withColumnRenamed("supplier", "manufacturer")

itemsDf.withColumnRenamed(col("manufacturer"), col("supplier"))

itemsDf.withColumnsRenamed("supplier", "manufacturer")

39. The code block displayed below contains an error. The code block should return the average of rows in column value grouped by unique storeId. Find the error.

Code block:

transactionsDf.agg("storeId").avg("value")

Instead of avg("value"), avg(col("value")) should be used.

The avg("value") should be specified as a second argument to agg() instead of being appended to it.

All column names should be wrapped in col() operators.

agg should be replaced by groupBy.

"storeId" and "value" should be swapped.

40. +-------------+---------+-----+-------+---------+----+

transactionsDf.drop(col("value"), col("predError"))

transactionsDf.drop("predError", "value")

transactionsDf.drop(value, predError)

transactionsDf.drop(["predError", "value"])

transactionsDf.drop([col("predError"), col("value")])

Page 5 of 5

41. col("productId")==2

42. print(itemsDf.types)

B. itemsDf.printSchema()

C. spark.schema(itemsDf)

D. itemsDf.rdd.printSchema()

E. itemsDf.print.schema()

43. Which of the following describes characteristics of the Dataset API?

The Dataset API does not support unstructured data.

In Python, the Dataset API mainly resembles Pandas' DataFrame AP

In Python, the Dataset API's schema is constructed via type hints.

The Dataset API is available in Scala, but it is not available in Python.

The Dataset API does not provide compile-time type safety.

44. "MM d (EEE)"

45. The code block displayed below contains an error. The code block should display the schema of DataFrame transactionsDf. Find the error.

Code block:

transactionsDf.rdd.printSchema

There is no way to print a schema directly in Spark, since the schema can be printed easily through using print(transactionsDf.columns), so that should be used instead.

The code block should be wrapped into a print() operation.

printSchema is only accessible through the spark session, so the code block should be rewritten as spark.printSchema(transactionsDf).

printSchema is a method and should be written as printSchema(). It is also not callable through transactionsDf.rdd, but should be called directly from transactionsDf. (Correct)

printSchema is a not a method of transactionsDf.rdd. Instead, the schema should be printed via transactionsDf.print_schema().

46. parquet