Unit 3: Using RDD for Creating Applications in Spark and Graph Analytics - Practice Quiz

1 What does RDD stand for in Apache Spark?

2 Which property means that an RDD cannot be changed after it is created?

3 Which Spark method creates an RDD from a local collection?

4 Which method is commonly used to create an RDD by reading a text file?

5 Which RDD function applies a given function to every element?

6 Which RDD function keeps only the elements that satisfy a condition?

7 Which of the following is an RDD action?

8 What is lazy evaluation in Spark?

9 Which command normally starts the Spark shell for Scala?

10 Which command normally starts the interactive Spark shell for Python?

11 Which shared variable efficiently sends a read-only value to all worker nodes?

12 Which Spark shared variable is commonly used to add values across tasks?

13 What type of data is Spark GraphX designed to process?

14 What are the two basic components of a graph in GraphX?

15 Which Spark capability can GraphX use for distributed graph processing?

16 Which built-in GraphX algorithm ranks important vertices in a graph?

17 Which GraphX operation creates a new graph by changing vertex attributes?

18 Which GraphX operation produces a graph with the direction of every edge reversed?

19 What is the main purpose of feature extraction in machine learning?

20 Which Spark ML transformer converts a text document into a vector of token counts?

21 A Spark application applies several transformations to an RDD and then executes an action. Which RDD feature allows Spark to recompute only lost partitions instead of copying all data?

22 Why are RDDs particularly useful for iterative machine-learning algorithms?

23 Which expression creates an RDD from a local collection in PySpark?

24 A text file is stored in HDFS at /data/logs.txt. Which PySpark statement creates an RDD whose records normally represent lines from the file?

25 Given rdd = sc.parallelize([1, 2, 3, 4]), which expression returns [2, 4, 6, 8] as an RDD?

26 An RDD contains sentences, and the goal is to produce one record for every word. Which transformation is most appropriate?

27 Given pair RDD records such as ("A", 3) and ("A", 5), which transformation produces the total value for each key?

28 What is the result of calling rdd.count() on an RDD containing five records?

29 Why does rdd.map(lambda x: x * 2) usually not execute immediately?

30 Which method should be used when an RDD will be reused by multiple actions and should remain available in memory when possible?

31 A developer wants to inspect all elements of a small RDD on the driver. Which action is appropriate?

32 Which command starts the Scala-based Spark shell with the Spark context commonly available as sc?

33 Which command is normally used to start the Python version of the Spark interactive shell?

34 A large read-only lookup table is needed by tasks on many executors. Which shared variable avoids sending a separate copy with every task?

35 Which shared variable is appropriate for counting invalid records across tasks?

36 In GraphX, a graph is primarily represented as which combination of distributed datasets?

37 Which GraphX feature allows graph algorithms to combine graph structure with distributed data processing?

38 Which GraphX operation creates a new graph by retaining only vertices and edges that satisfy a predicate?

39 Which GraphX operation is designed to send messages along edges and aggregate messages at destination vertices?

40 A dataset contains the categorical values red, blue, and green. Which transformation is commonly used before applying a machine-learning algorithm that requires numeric input?

41 An RDD is persisted using MEMORY_ONLY. After several partitions are cached, an executor holding one cached partition fails. The original data is still available in HDFS. What occurs when an action next requires the lost partition?

42 A pair RDD already has a HashPartitioner named p. Which transformation sequence preserves that partitioner, allowing a subsequent reduceByKey(p, f) to avoid an unnecessary reshuffle?

43 A directory contains thousands of small UTF-8 files. An application must create an RDD in which each record contains one file's path and its complete contents, rather than one record per line. Which creation method directly provides this representation?

44 Two ordinary RDDs without partitioners are created as a = sc.parallelize(dataA, 3) and b = sc.parallelize(dataB, 5). What is normally true of a.union(b) before any later repartitioning?

45 An RDD has exactly two partitions containing [1, 2] and [3, 4]. What does rdd.aggregate(10)((x, y) => x + y, (x, y) => x + y) return?

46 Suppose empty is an RDD with no elements but several empty partitions. Which statement correctly distinguishes empty.fold(0)(_ + _) from empty.reduce(_ + _)?

47 Two RDDs have the same total element count and the same number of partitions, but corresponding partitions contain different numbers of elements. What happens when left.zip(right) is evaluated?

48 An RDD currently has four partitions. The application executes rdd.coalesce(8, shuffle = false) and, separately, rdd.repartition(8). What partition counts should be expected?

49 A developer launches spark-shell --master local[2]. Assuming tasks are otherwise runnable and each task uses one CPU, what does the master setting primarily permit?

50 A Spark shell requires an GB driver heap. Which approach configures this reliably?

51 An accumulator is incremented inside a lazy map transformation, and the transformed RDD is evaluated by two separate actions without persistence. Why is the accumulator unsuitable for reporting an exact number of input records?

52 A large lookup table is distributed using a broadcast variable. The driver later creates a substantially updated version that future tasks must use. What is the appropriate design?

53 A GraphX graph is constructed with a vertex RDD containing only vertex 1, an edge from vertex 1 to vertex 2, and "unknown" as the default vertex attribute. How is vertex 2 represented?

54 Which information is available directly from each element of a GraphX graph's triplets view?

55 A graph contains many parallel edges in both directions between the same pairs of vertices. Which GraphX partition strategy hashes the canonicalized endpoint pair so that direction-reversed edges for a pair are assigned consistently?

56 A GraphX subgraph operation receives both a vertex predicate and an edge-triplet predicate. Which statement describes the resulting graph?

57 A graph has weighted edges (1,2,3), (3,2,5), and (2,3,7), where each tuple is (source, destination, weight). aggregateMessages sends each edge's weight only to its destination and merges messages using addition. Which result is produced?

58 An auxiliary RDD[(VertexId, U)] has data for only some graph vertices. Which operation invokes an update function for every graph vertex while representing a missing auxiliary value with None?

59 A text pipeline applies HashingTF with a small numFeatures value and then fits IDF. Two frequent terms collide into the same hash index. What is the consequence?

60 A dataset contains million-dimensional vectors with only a few nonzero entries per row. What is the principal risk of applying StandardScaler with withMean = true?