Learn AI

AI Deep Dive

Tag: Spark

  • Apache Spark

    Let’s illustrate Apache Spark with a classic “word count” example using PySpark (the Python API for Spark). This example demonstrates the fundamental concepts of distributed data processing with Spark. Scenario: You have a large text file (or multiple files) and you want to count the occurrences of each unique word in the file(s). Steps: from Read more

  • Inner workings of Apache Spark

    Here’s a breakdown of key internal aspects of the inner workings of Apache Spark. : 1. Architecture: 2. Execution Model: 3. Data Partitioning: 4. Shuffle Operations: 5. Memory Management: In essence, Spark’s internal workings involve: Understanding these internal mechanisms is key to writing efficient and scalable Spark applications. Read more