This page shows you how to use different Apache Spark APIs with simple examples. Spark is a great engine for small and large datasets. It can be used with single-node/localhost environments, or distributed clusters.

Quickstart: Spark Connect. Launch Spark server with Spark Connect; Connect to Spark Connect server; Create DataFrame; Quickstart: Pandas API on Spark. Object Creation; Missing Data; Operations; Grouping; Plotting; Getting data in/out; Testing PySpark. Build a PySpark Application; Testing your PySpark Application; Putting It All Together!

PySpark is the Python API for Apache Spark. It enables you to perform real-time, large-scale data processing in a distributed environment using Python. It also provides a PySpark shell for interactively analyzing your data.

We will first introduce the API through Spark’s interactive shell (in Python or Scala), then show how to write applications in Java, Scala, and Python. To follow along with this guide, first, download a packaged release of Spark from the Spark website .

PySpark supports various UDFs and APIs to allow users to execute Python native functions. See also the latest Pandas UDFs and Pandas Function APIs. For instance, the example below allows users to directly use the APIs in a pandas Series within Python native function.

Find full example code at "examples/src/main/scala/org/apache/spark/examples/sql/SparkSQLExample.scala" in the Spark repo. The entry point into all functionality in Spark is the SparkSession class. To create a basic SparkSession , just use SparkSession.builder() :

PySpark is included in the official releases of Spark available in the Apache Spark website. For Python users, PySpark also provides pip installation from PyPI. This is usually for local usage or as a client to connect to a cluster instead of setting up a cluster itself.

See the Python examples and the Converter examples for examples of using Cassandra / HBase InputFormat and OutputFormat with custom converters. Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3 , etc. Spark supports text files, SequenceFiles ...

There are also basic programming guides covering multiple languages available in the Spark documentation, including these: Spark SQL, DataFrames and Datasets Guide. Structured Streaming Programming Guide. Machine Learning Library (MLlib) Guide

Running the Examples and Shell. Spark comes with several sample programs. Python, Scala, Java, and R examples are in the examples/src/main directory. To run Spark interactively in a Python interpreter, use bin/pyspark: