Igor MiG-Kharkov

MiG-Kharkov / Spark Dataframe Cheat Sheet.py

Created February 4, 2018 20:57 — forked from evenv/Spark Dataframe Cheat Sheet.py

Cheat sheet for Spark Dataframes (using Python)

	# A simple cheat sheet of Spark Dataframe syntax
	# Current for Spark 1.6.1

	# import statements
	from pyspark.sql import SQLContext
	from pyspark.sql.types import *
	from pyspark.sql.functions import *

	#creating dataframes
	df = sqlContext.createDataFrame([(1, 4), (2, 5), (3, 6)], ["A", "B"]) # from manual data