/Python/Create spark dataframe from pandas

Create spark dataframe from pandas

Create spark dataframe in python

spark.createDataFrame(["10","11","13"], "string").toDF("age")

Source: stackoverflow.com

Dataframe pandas to spark

from pyspark.sql import SparkSession
#Create PySpark SparkSession
spark = SparkSession.builder \
    .master("local[1]") \
    .appName("SparkByExamples.com") \
    .getOrCreate()
#Create PySpark DataFrame from Pandas
sparkDF=spark.createDataFrame(pandasDF) 
sparkDF.printSchema()
sparkDF.show()

#Outputs below schema & DataFrame

root
 |-- Name: string (nullable = true)
 |-- Age: long (nullable = true)

+------+---+
|  Name|Age|
+------+---+
| Scott| 50|
|  Jeff| 45|
|Thomas| 54|
|   Ann| 34|
+------+---+

Source: sparkbyexamples.com

Create spark dataframe from pandas

import numpy as np
import pandas as pd

# Enable Arrow-based columnar data transfers
spark.conf.set("spark.sql.execution.arrow.enabled", "true")

# Generate a pandas DataFrame
pdf = pd.DataFrame(np.random.rand(100, 3))

# Create a Spark DataFrame from a pandas DataFrame using Arrow
df = spark.createDataFrame(pdf)

Similar codes

47 votes108 views6 answers
Today date python
Python
0 vote101 views1 answer
Django dump previous migrations
Python
0 vote107 views1 answer
How to export schema in graphene django
Python
2 votes130 views1 answer
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 1968: invalid start byte
Python
0 vote142 views1 answer
Error: The file/path provided (flaskr) does not appear to exist. Please verify the path is correct. If app is not on PYTHONPATH, ensure the extension is .py
Python Flask
9 votes112 views1 answer
Save an image in python as grayscale cv2
Python
1 vote70 views2 answers
'str' object has no attribute 'strftime'
Python
8 votes112 views3 answers
Get_dummies
Python
0 vote75 views1 answer
Get parameter value dynamo python
Python
0 vote151 views1 answer
Django today date in template
Python Django