Spark to pandas

Convert pandas dataframe to spark dataframe

import pandas as pd
from pyspark.sql import SparkSession

filename = <'path to file'>
spark = SparkSession.build.appName('pandasToSpark').getOrCreate()
# Assuming file is csv
pandas_df = pd.read_csv(filename)
spark_df = spark.CreateDataFrame(pandas_df)

Dataframe pandas to spark

from pyspark.sql import SparkSession
#Create PySpark SparkSession
spark = SparkSession.builder \
    .master("local[1]") \
    .appName("SparkByExamples.com") \
    .getOrCreate()
#Create PySpark DataFrame from Pandas
sparkDF=spark.createDataFrame(pandasDF) 
sparkDF.printSchema()
sparkDF.show()

#Outputs below schema & DataFrame

root
 |-- Name: string (nullable = true)
 |-- Age: long (nullable = true)

+------+---+
|  Name|Age|
+------+---+
| Scott| 50|
|  Jeff| 45|
|Thomas| 54|
|   Ann| 34|
+------+---+

Source: sparkbyexamples.com

Create spark dataframe from pandas

import numpy as np
import pandas as pd

# Enable Arrow-based columnar data transfers
spark.conf.set("spark.sql.execution.arrow.enabled", "true")

# Generate a pandas DataFrame
pdf = pd.DataFrame(np.random.rand(100, 3))

# Create a Spark DataFrame from a pandas DataFrame using Arrow
df = spark.createDataFrame(pdf)

Similar codes

0 vote302 views2 answers
How to create an empty 2d list in python
Python
93 votes362 views4 answers
Python 3.7 download
Python
0 vote330 views2 answers
The module in NAME could not be imported: django.contrhtmlib.auth.password_validation.UserAttributeSimilarityValidator. Check your AUTH_PASSWORD_VALIDATORS setting.
Python
16 votes319 views6 answers
How to take integer input in python
Python
1 vote338 views1 answer
Truncate date to midnight in pandas column
Python
0 vote303 views1 answer
Find and replace subword in word python regex
Python
0 vote319 views1 answer
How to add textbox in pygame window
Python
0 vote272 views1 answer
Create_polygon tkinter
Python
0 vote310 views1 answer
Why am i not able to import wtf flask
Python
17 votes304 views2 answers
Count lines in file python
Python