/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote309 views3 answers
Get count of null in column sql
SQL
0 vote311 views1 answer
Dump multiple tables mysql
SQL
0 vote329 views1 answer
Sqlites studio red exclamation mark when poening databse
SQL
0 vote310 views1 answer
Mysql insert into select with recursive
SQL
0 vote305 views1 answer
How to find top 3 salary in sql
SQL
1 vote356 views1 answer
Alter database datafile maxsize 32g
SQL
0 vote304 views1 answer
Inner joint
SQL
0 vote299 views1 answer
Mysql local password denied
SQL
3 votes318 views1 answer
Stpop start psql server
SQL
0 vote323 views1 answer
How to insert a uniqueidentifier in sql
SQL