/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote268 views1 answer
Sub query in linq
SQL
0 vote271 views1 answer
Check psql validity function
SQL
0 vote294 views1 answer
Sql server user permissions and roles
SQL
39 votes333 views5 answers
Sql drop column
SQL
0 vote255 views1 answer
Generate sequence number in sql server
SQL
2 votes286 views2 answers
Run mysql command from bash
SQL
1 vote293 views1 answer
SQL query to get all table records count from a database
SQL
1 vote304 views1 answer
Dropping a constraint in ms sql
SQL
0 vote291 views1 answer
Mysql vs sql server performance benchmark
SQL
0 vote300 views1 answer
One table with another
SQL