/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote259 views1 answer
Mysql export search results
SQL
1 vote266 views1 answer
Sql first
SQL
1 vote257 views1 answer
Drop df constraint sql server
SQL
1 vote268 views1 answer
Sql where max date
SQL
1 vote238 views1 answer
Clear screen command in psql
SQL
4 votes271 views1 answer
W3schools create table
SQL
0 vote245 views1 answer
Mysql delete from where like
SQL
0 vote264 views1 answer
Sql alter table add column if exists
SQL
6 votes244 views1 answer
How to count number of rows in sql
SQL
1 vote260 views1 answer
Mysql copy table1 to table2
SQL