/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

1 vote209 views1 answer
Mysqli connect
SQL
1 vote228 views1 answer
Sql padd let with zeros
SQL
11 votes216 views2 answers
Mysql disable safe mode
SQL
0 vote233 views1 answer
Update All tables COLLATE DATABASE_DEFAULT
SQL
0 vote239 views1 answer
How do you insert boolean to postgresql
SQL
13 votes226 views6 answers
Mysql update from select
SQL
0 vote227 views1 answer
Postgresql not in alternative
SQL
0 vote172 views1 answer
Mysql collation for all languages
SQL
3 votes233 views2 answers
MySQL query to get column names
SQL
0 vote248 views1 answer
Oracle drop type
SQL