/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

3 votes330 views1 answer
Créer une table sql
SQL
0 vote437 views1 answer
Oracle db timestamp auto add
SQL
0 vote353 views1 answer
Google sheets data validation custom formula filter
SQL
0 vote365 views1 answer
How to inner join 4 tables in sql
SQL
1 vote327 views1 answer
DATE_SUB postgres
SQL
1 vote340 views3 answers
Rows, INSERT INTO
SQL
3 votes335 views3 answers
Order by postgres
SQL
8 votes336 views1 answer
Today minus 15 days postgresql
SQL
1 vote270 views1 answer
Unique vs distinct sql
SQL
5 votes378 views3 answers
Sql limit offset
SQL