/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

1 vote345 views1 answer
Postgresql can you have null foreign key
SQL
0 vote258 views1 answer
Get created time of database psql 8
SQL
0 vote262 views1 answer
Can we rollback deleted data in sql
SQL
2 votes274 views1 answer
Drop all database tables oracle sql developer
SQL
2 votes276 views2 answers
Mysql alter column default
SQL
0 vote449 views2 answers
Create new table from existing table with data in sql server
SQL
0 vote283 views1 answer
Count database access django
SQL
0 vote268 views2 answers
Python mysql create table if not exists
SQL
0 vote293 views1 answer
Sqlcl format
SQL
11 votes290 views6 answers
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
SQL