/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

1 vote351 views1 answer
Convert epoch to date in sql server
SQL
0 vote355 views1 answer
Delete first row in sql
SQL
11 votes330 views2 answers
Safe mode off mysql
SQL
0 vote311 views1 answer
Declaring variables in pl sql
SQL
1 vote344 views1 answer
Ubuntu mysql cannot connect to database server remote
SQL
0 vote348 views1 answer
Sql where part of string match
SQL
0 vote292 views1 answer
Mysqli or pdo
SQL
10 votes329 views2 answers
Sql rtrim
SQL
37 votes340 views3 answers
Duplicate records in sql
SQL
0 vote330 views1 answer
Check if table exists oracle
SQL