/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote378 views2 answers
Sql stored procedure for access frontend
SQL
1 vote331 views1 answer
Input in mysql
SQL
1 vote389 views1 answer
Psql filed name alter
SQL
2 votes484 views1 answer
Ssh mysql port forwarding
SQL
1 vote362 views1 answer
Right outer vs left outer join
SQL
0 vote355 views2 answers
Call to undefined function mysqli_fetch_all()
SQL
0 vote352 views1 answer
How to use lower case in mysql
SQL
0 vote328 views1 answer
Get duplicate entry sql
SQL
1 vote352 views1 answer
Oracle list data files
SQL
1 vote307 views1 answer
Strict in postgres SQL
SQL