/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

4 votes206 views1 answer
Oracle error compilation line
SQL
3 votes210 views3 answers
How to add month in update sql
SQL
25 votes256 views4 answers
Import mysql dump command line
SQL
0 vote232 views1 answer
Oracle sql how to overcome 999 limit for IN
SQL
-1 vote196 views1 answer
How to format tables in sqlplus
SQL
1 vote213 views1 answer
Mysql return if it contains
SQL
11 votes219 views4 answers
Round in sql
SQL
0 vote194 views1 answer
Call to undefined function mysql_query() in D:\xampp\htdocs\image\index.php:11 Stack trace: #0 {main} thrown in site:stackoverflow.com
SQL
0 vote215 views1 answer
Mysql could not fetch tables
SQL
5 votes214 views3 answers
String concatenation sql server
SQL