/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote362 views1 answer
How to declare variable date in mysql
SQL
0 vote344 views1 answer
Bus source and destination equal to destination and source of another by sql query
SQL
0 vote371 views1 answer
Mysql db dump restore max file size issue
SQL
0 vote388 views1 answer
Add user mysql wordpress
SQL
0 vote326 views0 answer
SQL
7 votes401 views3 answers
Sql server check if temp table exists
SQL
6 votes356 views2 answers
Stored procedure to delete data from table in mysql
SQL
0 vote414 views1 answer
Mysql update column default value CURRENT_TIMESTAMP error
SQL
1 vote383 views1 answer
Mysql take number in string
SQL
0 vote357 views1 answer
Access denied for user 'root'@'localhost' (using password no) stackoverflow
SQL