/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

4 votes192 views2 answers
Sql server date now
SQL
0 vote190 views1 answer
Failed to process SQL command - ORA-28014: cannot drop administrative user or role
SQL
15 votes191 views4 answers
Mysql export database command line
SQL
0 vote194 views1 answer
Create table in mysql mariadb primary key foreign key
SQL
8 votes218 views2 answers
Oracle undo usage
SQL
1 vote159 views2 answers
OSError: mysql_config not found mysql_config --version mariadb_config --version mysql_config --libs ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command outpu
SQL
10 votes177 views3 answers
Sql limit results
SQL
0 vote186 views1 answer
Oracle exchange partition
SQL
0 vote187 views1 answer
Turn on foreign keys check mysql
SQL
23 votes200 views4 answers
How to show all users in mysql
SQL