/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote462 views1 answer
Delet all record statment sql
SQL
0 vote343 views1 answer
Change mysql version to 5.7 in ubuntu
SQL
1 vote370 views1 answer
How to start with sql
SQL
0 vote351 views1 answer
Update sqlaclehmy sqlalchemy.orm.evaluator.UnevaluatableError: Cannot evaluate BinaryExpression with operator <function like_op at 0x03CF1DF0>
SQL
5 votes381 views1 answer
How to get all table names in sql query
SQL
0 vote407 views1 answer
Manage Database in MySQL
SQL
4 votes360 views2 answers
Sql select except null
SQL
10 votes482 views4 answers
Sql server pagination
SQL
12 votes367 views1 answer
Docker mysql random root password
SQL
0 vote352 views1 answer
Get the First and Last Word from a String or Sentence
SQL