/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

-1 vote123 views1 answer
Get id from just inserted row mysql server python
SQL
1 vote107 views1 answer
How to delete a record from sqlite in android
SQL
1 vote128 views1 answer
Update mysql centos
SQL
0 vote99 views1 answer
Min mysql
SQL
2 votes154 views5 answers
Tablespace ORACLE
SQL
10 votes123 views6 answers
Sql between
SQL
0 vote120 views1 answer
Retornar apenas o ano mysql date
SQL
15 votes118 views2 answers
Select where duplicate mysql
SQL
0 vote120 views1 answer
Oracle query archive mode
SQL
0 vote123 views1 answer
Which sql statement would you use to remove a view called emp_dept_vu from your schema?
SQL