/SQL/Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

Org.apache.spark.sql.avro.IncompatibleSchemaException: Unexpected type org.apache.spark.ml.linalg.VectorUDT

# To convert any Vector to an Array[Double] you can use the following UDF:

import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.functions.col
import org.apache.spark.ml.linalg.Vector

val vectorToArrayUdf = udf((vector: Vector) => vector.toArray)

// The following will work
val output = dataPredictions
    .withColumn("probabilities", vectorToArrayUdf(col("probability")))
    .select("id", "probabilities", "prediction")

output.write.format("com.databricks.spark.avro").save(path)

Source: stackoverflow.com

Similar codes

0 vote262 views1 answer
Sqlite woth cmake
SQL
3 votes239 views2 answers
How to start postgresql on windows
SQL
19 votes231 views2 answers
Mysql add column after another
SQL
0 vote249 views1 answer
Set column in all row mysql
SQL
0 vote220 views1 answer
Como verificar registros duplicados no sql postgresql
SQL
0 vote260 views1 answer
DATEDIFF minute postgres
SQL
0 vote265 views1 answer
Mysql collation portugues brasil
SQL
1 vote250 views1 answer
Illuminate\Database\QueryException SQLSTATE[HY000] [2002] Connection refused (SQL: select * from information_schema.tables where table_schema = laravel and table_name = migrations and table_type = 'BASE TABLE')
SQL
0 vote232 views1 answer
Mysql select where field is a value
SQL
-1 vote257 views1 answer
Brisanje dupliranih konatakata
SQL