[bug] Scala reflection "assertion failed: class Byte" in Dataset.toJSON

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[bug] Scala reflection "assertion failed: class Byte" in Dataset.toJSON

Brandon Vincent
Hi all,

I have a job that executes a query and collects the results as JSON using Dataset.toJSON. For the most part it is stable, but sometimes it fails randomly with a scala assertion error. Here is the stack trace:

                                      org.apache.spark.sql.Dataset.toJSON            Dataset.scala: 3222
                                    org.apache.spark.sql.Encoders$.STRING           Encoders.scala:   96
          org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply  ExpressionEncoder.scala:   72
           org.apache.spark.sql.catalyst.ScalaReflection$.deserializerFor    ScalaReflection.scala:  161
           org.apache.spark.sql.catalyst.ScalaReflection$.deserializerFor    ScalaReflection.scala:  173
  org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects    ScalaReflection.scala:   49
  org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects$    ScalaReflection.scala:  925
   org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects    ScalaReflection.scala:  926
                  scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo    TypeConstraints.scala:   68
org.apache.spark.sql.catalyst.ScalaReflection$.$anonfun$deserializerFor$1    ScalaReflection.scala:  260
               org.apache.spark.sql.catalyst.ScalaReflection$.localTypeOf    ScalaReflection.scala:   49
               org.apache.spark.sql.catalyst.ScalaReflection.localTypeOf$    ScalaReflection.scala:  939
                org.apache.spark.sql.catalyst.ScalaReflection.localTypeOf    ScalaReflection.scala:  941
                           scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe           TypeTags.scala:  237
                scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute           TypeTags.scala:  237
      org.apache.spark.sql.catalyst.ScalaReflection$$typecreator7$1.apply    ScalaReflection.scala:  260
              scala.reflect.internal.Symbols$TypeSymbol.toTypeConstructor            Symbols.scala: 3081
    scala.reflect.internal.Symbols$SymbolContextApiImpl.toTypeConstructor            Symbols.scala:  194
                scala.reflect.internal.Symbols$TypeSymbol.typeConstructor            Symbols.scala: 3154
                  scala.reflect.internal.Symbols$TypeSymbol.setTyconCache            Symbols.scala: 3163
                   scala.reflect.internal.SymbolTable.throwAssertionError        SymbolTable.scala:  183
java.lang.AssertionError: assertion failed: class Byte


It can also come up with "class Boolean" with the same stack trace.
Any clues on this? I wasn't able to find information about this specific assertion error.

The spark version is 2.4.4 compiled with scala 2.12.

Thank you.