spark datatypes

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

spark datatypes

David Hodefi
Looking on the source code, it seems like DateDataType  is Int.

"class DateType private() extends AtomicType {
  // The companion object and this class is separated so the companion object also subclasses
// this type. Otherwise, the companion object would be of type "DateType$" in byte code.
// Defined with a private constructor so the companion object is the only possible instantiation.
private[sql] type InternalType = Int

I know that Timestamp requires 10 digits that's why we use Long and not Int.
Can someone explain why InternalType is Int? 

Thanks David