Spark SQL data type

Spark SQL data type

• The type of number

• ByteType: Represents an integer of a byte. The range is -128 to 127
• ShortType: Represents an integer of two bytes. The range is -32768 to 32767
• IntegerType: Represents an integer of 4 bytes. The range is -2147483648 to 2147483647
• LongType: Represents an integer of 8 bytes. The range is -9223372036854775808 to 9223372036854775807
• FloatType: Represents a single-precision floating point of 4 bytes
• DoubleType: Represents a double float of 8 bytes
• DecimalType: Represents 10-step data for any precision. S upported by internal java.math.BigDecimal. BigDecimal consists of an integer non-scale value with arbitrary precision and a 32-bit integer
• StringType: Represents a string value
• BinaryType: Represents a byte sequence value
• Boolean Type: Stands for Boolean value

• Datetime type

• TimestampType: Represents the value that contains the field year, month, day, time, minute, and second

• DateType: Represents the value of the year, month, and day that contains the field

• Complex type

• ArrayType (elementType, containsNull): Represents a sequence value consisting of elements of the type elementType. containsNull to indicate whether ArrayType has a null value
• MapType (keyType, valueType, valueContainsNull): represents a set of keys - value pairs. T he type of key data is represented by keyType and the type of value data is represented by valueType. valueContainsNull to indicate whether MapType has a null value

• StructType (fields): Represents a value that has a StructFields (fields) structure

• StructField (name, dataType, nullable): represents a field in StructType the name of the name dataType type of the field, and nullable value of the field has a null value.

All of Spark's data types are defined in package org.apache.spark.sql which you can access import org.apache.spark.sql._