Write queries about Language-Integration

Write queries about Language-Integration

The queries related to language integration are experimental and now only scala is supported for the time being.

Spark SQL also supports writing queries in domain-specific languages.

// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
// Importing the SQL context gives access to all the public SQL functions and implicit conversions.
import sqlContext._
val people: RDD[Person] = ... // An RDD of case class objects, from the first example.

// The following is the same as 'SELECT name FROM people WHERE age >= 10 AND age <= 19'
val teenagers = people.where('age >= 10).where('age <= 19).select('name)
teenagers.map(t => "Name: " + t(0)).collect().foreach(println)

DSLs use Scala symbols to represent columns in potential tables, which are previously marked with the prefix ('). T hese symbols are implicitly converted into expressions evaluated by the SQL execution engine. You can find out more in ScalaDoc.