Supported SQL syntax in Spark SQL

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Supported SQL syntax in Spark SQL

Nick Chammas
Is there a place where we can find an up-to-date list of supported SQL syntax in Spark SQL?

Nick

Reply | Threaded
Open this post in threaded view
|

Re: Supported SQL syntax in Spark SQL

Nick Chammas

For example, are LIKE 'string%' queries supported? Trying one on 1.0.1 yields java.lang.ExceptionInInitializerError.

Nick



On Sat, Jul 12, 2014 at 10:16 PM, Nick Chammas <[hidden email]> wrote:
Is there a place where we can find an up-to-date list of supported SQL syntax in Spark SQL?

Nick



View this message in context: Supported SQL syntax in Spark SQL
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Supported SQL syntax in Spark SQL

Nick Chammas

Actually, this looks like its some kind of regression in 1.0.1, perhaps related to assembly and packaging with spark-ec2. I don’t see this issue with the same data on a 1.0.0 EC2 cluster.

How can I trace this down for a bug report?

Nick



On Sun, Jul 13, 2014 at 11:18 PM, Nicholas Chammas <[hidden email]> wrote:

For example, are LIKE 'string%' queries supported? Trying one on 1.0.1 yields java.lang.ExceptionInInitializerError.

Nick



On Sat, Jul 12, 2014 at 10:16 PM, Nick Chammas <[hidden email]> wrote:
Is there a place where we can find an up-to-date list of supported SQL syntax in Spark SQL?

Nick



View this message in context: Supported SQL syntax in Spark SQL
Sent from the Apache Spark User List mailing list archive at Nabble.com.


Reply | Threaded
Open this post in threaded view
|

Re: Supported SQL syntax in Spark SQL

Michael Armbrust
Are you sure the code running on the cluster has been updated?  We recently optimized the execution of LIKE queries that can be evaluated without using full regular expressions.  So it's possible this error is due to missing functionality on the executors.

How can I trace this down for a bug report?

If the above doesn't fix it, the following would be helpful:
 - The full stack trace
 - The queryExecution from the SchemaRDD (i.e. println(sql("SELECT ...").queryExecution))
Reply | Threaded
Open this post in threaded view
|

Re: Supported SQL syntax in Spark SQL

Nick Chammas

Are you sure the code running on the cluster has been updated?

I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m assuming that’s taken care of, at least in theory.

I just spun down the clusters I had up, but I will revisit this tomorrow and provide the information you requested.

Nick

Reply | Threaded
Open this post in threaded view
|

Re: Supported SQL syntax in Spark SQL

Martin Gammelsæter
I am very interested in the original question as well, is there any
list (even if it is simply in the code) of all supported syntax for
Spark SQL?

On Mon, Jul 14, 2014 at 6:41 AM, Nicholas Chammas
<[hidden email]> wrote:
>> Are you sure the code running on the cluster has been updated?
>
> I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m
> assuming that’s taken care of, at least in theory.
>
> I just spun down the clusters I had up, but I will revisit this tomorrow and
> provide the information you requested.
>
> Nick



--
Mvh.
Martin Gammelsæter
92209139
Reply | Threaded
Open this post in threaded view
|

Re: Supported SQL syntax in Spark SQL

Michael Armbrust

In general the hive parser provided by HQL is much more complete at the moment.  Long term we will likely stop using parser combinators and either write a more complete parser, or adopt one from an existing project.


On Mon, Jul 14, 2014 at 12:25 AM, Martin Gammelsæter <[hidden email]> wrote:
I am very interested in the original question as well, is there any
list (even if it is simply in the code) of all supported syntax for
Spark SQL?

On Mon, Jul 14, 2014 at 6:41 AM, Nicholas Chammas
<[hidden email]> wrote:
>> Are you sure the code running on the cluster has been updated?
>
> I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m
> assuming that’s taken care of, at least in theory.
>
> I just spun down the clusters I had up, but I will revisit this tomorrow and
> provide the information you requested.
>
> Nick



--
Mvh.
Martin Gammelsæter
92209139

Reply | Threaded
Open this post in threaded view
|

Re: Supported SQL syntax in Spark SQL

Nick Chammas
FYI: I've created SPARK-2560 to track creating SQL reference docs for Spark SQL.


On Mon, Jul 14, 2014 at 2:06 PM, Michael Armbrust <[hidden email]> wrote:

In general the hive parser provided by HQL is much more complete at the moment.  Long term we will likely stop using parser combinators and either write a more complete parser, or adopt one from an existing project.


On Mon, Jul 14, 2014 at 12:25 AM, Martin Gammelsæter <[hidden email]> wrote:
I am very interested in the original question as well, is there any
list (even if it is simply in the code) of all supported syntax for
Spark SQL?

On Mon, Jul 14, 2014 at 6:41 AM, Nicholas Chammas
<[hidden email]> wrote:
>> Are you sure the code running on the cluster has been updated?
>
> I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m
> assuming that’s taken care of, at least in theory.
>
> I just spun down the clusters I had up, but I will revisit this tomorrow and
> provide the information you requested.
>
> Nick



--
Mvh.
Martin Gammelsæter
92209139