Run SQL on files directly

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Run SQL on files directly

David Markovitz

Hi

Spark SQL supports direct querying on files (here), e.g. –

 

select * from csv.`/my/path/myfile.csv`

 

Does anybody know if it possible to pass options (sep, header, encoding etc.) with this syntax?

 

Thanks

 

 

Best regards,

 

David (דודו) Markovitz

Technology Solutions Professional, Data Platform

Microsoft Israel

 

Mobile: +972-525-834-304

Office: +972-747-119-274

 

cid:image002.png@01D166A7.36DE1270

 

Reply | Threaded
Open this post in threaded view
|

Re: Run SQL on files directly

Subhash Sriram
Hi David,

I’m not sure if that is possible, but why not just read the CSV file using the Scala API, specifying those options, and then query it using SQL by creating a temp view?

Thanks,
Subhash 

Sent from my iPhone

On Dec 8, 2018, at 12:39 PM, David Markovitz <[hidden email]> wrote:

Hi

Spark SQL supports direct querying on files (here), e.g. –

 

select * from csv.`/my/path/myfile.csv`

 

Does anybody know if it possible to pass options (sep, header, encoding etc.) with this syntax?

 

Thanks

 

 

Best regards,

 

David (דודו) Markovitz

Technology Solutions Professional, Data Platform

Microsoft Israel

 

Mobile: +972-525-834-304

Office: +972-747-119-274

 

<image001.png>

 

Reply | Threaded
Open this post in threaded view
|

RE: Run SQL on files directly

David Markovitz

Thanks Subhash

I am familiar with the other APIs but I am curios about this specific one and I could not figure it out from the git repository.

 

Best regards,

 

David (דודו) Markovitz

Technology Solutions Professional, Data Platform

Microsoft Israel

 

Mobile: +972-525-834-304

Office: +972-747-119-274

 

cid:image002.png@01D166A7.36DE1270

 

From: Subhash Sriram <[hidden email]>
Sent: Saturday, December 8, 2018 10:38 PM
To: David Markovitz <[hidden email]>
Cc: [hidden email]
Subject: Re: Run SQL on files directly

 

Hi David,

 

I’m not sure if that is possible, but why not just read the CSV file using the Scala API, specifying those options, and then query it using SQL by creating a temp view?

 

Thanks,

Subhash 

Sent from my iPhone


On Dec 8, 2018, at 12:39 PM, David Markovitz <[hidden email]> wrote:

Hi

Spark SQL supports direct querying on files (here), e.g. –

 

select * from csv.`/my/path/myfile.csv`

 

Does anybody know if it possible to pass options (sep, header, encoding etc.) with this syntax?

 

Thanks

 

 

Best regards,

 

David (דודו) Markovitz

Technology Solutions Professional, Data Platform

Microsoft Israel

 

Mobile: +972-525-834-304

Office: +972-747-119-274

 

<image001.png>