XPATH_INT behavior - XML - Function in Spark

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

XPATH_INT behavior - XML - Function in Spark

Chetan Khatri
Hi Spark Users,

I want to parse xml coming in the query columns and get the value, I am using xpath_int which works as per my requirement but When I am embedding in the Spark SQL query columns it is failing.

select timesheet_profile_id, 
xpath_int(timesheet_profile_code, '(/timesheetprofile/weeks/week[td.current_week]/td.day)[1]')

this failed 
where Hardcoded values work for the above scenario

scala> spark.sql("select timesheet_profile_id, xpath_int(timesheet_profile_code, '(/timesheetprofile/weeks/week[2]/friday)[1]') from TIMESHEET_PROFILE_ATT").show(false)

Anyone has worked on this? Thanks in advance.

Thanks
- Chetan

Reply | Threaded
Open this post in threaded view
|

Re: XPATH_INT behavior - XML - Function in Spark

Chetan Khatri
Can someone please help.. Thanks in advance. 

On Mon, May 11, 2020 at 5:29 PM Chetan Khatri <[hidden email]> wrote:
Hi Spark Users,

I want to parse xml coming in the query columns and get the value, I am using xpath_int which works as per my requirement but When I am embedding in the Spark SQL query columns it is failing.

select timesheet_profile_id, 
xpath_int(timesheet_profile_code, '(/timesheetprofile/weeks/week[td.current_week]/td.day)[1]')

this failed 
where Hardcoded values work for the above scenario

scala> spark.sql("select timesheet_profile_id, xpath_int(timesheet_profile_code, '(/timesheetprofile/weeks/week[2]/friday)[1]') from TIMESHEET_PROFILE_ATT").show(false)

Anyone has worked on this? Thanks in advance.

Thanks
- Chetan

Reply | Threaded
Open this post in threaded view
|

Re: XPATH_INT behavior - XML - Function in Spark

Jeff Evans
It sounds like you're expecting the XPath expression to evaluate embedded Spark SQL expressions?  From the documentation, there appears to be no reason to expect that to work.

On Tue, May 12, 2020 at 2:09 PM Chetan Khatri <[hidden email]> wrote:
Can someone please help.. Thanks in advance. 

On Mon, May 11, 2020 at 5:29 PM Chetan Khatri <[hidden email]> wrote:
Hi Spark Users,

I want to parse xml coming in the query columns and get the value, I am using xpath_int which works as per my requirement but When I am embedding in the Spark SQL query columns it is failing.

select timesheet_profile_id, 
xpath_int(timesheet_profile_code, '(/timesheetprofile/weeks/week[td.current_week]/td.day)[1]')

this failed 
where Hardcoded values work for the above scenario

scala> spark.sql("select timesheet_profile_id, xpath_int(timesheet_profile_code, '(/timesheetprofile/weeks/week[2]/friday)[1]') from TIMESHEET_PROFILE_ATT").show(false)

Anyone has worked on this? Thanks in advance.

Thanks
- Chetan

Reply | Threaded
Open this post in threaded view
|

Re: XPATH_INT behavior - XML - Function in Spark

Chetan Khatri
Thank you for the clarification. 
What do you suggest to get this use case achieved.

On Tue, May 12, 2020 at 5:35 PM Jeff Evans <[hidden email]> wrote:
It sounds like you're expecting the XPath expression to evaluate embedded Spark SQL expressions?  From the documentation, there appears to be no reason to expect that to work.

On Tue, May 12, 2020 at 2:09 PM Chetan Khatri <[hidden email]> wrote:
Can someone please help.. Thanks in advance. 

On Mon, May 11, 2020 at 5:29 PM Chetan Khatri <[hidden email]> wrote:
Hi Spark Users,

I want to parse xml coming in the query columns and get the value, I am using xpath_int which works as per my requirement but When I am embedding in the Spark SQL query columns it is failing.

select timesheet_profile_id, 
xpath_int(timesheet_profile_code, '(/timesheetprofile/weeks/week[td.current_week]/td.day)[1]')

this failed 
where Hardcoded values work for the above scenario

scala> spark.sql("select timesheet_profile_id, xpath_int(timesheet_profile_code, '(/timesheetprofile/weeks/week[2]/friday)[1]') from TIMESHEET_PROFILE_ATT").show(false)

Anyone has worked on this? Thanks in advance.

Thanks
- Chetan

Reply | Threaded
Open this post in threaded view
|

Re: XPATH_INT behavior - XML - Function in Spark

Chetan Khatri
In reply to this post by Jeff Evans
Anyone can please suggest how can I achieve this?

On Tue, May 12, 2020 at 5:35 PM Jeff Evans <[hidden email]> wrote:
It sounds like you're expecting the XPath expression to evaluate embedded Spark SQL expressions?  From the documentation, there appears to be no reason to expect that to work.

On Tue, May 12, 2020 at 2:09 PM Chetan Khatri <[hidden email]> wrote:
Can someone please help.. Thanks in advance. 

On Mon, May 11, 2020 at 5:29 PM Chetan Khatri <[hidden email]> wrote:
Hi Spark Users,

I want to parse xml coming in the query columns and get the value, I am using xpath_int which works as per my requirement but When I am embedding in the Spark SQL query columns it is failing.

select timesheet_profile_id, 
xpath_int(timesheet_profile_code, '(/timesheetprofile/weeks/week[td.current_week]/td.day)[1]')

this failed 
where Hardcoded values work for the above scenario

scala> spark.sql("select timesheet_profile_id, xpath_int(timesheet_profile_code, '(/timesheetprofile/weeks/week[2]/friday)[1]') from TIMESHEET_PROFILE_ATT").show(false)

Anyone has worked on this? Thanks in advance.

Thanks
- Chetan