Creating spark Row from database values

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Creating spark Row from database values

Robin Kuttaiah
Hello,

Currently I have Oracle database table with description as shown below;

Table INSIGHT_ID_FED_IDENTIFIERS
-------------------- ---- ------------- 
CURRENT_INSTANCE_ID       VARCHAR2(100) 
PREVIOUS_INSTANCE_ID      VARCHAR2(100) 


Sample values in the table basically output of select * from INSIGHT_ID_FED_IDENTIFIERS. For simplicity I have put only one row.


CURRENT_INSTANCE_ID           PREVIOUS_INSTANCE_ID
-------------------                               -------------------
curInstanceId1                      prevInstanceId1


I have the spark schema associated with it.


Now I need to create a Spark row(org.apache.spark.sql.Row) out of it.  

Can someone help me understanding on how this can be achieved?

regards,
Robin Kuttaiah
Reply | Threaded
Open this post in threaded view
|

Re: Creating spark Row from database values

Shahab Yunus
Hi there. Have you seen this link? https://medium.com/@mrpowers/manually-creating-spark-dataframes-b14dae906393 

It shows you multiple ways to manually create a dataframe.

Hope it helps.

Regards,
Shahab

On Wed, Sep 26, 2018 at 8:02 AM Kuttaiah Robin <[hidden email]> wrote:
Hello,

Currently I have Oracle database table with description as shown below;

Table INSIGHT_ID_FED_IDENTIFIERS
-------------------- ---- ------------- 
CURRENT_INSTANCE_ID       VARCHAR2(100) 
PREVIOUS_INSTANCE_ID      VARCHAR2(100) 


Sample values in the table basically output of select * from INSIGHT_ID_FED_IDENTIFIERS. For simplicity I have put only one row.


CURRENT_INSTANCE_ID           PREVIOUS_INSTANCE_ID
-------------------                               -------------------
curInstanceId1                      prevInstanceId1


I have the spark schema associated with it.


Now I need to create a Spark row(org.apache.spark.sql.Row) out of it.  

Can someone help me understanding on how this can be achieved?

regards,
Robin Kuttaiah
Reply | Threaded
Open this post in threaded view
|

Re: Creating spark Row from database values

Robin Kuttaiah
Thanks, I'll check it out. 

On Wed, Sep 26, 2018 at 6:25 PM Shahab Yunus <[hidden email]> wrote:
Hi there. Have you seen this link? https://medium.com/@mrpowers/manually-creating-spark-dataframes-b14dae906393 

It shows you multiple ways to manually create a dataframe.

Hope it helps.

Regards,
Shahab

On Wed, Sep 26, 2018 at 8:02 AM Kuttaiah Robin <[hidden email]> wrote:
Hello,

Currently I have Oracle database table with description as shown below;

Table INSIGHT_ID_FED_IDENTIFIERS
-------------------- ---- ------------- 
CURRENT_INSTANCE_ID       VARCHAR2(100) 
PREVIOUS_INSTANCE_ID      VARCHAR2(100) 


Sample values in the table basically output of select * from INSIGHT_ID_FED_IDENTIFIERS. For simplicity I have put only one row.


CURRENT_INSTANCE_ID           PREVIOUS_INSTANCE_ID
-------------------                               -------------------
curInstanceId1                      prevInstanceId1


I have the spark schema associated with it.


Now I need to create a Spark row(org.apache.spark.sql.Row) out of it.  

Can someone help me understanding on how this can be achieved?

regards,
Robin Kuttaiah