Read text file row by row and apply conditions

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Read text file row by row and apply conditions

swetha kadiyala
dear friends,

I am new to spark. can you please help me to read the below text file using spark and scala.

Sample data

bret|lee|A|12345|ae545|gfddfg|86786786
142343345|aaaa|D|ae342
67567|67777|U|aadfsd|34k4|84304|020|sdnfsdfn|3243|benej|32432|jsfsdf|3423
67564|67747|U|aad434|3435|843454|203|sdn454dfn|3233|gdfg|34325|sdfsddf|7657


I am receiving indicator type with 3 rd column of each row. if my indicator type=A, then i need to store that particular row data into a table called Table1.
if indicator type=D then i have to store data into seperate table called TableB and same as indicator type=U then i have to store all rows data into a separate table called Table3.

Can anyone help me how to read row by row and split the columns and apply the condition based on indicator type and store columns data into respective tables.

Thanks,
Swetha
Reply | Threaded
Open this post in threaded view
|

Re: Read text file row by row and apply conditions

vaquar khan
Hi Swetha,

It would be great if you ask same question in stackoverflow , we have very active community and monitor stack for each spark questions.

If you ask same question via stack other ppl also get benefits for similar problems.

Regards,
Vaquar khan

On Sun, Sep 29, 2019, 10:26 PM swetha kadiyala <[hidden email]> wrote:
dear friends,

I am new to spark. can you please help me to read the below text file using spark and scala.

Sample data

bret|lee|A|12345|ae545|gfddfg|86786786
142343345|aaaa|D|ae342
67567|67777|U|aadfsd|34k4|84304|020|sdnfsdfn|3243|benej|32432|jsfsdf|3423
67564|67747|U|aad434|3435|843454|203|sdn454dfn|3233|gdfg|34325|sdfsddf|7657


I am receiving indicator type with 3 rd column of each row. if my indicator type=A, then i need to store that particular row data into a table called Table1.
if indicator type=D then i have to store data into seperate table called TableB and same as indicator type=U then i have to store all rows data into a separate table called Table3.

Can anyone help me how to read row by row and split the columns and apply the condition based on indicator type and store columns data into respective tables.

Thanks,
Swetha
Reply | Threaded
Open this post in threaded view
|

Re: Read text file row by row and apply conditions

hemant singh
In reply to this post by swetha kadiyala
You can use csv reader with delimiter as '|' to split data and create a dataframe on top of the file data.
Second step, filter the dataframe on column value like indicator type=A,D etc and put it in tables. Saving to tables you can use dataframewriter(not sure what is you destination db type here).

You can keep this part where based on column type you have to filter data generic and read the configurations like say - col value | tableName from config file or metadata table and write the data to destination.

Thanks.

On Mon, Sep 30, 2019 at 8:56 AM swetha kadiyala <[hidden email]> wrote:
dear friends,

I am new to spark. can you please help me to read the below text file using spark and scala.

Sample data

bret|lee|A|12345|ae545|gfddfg|86786786
142343345|aaaa|D|ae342
67567|67777|U|aadfsd|34k4|84304|020|sdnfsdfn|3243|benej|32432|jsfsdf|3423
67564|67747|U|aad434|3435|843454|203|sdn454dfn|3233|gdfg|34325|sdfsddf|7657


I am receiving indicator type with 3 rd column of each row. if my indicator type=A, then i need to store that particular row data into a table called Table1.
if indicator type=D then i have to store data into seperate table called TableB and same as indicator type=U then i have to store all rows data into a separate table called Table3.

Can anyone help me how to read row by row and split the columns and apply the condition based on indicator type and store columns data into respective tables.

Thanks,
Swetha