Search and Replace issue

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Search and Replace issue

nischay21
This post has NOT been accepted by the mailing list yet.
We have created two dataset sentenceDataFrame, sentenceDataFrame2 where search replace should happen.

sentenceDataFrame2 stores the search and replace terms.

We also performed all 11 types of join 'inner', 'outer', 'full', 'fullouter', 'leftouter', 'left', 'rightouter', 'right', 'leftsemi', 'leftanti', 'cross' none of them gave us the result.

Can you please let us know Where we are going Wrong and Kindly point us in the right direction.

        List<Row> data = Arrays.asList(
                RowFactory.create(0, "Allen jeevi pramod Allen"),
                RowFactory.create(1,"sandesh Armstrong jeevi"),
                RowFactory.create(2,"harsha nischay DeWALT"));

        StructType schema = new StructType(new StructField[] {
        new StructField("label", DataTypes.IntegerType, false,
          Metadata.empty()),
        new StructField("sentence", DataTypes.StringType, false,
          Metadata.empty()) });
        Dataset<Row> sentenceDataFrame = spark.createDataFrame(data, schema);


        List<Row> data2 = Arrays.asList(
          RowFactory.create("Allen", "Apex Tool Group"),
          RowFactory.create("Armstrong","Apex Tool Group"),
          RowFactory.create("DeWALT","StanleyBlack"));

        StructType schema2 = new StructType(new StructField[] {
        new StructField("label2", DataTypes.StringType, false,
          Metadata.empty()),
        new StructField("sentence2", DataTypes.StringType, false,
          Metadata.empty()) });
        Dataset<Row> sentenceDataFrame2 = spark.createDataFrame(data2, schema2);

        Dataset<Row> remainingElements=sentenceDataFrame.join(sentenceDataFrame2,sentenceDataFrame.col("label").equalTo(sentenceDataFrame2.col("label2")),"cross");
        System.out.println("Left anti join count :"+remainingElements.count());
       

Expected Output

Apex Tool Group jeevi pramod Apex Tool Group
sandesh Apex Tool Group jeevi
harsha nischay StanleyBlack