Re: How to print plan of Structured Streaming DataFrame

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Re: How to print plan of Structured Streaming DataFrame

Shixiong(Ryan) Zhu
-dev +user

Which Spark version are you using? There is a bug in the old Spark. Try to use the latest version.

In addition, you can call `query.explain()` as well.

On Mon, Nov 20, 2017 at 4:00 AM, Chang Chen <[hidden email]> wrote:
Hi Guys

I modified StructuredNetworkWordCount to see what the executed plan is,  here are my codes:

val wordCounts = words.groupBy("value").count()

// Start running the query that prints the running counts to the console
val query = wordCounts.writeStream
  .outputMode("complete")
  .format("console")
  .start()

wordCounts.explain()  // additional codes


But it failed with “AnalysisException: Queries with streaming sources must be executed with writeStream.start()”?


Thanks
Chang