WARN: Truncated the string representation with df.describe()

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

WARN: Truncated the string representation with df.describe()

Md. Rezaul Karim

When I try to see the statistics in a DataFrame using the df.describe() method, I am experiencing the following WARN and as a result, nothing is getting printed:
17/10/16 18:37:54 WARN Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.

Now, is it possible to configure it from Spark application or IDE such as Eclipse other than doing this in the SparkEnv.conf? I tried the following on Spark application from Eclipse but does not work:

Try 1:
spark.conf.set("spark.debug.maxToStringFields", 10000)

Try 2:
    if (SparkEnv.get != null) {
      SparkEnv.get.conf.getInt("spark.debug.maxToStringFields", DEFAULT_MAX_TO_STRING_FIELDS)
    } else {

Any clue?