WARN: Truncated the string representation with df.describe()

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

WARN: Truncated the string representation with df.describe()

Md. Rezaul Karim
Hi,

When I try to see the statistics in a DataFrame using the df.describe() method, I am experiencing the following WARN and as a result, nothing is getting printed:
                                                                      
17/10/16 18:37:54 WARN Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.

Now, is it possible to configure it from Spark application or IDE such as Eclipse other than doing this in the SparkEnv.conf? I tried the following on Spark application from Eclipse but does not work:

Try 1:
spark.conf.set("spark.debug.maxToStringFields", 10000)

Try 2:
    val DEFAULT_MAX_TO_STRING_FIELDS = 2500
    if (SparkEnv.get != null) {
      SparkEnv.get.conf.getInt("spark.debug.maxToStringFields", DEFAULT_MAX_TO_STRING_FIELDS)
    } else {
      DEFAULT_MAX_TO_STRING_FIELDS
}

Any clue?