Fwd: [ Write JSON ] - An error occurred while calling o545.save

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Fwd: [ Write JSON ] - An error occurred while calling o545.save

Sanyal Arnab
Hi,

I am getting "Py4JJavaError: An error occurred while calling o545.save" error while executing below code.

myDF = spark.read.format("csv")\
.option("header",True)\
.option("mode","FAILFAST")\
.schema(myManualSchema)\
.load("C:\\Arnab\\Spark\\data\\2015-summary.csv")

myDF.write.format("json").mode("overwrite").save("C:\\Arnab\\Spark\\data\\tmp\\my_json_file")

Note:- "2015-summary.csv" is 8KB and I have 8GB RAM. No other application is running while I run this code.
Stack Trace -

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-23-a7940ceaf77d> in <module>
----> 1 myDF.write.format("json").mode("overwrite").save("C:\\Arnab\\Spark\\data\\tmp\\my_json_file")

C:\Arnab\Spark\spark-3.0.0-preview2-bin-hadoop3.2\python\pyspark\sql\readwriter.py in save(self, path, format, mode, partitionBy, **options)
    767             self._jwrite.save()
    768         else:
--> 769             self._jwrite.save(path)
    770 
    771     @since(1.4)

C:\Arnab\Spark\spark-3.0.0-preview2-bin-hadoop3.2\python\lib\py4j-0.10.8.1-src.zip\py4j\java_gateway.py in __call__(self, *args)
   1284         answer = self.gateway_client.send_command(command)
   1285         return_value = get_return_value(
-> 1286             answer, self.gateway_client, self.target_id, self.name)
   1287 
   1288         for temp_arg in temp_args:

C:\Arnab\Spark\spark-3.0.0-preview2-bin-hadoop3.2\python\pyspark\sql\utils.py in deco(*a, **kw)
     96     def deco(*a, **kw):
     97         try:
---> 98             return f(*a, **kw)
     99         except py4j.protocol.Py4JJavaError as e:
    100             converted = convert_exception(e.java_exception)

C:\Arnab\Spark\spark-3.0.0-preview2-bin-hadoop3.2\python\lib\py4j-0.10.8.1-src.zip\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o799.save.
: org.apache.spark.SparkException: Job aborted.
Full Stack Trace is attached.

Thanks in advance for your help.

Regards
Arnab


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

full_stack_trace.txt (15K) Download Attachment