Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend

James Starks
This is not problem directly caused by Spark, but it's related; thus asking here. I use spark to read data from parquet and processing some http call with sttp (https://github.com/softwaremill/sttp). However, spark throws 

    Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend

It's understood why such exception is thrown because
FollowRedirectsBackend is not seralizable. So I would like know in such case -  are there any ways to get around this problem without modifying, recompiling original code? 

Thanks

Reply | Threaded
Open this post in threaded view
|

Re: Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend

Koert Kuipers
if you only use it in the executors sometimes using lazy works

On Thu, Nov 29, 2018 at 9:45 AM James Starks <[hidden email]> wrote:
This is not problem directly caused by Spark, but it's related; thus asking here. I use spark to read data from parquet and processing some http call with sttp (https://github.com/softwaremill/sttp). However, spark throws 

    Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend

It's understood why such exception is thrown because
FollowRedirectsBackend is not seralizable. So I would like know in such case -  are there any ways to get around this problem without modifying, recompiling original code? 

Thanks

Reply | Threaded
Open this post in threaded view
|

Re: Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend

chris-2
If it’s just a couple of classes and they are actually suitable for serializing and you have the source code then you can shadow them in your own project with the serializable interface added. Your shadowed classes should be on the classpath before the library’s versions which should lead to spark being able to use the serializable versions.

That’s very much a last resort though! 

Chris 

On 30 Nov 2018, at 05:08, Koert Kuipers <[hidden email]> wrote:

if you only use it in the executors sometimes using lazy works

On Thu, Nov 29, 2018 at 9:45 AM James Starks <[hidden email]> wrote:
This is not problem directly caused by Spark, but it's related; thus asking here. I use spark to read data from parquet and processing some http call with sttp (https://github.com/softwaremill/sttp). However, spark throws 

    Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend

It's understood why such exception is thrown because
FollowRedirectsBackend is not seralizable. So I would like know in such case -  are there any ways to get around this problem without modifying, recompiling original code? 

Thanks

Reply | Threaded
Open this post in threaded view
|

Re: Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend

James Starks
Shadowed with

object MyObject {
  def mymethod(param: MyParam) = actual_function(param)
}
class MyObject {
  import MyObject._
  session.map { ... =>
     mymethod(...)
  }
}

does the job.

Thanks for the advice!

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, November 30, 2018 9:26 AM, <[hidden email]> wrote:


If it’s just a couple of classes and they are actually suitable for serializing and you have the source code then you can shadow them in your own project with the serializable interface added. Your shadowed classes should be on the classpath before the library’s versions which should lead to spark being able to use the serializable versions.

That’s very much a last resort though! 

Chris 

On 30 Nov 2018, at 05:08, Koert Kuipers <[hidden email]> wrote:
if you only use it in the executors sometimes using lazy works

On Thu, Nov 29, 2018 at 9:45 AM James Starks <[hidden email]> wrote:
This is not problem directly caused by Spark, but it's related; thus asking here. I use spark to read data from parquet and processing some http call with sttp (https://github.com/softwaremill/sttp). However, spark throws 

    Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend

It's understood why such exception is thrown because
FollowRedirectsBackend is not seralizable. So I would like know in such case -  are there any ways to get around this problem without modifying, recompiling original code? 

Thanks