You can set the log level to INFO, it looks like spark is logging applicative errors as INFO. When I have errors that I can reproduce only on live data, I am running a spark shell with my job in its classpath, then I debug & tweak things to find out what happens.
I ran into a similar problem earlier. The issue is that spark does not actually depend on log4j any more. You need to manually add the dependency to your build system. For example, in sbt, I added the following to build.sbt