disable log4j for spark-shell

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

disable log4j for spark-shell

Gil Vernik
Hi,

I would like to run spark-shell without any INFO messages printed.
To achieve this I edited /conf/log4j.properties and added line
log4j.rootLogger=OFF
that suppose to disable all logging.

However, when I  run ./spark-shell I see the message
4/08/03 16:02:15 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

And then spark-shell prints all INFO messages as is.

What did i missed? Why spark-shell uses default log4j properties and not the one defined in /conf directory?
Is there another solution to prevent spark-shell from printing INFO messages?

Thanks,
Gil.


Reply | Threaded
Open this post in threaded view
|

Re: disable log4j for spark-shell

sowen
That's just a template. Nothing consults that file by default.

It's looking inside the Spark .jar. If you edit
core/src/main/resources/org/apache/spark/log4j-defaults.properties and
rebuild Spark, it will pick up those changes.

I think you could also use the JVM argument
"-Dlog4j.configuration=conf/log4j-defaults.properties" to force it to
look at your local, edited props file.

Someone  may have to correct me, but I think that in master right now,
that means using --driver-java-options=".." to set an argument to the
JVM that runs the shell?

On Sun, Aug 3, 2014 at 2:07 PM, Gil Vernik <[hidden email]> wrote:

> Hi,
>
> I would like to run spark-shell without any INFO messages printed.
> To achieve this I edited /conf/log4j.properties and added line
> log4j.rootLogger=OFF
> that suppose to disable all logging.
>
> However, when I  run ./spark-shell I see the message
> 4/08/03 16:02:15 INFO SecurityManager: Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
>
> And then spark-shell prints all INFO messages as is.
>
> What did i missed? Why spark-shell uses default log4j properties and not the
> one defined in /conf directory?
> Is there another solution to prevent spark-shell from printing INFO
> messages?
>
> Thanks,
> Gil.
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: disable log4j for spark-shell

Patrick Wendell
If you want to customize the logging behavior - the simplest way is to copy conf/log4j.properties.tempate to conf/log4j.properties. Then you can go and modify the log level in there. The spark shells should pick this up.




On Sun, Aug 3, 2014 at 6:16 AM, Sean Owen <[hidden email]> wrote:
That's just a template. Nothing consults that file by default.

It's looking inside the Spark .jar. If you edit
core/src/main/resources/org/apache/spark/log4j-defaults.properties and
rebuild Spark, it will pick up those changes.

I think you could also use the JVM argument
"-Dlog4j.configuration=conf/log4j-defaults.properties" to force it to
look at your local, edited props file.

Someone  may have to correct me, but I think that in master right now,
that means using --driver-java-options=".." to set an argument to the
JVM that runs the shell?

On Sun, Aug 3, 2014 at 2:07 PM, Gil Vernik <[hidden email]> wrote:
> Hi,
>
> I would like to run spark-shell without any INFO messages printed.
> To achieve this I edited /conf/log4j.properties and added line
> log4j.rootLogger=OFF
> that suppose to disable all logging.
>
> However, when I  run ./spark-shell I see the message
> 4/08/03 16:02:15 INFO SecurityManager: Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
>
> And then spark-shell prints all INFO messages as is.
>
> What did i missed? Why spark-shell uses default log4j properties and not the
> one defined in /conf directory?
> Is there another solution to prevent spark-shell from printing INFO
> messages?
>
> Thanks,
> Gil.
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: disable log4j for spark-shell

Aaron
If someone doesn't have the access to do that is there any easy to specify a different properties file to be used?

Patrick Wendell wrote
If you want to customize the logging behavior - the simplest way is to copy
conf/log4j.properties.tempate to conf/log4j.properties. Then you can go and
modify the log level in there. The spark shells should pick this up.
Reply | Threaded
Open this post in threaded view
|

Re: disable log4j for spark-shell

Yana
You just have to tell Spark which log4j properties file to use. I think --driver-java-options="-Dlog4j.configuration=log4j.properties" should work but it didn't for me. set SPARK_JAVA_OPTS=-Dlog4j.configuration=log4j.properties did work though (this was on Windows, in local mode, assuming you put a file called log4j.properties in the bin directory where you run the shell from). In either case, like Sean said -Dlog4j.configuration=... is the magic incantation, you just have to figure out how to pass it depending on what version of SPARK you're using ( I personally find setting PRINT_SPARK_LAUNCH_COMMAND=1 very very useful when I'm trying to track a property)
Reply | Threaded
Open this post in threaded view
|

Re: disable log4j for spark-shell

hmxxyy
Tried --driver-java-options and SPARK_JAVA_OPTS, none of them worked

Had to change the default one and rebuilt.
Reply | Threaded
Open this post in threaded view
|

Re: disable log4j for spark-shell

hmxxyy
Even after changing core/src/main/resources/org/apache/spark/log4j-defaults.properties to WARN followed by a rebuild, the log level is still INFO.

Any other suggestions?
Reply | Threaded
Open this post in threaded view
|

Re: disable log4j for spark-shell

hmxxyy
Some console messages:

14/11/10 20:04:33 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:46713
14/11/10 20:04:33 INFO util.Utils: Successfully started service 'HTTP file server' on port 46713.
14/11/10 20:04:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/11/10 20:04:34 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
14/11/10 20:04:34 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
14/11/10 20:04:34 INFO netty.NettyBlockTransferService: Server created on 46997
14/11/10 20:04:34 INFO storage.BlockManagerMaster: Trying to register BlockManager
14/11/10 20:04:34 INFO storage.BlockManagerMasterActor: Registering block manager localhost:46997 with 265.0 MB RAM, BlockManagerId(<driver>, localhost, 46997)
14/11/10 20:04:35 INFO storage.BlockManagerMaster: Registered BlockManager

and the log4j-default.properties looks like:

cat core/src/main/resources/org/apache/spark/log4j-defaults.properties
# Set everything to be logged to the console
log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=WARN
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=WARN

Any suggestions?
Reply | Threaded
Open this post in threaded view
|

Re: disable log4j for spark-shell

lordjoe
    public static void main(String[] args) throws Exception {
         System.out.println("Set Log to Warn");
        Logger rootLogger = Logger.getRootLogger();
        rootLogger.setLevel(Level.WARN);
...
 works for me
Reply | Threaded
Open this post in threaded view
|

Re: disable log4j for spark-shell

Ritesh Kumar Singh
go to your spark home and then into the conf/ directory and then edit the log4j.properties file i.e. :

>gedit $SPARK_HOME/conf/log4j.properties

and set root logger to:
   log4j.rootCategory=WARN, console

U don't need to build spark for the changes to take place. Whenever you open spark-shel, it by default looks into the conf directories and loads all the properties.

Thanks

On Tue, Nov 11, 2014 at 6:34 AM, lordjoe <[hidden email]> wrote:
    public static void main(String[] args) throws Exception {
         System.out.println("Set Log to Warn");
        Logger rootLogger = Logger.getRootLogger();
        rootLogger.setLevel(Level.WARN);
...
 works for me




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/disable-log4j-for-spark-shell-tp11278p18535.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Fwd: disable log4j for spark-shell

Ritesh Kumar Singh

---------- Forwarded message ----------
From: Ritesh Kumar Singh <[hidden email]>
Date: Tue, Nov 11, 2014 at 2:18 PM
Subject: Re: disable log4j for spark-shell
To: lordjoe <[hidden email]>
Cc: [hidden email]


go to your spark home and then into the conf/ directory and then edit the log4j.properties file i.e. :

>gedit $SPARK_HOME/conf/log4j.properties

and set root logger to:
   log4j.rootCategory=WARN, console

U don't need to build spark for the changes to take place. Whenever you open spark-shel, it by default looks into the conf directories and loads all the properties.

Thanks

On Tue, Nov 11, 2014 at 6:34 AM, lordjoe <[hidden email]> wrote:
    public static void main(String[] args) throws Exception {
         System.out.println("Set Log to Warn");
        Logger rootLogger = Logger.getRootLogger();
        rootLogger.setLevel(Level.WARN);
...
 works for me




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/disable-log4j-for-spark-shell-tp11278p18535.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



Reply | Threaded
Open this post in threaded view
|

Re: disable log4j for spark-shell

brichards
This post has NOT been accepted by the mailing list yet.
In reply to this post by Yana
FYI its --driver-java-options "-Dkey=value"  

no equal sign between the flag and the arguments.

Chewed up some time figuring that out.

Bobby
Reply | Threaded
Open this post in threaded view
|

Re: disable log4j for spark-shell

Akhil
This post has NOT been accepted by the mailing list yet.
Edit your conf/log4j.properties file and Change the following line:

       log4j.rootCategory=INFO, console

to

        log4j.rootCategory=ERROR, console

Another approach would be to :

Fireup spark-shell and type in the following:

    import org.apache.log4j.Logger
    import org.apache.log4j.Level

    Logger.getLogger("org").setLevel(Level.OFF)
    Logger.getLogger("akka").setLevel(Level.OFF)

You won't see any logs after that.