Need help regarding logging / log4j.properties

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Need help regarding logging / log4j.properties

Debabrata Ghosh
Greetings All !

I needed some help in obtaining the application logs but I am really confused where it's currently located. Please allow me to explain my problem:
1. I am running the Spark application (written in Java) in a Hortonworks Data Platform Hadoop cluster

2. My spark-submit command is the following:
spark-submit --class com.ibm.spark.sparkkafkamapper.ServiceMapperMain --master yarn --packages org.apache.kafka:kafka-clients:0.10.0.1,org.apache.spark:spark-streaming-kafka-0-10_2.11:2.3.0 --deploy-mode client --files job-configuration.properties,log4j.properties,/usr/hdp/current/spark2-client/conf/hive-site.xml ./spark-job.jar --settings=job-configuration.properties log4j-spark.properties --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties"

3. Content of the log4.properties is the following:
# Set everything to be logged to the console
log4j.rootCategory=ALL, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

 # Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=ALL
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

 # SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

#Any custom class debug
log4j.logger.com.vmeg.code=DEBUG

4. Inside the spark application, I am having plenty of logger functions and one example is like this:
logger.info("No dataset recieved for this batch");

Question: My question is where is the application log files being written ? I have checked in the yarn logs but couldn't find the messages I have written in the java file.
Request your help please as I am little confused and know that there is something very silly which I am missing.

Thanks in advance !

Debu
Reply | Threaded
Open this post in threaded view
|

Re: Need help regarding logging / log4j.properties

Roland Johann
Hi Debu,

you need to define spark config properties before the jar file path at spark-submit. Everything after the jar path will be passed as arguments to your application.

Best Regards

Debabrata Ghosh <[hidden email]> schrieb am Do. 31. Okt. 2019 um 03:26:
Greetings All !

I needed some help in obtaining the application logs but I am really confused where it's currently located. Please allow me to explain my problem:
1. I am running the Spark application (written in Java) in a Hortonworks Data Platform Hadoop cluster

2. My spark-submit command is the following:
spark-submit --class com.ibm.spark.sparkkafkamapper.ServiceMapperMain --master yarn --packages org.apache.kafka:kafka-clients:0.10.0.1,org.apache.spark:spark-streaming-kafka-0-10_2.11:2.3.0 --deploy-mode client --files job-configuration.properties,log4j.properties,/usr/hdp/current/spark2-client/conf/hive-site.xml ./spark-job.jar --settings=job-configuration.properties log4j-spark.properties --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties"

3. Content of the log4.properties is the following:
# Set everything to be logged to the console
log4j.rootCategory=ALL, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

 # Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=ALL
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

 # SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

#Any custom class debug
log4j.logger.com.vmeg.code=DEBUG

4. Inside the spark application, I am having plenty of logger functions and one example is like this:
logger.info("No dataset recieved for this batch");

Question: My question is where is the application log files being written ? I have checked in the yarn logs but couldn't find the messages I have written in the java file.
Request your help please as I am little confused and know that there is something very silly which I am missing.

Thanks in advance !

Debu
--

Roland Johann

Software Developer/Data Engineer

phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany

Mobil: +49 172 365 26 46
Mail: [hidden email]
Web: phenetic.io

Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann