Is there any way to set the location of the history for the spark-shell per session?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Is there any way to set the location of the history for the spark-shell per session?

Yeikel
In my team , we get elevated access to our Spark cluster using a common
username which means that we all share the same history. I am not sure if
this is common , but unfortunately there is nothing I can do about it.

Is there any option to set the location of the history?  I am looking for
something like spark-shell --history=...path or something similar that can
be reused for other sessions.

By default , history seems to be stored in $HOME/.scala_history and I did
not see anything in the documentation about it , but maybe there is an
undocumented way to do it.

Thanks for your help!



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Is there any way to set the location of the history for the spark-shell per session?

ZHANG Wei
From my understanding, you are talking about spark-shell command history, aren't you?

If yes, you can try adding `--conf 'spark.driver.extraJavaOptions=-Dscala.shell.histfile=<history file you'd like to set>` into spark-shell command arguments since Spark shell is leveraging Scala REPL JLine file backend history settings.

Cheers,
-z

________________________________________
From: Yeikel <[hidden email]>
Sent: Wednesday, April 15, 2020 2:20
To: [hidden email]
Subject: Is there any way to set the location of the history for the spark-shell per session?

In my team , we get elevated access to our Spark cluster using a common
username which means that we all share the same history. I am not sure if
this is common , but unfortunately there is nothing I can do about it.

Is there any option to set the location of the history?  I am looking for
something like spark-shell --history=...path or something similar that can
be reused for other sessions.

By default , history seems to be stored in $HOME/.scala_history and I did
not see anything in the documentation about it , but maybe there is an
undocumented way to do it.

Thanks for your help!



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Is there any way to set the location of the history for the spark-shell per session?

Yeikel
Thank you. That's what I was looking for.

I only found a PR from Scala when I googled it , so if you remember your
source , please do so.

Thanks!




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Is there any way to set the location of the history for the spark-shell per session?

ZHANG Wei
You are welcome!

It's not in Spark sourcecode. It's in Scala source:

https://github.com/scala/scala/blob/2.11.x/src/repl-jline/scala/tools/nsc/interpreter/jline/FileBackedHistory.scala#L26

Reference Code:

  // For a history file in the standard location, always try to restrict permission,
  // creating an empty file if none exists.
  // For a user-specified location, only lock down permissions if we're the ones
  // creating it, otherwise responsibility for permissions is up to the caller.
  protected lazy val historyFile: File = File {
    propOrNone("scala.shell.histfile").map(Path.apply) match {
      case Some(p) => if (!p.exists) secure(p) else p
      case None => secure(Path(userHome) / FileBackedHistory.defaultFileName)
    }
  }

Cheers,
-z

________________________________________
From: Yeikel <[hidden email]>
Sent: Friday, April 17, 2020 5:24
To: [hidden email]
Subject: Re: Is there any way to set the location of the history for the spark-shell per session?

Thank you. That's what I was looking for.

I only found a PR from Scala when I googled it , so if you remember your
source , please do so.

Thanks!




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]