Access SBT with proxy

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Access SBT with proxy

Mayur Rustagi

export JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=yourserver -Dhttp.proxyPort=8080 -Dhttp.proxyUser=username -Dhttp.proxyPassword=password"
Also please use separate thread for different questions.

Regards

Mayur

Mayur Rustagi
Ph: +1 (760) 203 3257


On Thu, Mar 6, 2014 at 10:03 AM, Jeyaraj, Arockia R (Arockia) <[hidden email]> wrote:
Hi,

I am trying to setup Spark in windows for development environment. I get following error when I run sbt. Pl help me to resolve this issue. I am working for Verizon and am in my company network and can't access internet without proxy.

C:\Users>sbt
Getting org.fusesource.jansi jansi 1.11 ...
You probably access the destination server through a proxy server that is not we
ll configured.
You probably access the destination server through a proxy server that is not we
ll configured.
You probably access the destination server through a proxy server that is not we
ll configured.

:: problems summary ::
:::: WARNINGS
        Host repo.typesafe.com not found. url=<a href="http://repo.typesafe.com/typesafe/ ivy-releases/org.fusesource.jansi/jansi/1.11/ivys/ivy.xml" target="_blank">http://repo.typesafe.com/typesafe/
ivy-releases/org.fusesource.jansi/jansi/1.11/ivys/ivy.xml

        Host repo1.maven.org not found. url=<a href="http://repo1.maven.org/maven2/org/fu sesource/jansi/jansi/1.11/jansi-1.11.pom" target="_blank">http://repo1.maven.org/maven2/org/fu
sesource/jansi/jansi/1.11/jansi-1.11.pom

        Host repo1.maven.org not found. url=<a href="http://repo1.maven.org/maven2/org/fu sesource/jansi/jansi/1.11/jansi-1.11.jar" target="_blank">http://repo1.maven.org/maven2/org/fu
sesource/jansi/jansi/1.11/jansi-1.11.jar

                module not found: org.fusesource.jansi#jansi;1.11

        ==== local: tried

          C:\Users\v983654\.ivy2\local\org.fusesource.jansi\jansi\1.11\ivys\ivy.
xml

          -- artifact org.fusesource.jansi#jansi;1.11!jansi.jar:

          C:\Users\v983654\.ivy2\local\org.fusesource.jansi\jansi\1.11\jars\jans
i.jar

        ==== typesafe-ivy-releases: tried

          <a href="http://repo.typesafe.com/typesafe/ivy-releases/org.fusesource.jansi/ja nsi/1.11/ivys/ivy.xml" target="_blank">http://repo.typesafe.com/typesafe/ivy-releases/org.fusesource.jansi/ja
nsi/1.11/ivys/ivy.xml

        ==== Maven Central: tried

          http://repo1.maven.org/maven2/org/fusesource/jansi/jansi/1.11/jansi-1.
11.pom

          -- artifact org.fusesource.jansi#jansi;1.11!jansi.jar:

          http://repo1.maven.org/maven2/org/fusesource/jansi/jansi/1.11/jansi-1.
11.jar

                ::::::::::::::::::::::::::::::::::::::::::::::

                ::          UNRESOLVED DEPENDENCIES         ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: org.fusesource.jansi#jansi;1.11: not found

                ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
unresolved dependency: org.fusesource.jansi#jansi;1.11: not found
Error during sbt execution: Error retrieving required libraries
  (see C:\Users\v983654\.sbt\boot\update.log for complete log)
Error: Could not retrieve jansi 1.11

Thanks
Arockia Raja

-----Original Message-----
From: Matei Zaharia [mailto:[hidden email]]
Sent: Thursday, March 06, 2014 11:44 AM
To: [hidden email]
Subject: Re: Building spark with native library support

Is it an error, or just a warning? In any case, you need to get those libraries from a build of Hadoop for your platform. Then add them to the SPARK_LIBRARY_PATH environment variable in conf/spark-env.sh, or to your -Djava.library.path if launching an application separately.

These libraries just speed up some compression codecs BTW, so it should be fine to run without them too.

Matei

On Mar 6, 2014, at 9:04 AM, Alan Burlison <[hidden email]> wrote:

> Hi,
>
> I've successfully built 0.9.0-incubating on Solaris using sbt, following the instructions at http://spark.incubator.apache.org/docs/latest/ and it seems to work OK. However, when I start it up I get an error about missing Hadoop native libraries. I can't find any mention of how to build the native components in the instructions, how is that done?
>
> Thanks,
>
> --
> Alan Burlison
> --


Reply | Threaded
Open this post in threaded view
|

Re: Access SBT with proxy

Alan Burlison
On 06/03/2014 18:08, Mayur Rustagi wrote:

> export JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=yourserver
> -Dhttp.proxyPort=8080 -Dhttp.proxyUser=username
> -Dhttp.proxyPassword=password"

I had exactly the same problem and tried the above, it worked for some
of the components but the problem was that stuff that was pulled in from
git failed because some of the URLs in in the Spark wad are git:// ones
and not http:// ones. To get git to play nice through a HTTP proxy I had
to add the following to my .gitconfig:

----------
[http]
         proxy = http://yourserver:8080
[url "https://github.com/"]
         insteadOf = git://github.com/
----------

So you'll probably need to do that in addition to Mayur's suggestion.

--
Alan Burlison
--
Reply | Threaded
Open this post in threaded view
|

RE: Access SBT with proxy

Jeyaraj, Arockia R (Arockia)
Thanks Alan.

I am very new to Spark. I am trying to set Spark development environment in Windows. I added below mentioned "export" as "set" in sbt.bat file and tried, it was not working. Where will I see .gitconfig?

set JAVA_OPTS=%JAVA_OPTS% -Dhttp.proxyHost=myservername -Dhttp.proxyPort=8080 -Dhttp.proxyUser=username -Dhttp.proxyPassword=password

Thanks
Arockia Raja

-----Original Message-----
From: Alan Burlison [mailto:[hidden email]]
Sent: Thursday, March 06, 2014 12:29 PM
To: [hidden email]
Cc: Mayur Rustagi; Jeyaraj, Arockia R (Arockia)
Subject: Re: Access SBT with proxy

On 06/03/2014 18:08, Mayur Rustagi wrote:

> export JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=yourserver
> -Dhttp.proxyPort=8080 -Dhttp.proxyUser=username
> -Dhttp.proxyPassword=password"

I had exactly the same problem and tried the above, it worked for some of the components but the problem was that stuff that was pulled in from git failed because some of the URLs in in the Spark wad are git:// ones and not http:// ones. To get git to play nice through a HTTP proxy I had to add the following to my .gitconfig:

----------
[http]
         proxy = http://yourserver:8080
[url "https://github.com/"]
         insteadOf = git://github.com/
----------

So you'll probably need to do that in addition to Mayur's suggestion.

--
Alan Burlison
--
Reply | Threaded
Open this post in threaded view
|

Re: Access SBT with proxy

Alan Burlison
On 06/03/2014 20:30, Jeyaraj, Arockia R (Arockia) wrote:

> I am very new to Spark. I am trying to set Spark development
> environment in Windows. I added below mentioned "export" as "set" in
> sbt.bat file and tried, it was not working. Where will I see
> .gitconfig?

I'm only one jump ahead of you myself and I don't speak Windows so I'm
afraid I can't really help with the windows-specific ways of doing this.

--
Alan Burlison
--