Broadcast variable exception - too many open files

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Broadcast variable exception - too many open files

Sourav Chandra
Hi,

I am trying to persist processed DStream in Cassandra 3 tables.

For each of table I have corresponding class and created 3 broadcast variables.

Now while trying to run the application it throws 

java.io.FileNotFoundException: /tmp/spark-local-20140204172552-9eed/0c/broadcast_2 (Too many open files)


What could the possible reasons?

Thanks,

--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com

Reply | Threaded
Open this post in threaded view
|

Re: Broadcast variable exception - too many open files

Sebastian Schelter
This usually happens when you exceed the number of open files that your
OS allows. You have to increase that number.

--sebastian

On 02/04/2014 01:04 PM, Sourav Chandra wrote:

> Hi,
>
> I am trying to persist processed DStream in Cassandra 3 tables.
>
> For each of table I have corresponding class and created 3 broadcast
> variables.
>
> Now while trying to run the application it throws
>
> java.io.FileNotFoundException:
> /tmp/spark-local-20140204172552-9eed/0c/broadcast_2 (Too many open files)
>
>
> What could the possible reasons?
>
> Thanks,
>

Reply | Threaded
Open this post in threaded view
|

Re: Broadcast variable exception - too many open files

Anita Tailor
In reply to this post by Sourav Chandra
you can check "ulimit -n"


On 4 February 2014 17:34, Sourav Chandra <[hidden email]> wrote:
Hi,

I am trying to persist processed DStream in Cassandra 3 tables.

For each of table I have corresponding class and created 3 broadcast variables.

Now while trying to run the application it throws 

java.io.FileNotFoundException: /tmp/spark-local-20140204172552-9eed/0c/broadcast_2 (Too many open files)


What could the possible reasons?

Thanks,

--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com


Reply | Threaded
Open this post in threaded view
|

Re: Broadcast variable exception - too many open files

Marek Wiewiorka
In reply to this post by Sourav Chandra
Hi - try increasing hard limits at os level.


2014-02-04 Sourav Chandra <[hidden email]>:
Hi,

I am trying to persist processed DStream in Cassandra 3 tables.

For each of table I have corresponding class and created 3 broadcast variables.

Now while trying to run the application it throws 

java.io.FileNotFoundException: /tmp/spark-local-20140204172552-9eed/0c/broadcast_2 (Too many open files)


What could the possible reasons?

Thanks,

--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: <a href="tel:%2B91%2080%204121%208723" value="+918041218723" target="_blank">+91 80 4121 8723

m: <a href="tel:%2B91%20988%20699%203746" value="+919886993746" target="_blank">+91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com


Reply | Threaded
Open this post in threaded view
|

Re: Broadcast variable exception - too many open files

Sourav Chandra
Hi,

I did ulimit -n , it shows 1024. Is it not sufficient?

Earlier when I did not have any broadcast variables, there was no error.

Why broadcast variable caused this?

Thanks,
Sourav


On Tue, Feb 4, 2014 at 5:40 PM, Marek Wiewiorka <[hidden email]> wrote:
Hi - try increasing hard limits at os level.


2014-02-04 Sourav Chandra <[hidden email]>:

Hi,

I am trying to persist processed DStream in Cassandra 3 tables.

For each of table I have corresponding class and created 3 broadcast variables.

Now while trying to run the application it throws 

java.io.FileNotFoundException: /tmp/spark-local-20140204172552-9eed/0c/broadcast_2 (Too many open files)


What could the possible reasons?

Thanks,

--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: <a href="tel:%2B91%2080%204121%208723" value="+918041218723" target="_blank">+91 80 4121 8723

m: <a href="tel:%2B91%20988%20699%203746" value="+919886993746" target="_blank">+91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com





--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com

Reply | Threaded
Open this post in threaded view
|

Re: Broadcast variable exception - too many open files

Akhil Das
Hi Sourav,

If you do ulimit -n that will reset the limit for the current shell.

If you are using Linux and you got the permission error, you will need to raise the allowed limit in the /etc/limits.conf or /etc/security/limits.conf file (where the file is located depends on your specific Linux distribution).

For example to allow anyone on the machine to raise their number of open files up to 10000 add the line to the limits.conf file.

* hard nofile 10000

Then logout and relogin to your system and you should be able to do:

ulimit -n 10000


-
AkhilDas
CodeBreach.in
Reply | Threaded
Open this post in threaded view
|

Re: Broadcast variable exception - too many open files

Anita Tailor
value of broadcast variable is saved in a file on HTTP server in case of HttpBroadcast( default one). You can check files starting with "broadcast_" in /tmp ( or spark.local.dir) to verify number of files being created by broadcast variables in your case.



On 4 February 2014 18:03, Akhil Das <[hidden email]> wrote:
Hi Sourav,

If you do ulimit -n that will reset the limit for the current shell.

If you are using Linux and you got the permission error, you will need to raise the allowed limit in the /etc/limits.conf or /etc/security/limits.conf file (where the file is located depends on your specific Linux distribution).

For example to allow anyone on the machine to raise their number of open files up to 10000 add the line to the limits.conf file.

* hard nofile 10000

Then logout and relogin to your system and you should be able to do:

ulimit -n 10000


-
AkhilDas
CodeBreach.in