Pyspark Kafka Structured Stream not working.

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Pyspark Kafka Structured Stream not working.

Vijayant Kumar

Hi All,

 

I am getting the below error while using Pyspark Structured Streaming from Kafka Producer.

 

20/05/06 11:51:16 ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - kafka.common.InvalidConfigException: client.id Python Kafka streamer is illegal, contains a character other than ASCII alphanumerics, '.', '_' and '-'

 

I am using the below code to get the messages:

 

    broker='vm105:2181'

    topic='Hello-Kafka'

    print 'broker topic is ',broker,topic

    kvs = KafkaUtils.createStream(ssc, \

                                  broker, \

                                  "Python Kafka streamer",{topic:1})

 

And my Submit command is like below :-

spark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.4.4.jar test_kafka.py vm105:2181 Hello-Kafka

 

Can any one help me what am I missing. ?

 

Thanks,

Vijayant


This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to [hidden email]. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You

Reply | Threaded
Open this post in threaded view
|

Re: Pyspark Kafka Structured Stream not working.

Jungtaek Lim-2
Hi,

1. You seem to use DStream (Spark Streaming), not Structured Streaming.
2. I'm not familiar with pyspark, but looks like the error message is very clear - Kafka doesn't allow such name for "client.id". The error message guides the naming rule, so you may need to be adopted with such convention. (e.g. no space)

Hope this helps,

Thanks,
Jungtaek Lim (HeartSaVioR)

On Wed, May 6, 2020 at 5:36 PM Vijayant Kumar <[hidden email]> wrote:

Hi All,

 

I am getting the below error while using Pyspark Structured Streaming from Kafka Producer.

 

20/05/06 11:51:16 ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - kafka.common.InvalidConfigException: client.id Python Kafka streamer is illegal, contains a character other than ASCII alphanumerics, '.', '_' and '-'

 

I am using the below code to get the messages:

 

    broker='vm105:2181'

    topic='Hello-Kafka'

    print 'broker topic is ',broker,topic

    kvs = KafkaUtils.createStream(ssc, \

                                  broker, \

                                  "Python Kafka streamer",{topic:1})

 

And my Submit command is like below :-

spark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.4.4.jar test_kafka.py vm105:2181 Hello-Kafka

 

Can any one help me what am I missing. ?

 

Thanks,

Vijayant


This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to [hidden email]. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You

Reply | Threaded
Open this post in threaded view
|

RE: [E] Re: Pyspark Kafka Structured Stream not working.

Vijayant Kumar

Hi Jungtek,

 

Thanks for the response. It appears to be #1.

I will appreciate if you can share some sample command to submit the Spark application.?

 

From: Jungtaek Lim [mailto:[hidden email]]
Sent: Wednesday, May 06, 2020 8:24 PM
To: Vijayant Kumar <[hidden email]>
Cc: [hidden email]
Subject: [E] Re: Pyspark Kafka Structured Stream not working.

 

[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.

Hi,

 

1. You seem to use DStream (Spark Streaming), not Structured Streaming.

2. I'm not familiar with pyspark, but looks like the error message is very clear - Kafka doesn't allow such name for "client.id". The error message guides the naming rule, so you may need to be adopted with such convention. (e.g. no space)

 

Hope this helps,

 

Thanks,

Jungtaek Lim (HeartSaVioR)

 

On Wed, May 6, 2020 at 5:36 PM Vijayant Kumar <[hidden email]> wrote:

Hi All,

 

I am getting the below error while using Pyspark Structured Streaming from Kafka Producer.

 

20/05/06 11:51:16 ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - kafka.common.InvalidConfigException: client.id Python Kafka streamer is illegal, contains a character other than ASCII alphanumerics, '.', '_' and '-'

 

I am using the below code to get the messages:

 

    broker='vm105:2181'

    topic='Hello-Kafka'

    print 'broker topic is ',broker,topic

    kvs = KafkaUtils.createStream(ssc, \

                                  broker, \

                                  "Python Kafka streamer",{topic:1})

 

And my Submit command is like below :-

spark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.4.4.jar test_kafka.py vm105:2181 Hello-Kafka

 

Can any one help me what am I missing. ?

 

Thanks,

Vijayant


This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to [hidden email]. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You


This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to [hidden email]. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You

Reply | Threaded
Open this post in threaded view
|

Re: [E] Re: Pyspark Kafka Structured Stream not working.

Jungtaek Lim-2
It's not either 1 or 2. Both two items are applied. I haven't played with DStream + pyspark but given the error message is clear you'll probably want to change the client.id "Python Kafka streamer" to accommodate the naming convention guided in error message.

On Thu, May 7, 2020 at 3:55 PM Vijayant Kumar <[hidden email]> wrote:

Hi Jungtek,

 

Thanks for the response. It appears to be #1.

I will appreciate if you can share some sample command to submit the Spark application.?

 

From: Jungtaek Lim [mailto:[hidden email]]
Sent: Wednesday, May 06, 2020 8:24 PM
To: Vijayant Kumar <[hidden email]>
Cc: [hidden email]
Subject: [E] Re: Pyspark Kafka Structured Stream not working.

 

[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.

Hi,

 

1. You seem to use DStream (Spark Streaming), not Structured Streaming.

2. I'm not familiar with pyspark, but looks like the error message is very clear - Kafka doesn't allow such name for "client.id". The error message guides the naming rule, so you may need to be adopted with such convention. (e.g. no space)

 

Hope this helps,

 

Thanks,

Jungtaek Lim (HeartSaVioR)

 

On Wed, May 6, 2020 at 5:36 PM Vijayant Kumar <[hidden email]> wrote:

Hi All,

 

I am getting the below error while using Pyspark Structured Streaming from Kafka Producer.

 

20/05/06 11:51:16 ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - kafka.common.InvalidConfigException: client.id Python Kafka streamer is illegal, contains a character other than ASCII alphanumerics, '.', '_' and '-'

 

I am using the below code to get the messages:

 

    broker='vm105:2181'

    topic='Hello-Kafka'

    print 'broker topic is ',broker,topic

    kvs = KafkaUtils.createStream(ssc, \

                                  broker, \

                                  "Python Kafka streamer",{topic:1})

 

And my Submit command is like below :-

spark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.4.4.jar test_kafka.py vm105:2181 Hello-Kafka

 

Can any one help me what am I missing. ?

 

Thanks,

Vijayant


This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to [hidden email]. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You


This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to [hidden email]. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You