Spark Kinesis Connector SSL issue

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark Kinesis Connector SSL issue

shzshi

Hi Team,

we are trying access the endpoint thought library mentioned below and we get the SSL error i think internally it use KCL library. so if I have to skip the certificate is it possible through KCL utils call ? because I do not find any provision to do that to set no-verify=false within spark streaming kinesis library like we can do with KCL. Can you please help me with the same.

compile("org.apache.spark:spark-streaming-kinesis-asl_2.11:2.3.0")

{ exclude group: 'org.apache.spark', module: 'spark-streaming_2.11' }

Caused by: javax.net.ssl.SSLPeerUnverifiedException: Certificate for kinesis-endpoint> doesn't match any of the subject alternative names: [kinesis-fips.us-east-1.amazonaws.com, *.kinesis.us-east-1.vpce.amazonaws.com, kinesis.us-east-1.amazonaws.com]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:467)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:397)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
at shade.com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at shade.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
at shade.com.amazonaws.http.conn.$Proxy18.connect(Unknown Source)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at shade.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at shade.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1238)
at shade.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
... 20 more

 

 

Shashi

Reply | Threaded
Open this post in threaded view
|

Re: Spark Kinesis Connector SSL issue

shzshi
Hi team,

please help , we are kind of blocked here.

Cheers,
Shashi



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark Kinesis Connector SSL issue

email
Can you call this service with regular code(No Spark)?


---- On Mon, 07 Jan 2019 02:42:48 -0800 [hidden email] wrote ----

Hi team,

please help , we are kind of blocked here.

Cheers,
Shashi



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

RE: Re: Spark Kinesis Connector SSL issue

shzshi

Hi Valdes,

 

Thank you for your response, to answer to your question. yes I can

 

@ben : correct me if I am wrong.

 

Cheers,

Shashi

 

Shashikant Bangera | DevOps Engineer

Payment Services DevOps Engineering

Email: [hidden email]

Group email: [hidden email]

Tel: +44 (0)

Mob: +44 (0) 7440783885

 

 

From: yeikel valdes [mailto:[hidden email]]
Sent: 07 January 2019 12:15
To: Shashikant Bangera <[hidden email]>
Cc: [hidden email]
Subject: [EXTERNAL] Re: Spark Kinesis Connector SSL issue

 

CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected emails.

 

Can you call this service with regular code(No Spark)?

 


---- On Mon, 07 Jan 2019 02:42:48 -0800 [hidden email] wrote ----

Hi team,

please help , we are kind of blocked here.

Cheers,
Shashi



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

 

Reply | Threaded
Open this post in threaded view
|

RE: Re: Spark Kinesis Connector SSL issue

email
Any chance you can share a minimum example to replicate the issue?


---- On Mon, 07 Jan 2019 04:17:44 -0800 [hidden email] wrote ----

Hi Valdes,

 

Thank you for your response, to answer to your question. yes I can

 

@ben : correct me if I am wrong.

 

Cheers,

Shashi

 

Shashikant Bangera | DevOps Engineer

Payment Services DevOps Engineering

Email: [hidden email]

Group email: [hidden email]

Tel: +44 (0)

Mob: +44 (0) 7440783885

 

 

From: yeikel valdes [mailto:[hidden email]]
Sent: 07 January 2019 12:15
To: Shashikant Bangera <[hidden email]>
Cc: [hidden email]
Subject: [EXTERNAL] Re: Spark Kinesis Connector SSL issue

 

CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected emails.

 

Can you call this service with regular code(No Spark)?

 


---- On Mon, 07 Jan 2019 02:42:48 -0800 [hidden email] wrote ----

Hi team,

please help , we are kind of blocked here.

Cheers,
Shashi



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

 


Reply | Threaded
Open this post in threaded view
|

RE: RE: Re: Spark Kinesis Connector SSL issue

shzshi

Hi,

 

The issue is that the KCL inside the Spark Streaming connector does not provide a way to pass KCL configuration in, which means we can’t supply configuration to disable SSL cert checks. In a typical (non-Spark Streaming) KCL app, we can instantiate the KCL via:

 

       Worker worker = new Worker.Builder()

                .config(kclConfig) , //we can specify to disable SSL certs here.

                .kinesisClient(kinesisClient)        

                .recordProcessorFactory(processorFactory)

                .build();

 

However with the Kinesis Spark Streaming consumer, we do not have the ability to set this. Our only interface with Kinesis is:

 

KinesisUtils.createStream(context, appName, streamName, serviceEndpoint, regionName,

        InitialPositionInStream.TRIM_HORIZON, checkpoint, StorageLevel.MEMORY_AND_DISK_2())

 

and so we can’t pass config anywhere for the KCL to read.

 

Regards,

 

Ben

 

 

Shashikant Bangera | DevOps Engineer

Payment Services DevOps Engineering

Email: [hidden email]

Group email: [hidden email]

Tel: +44 (0)

Mob: +44 (0) 7440783885

 

 

From: Ben Watson
Sent: 07 January 2019 12:32
To: yeikel valdes <[hidden email]>; Shashikant Bangera <[hidden email]>
Cc: [hidden email]
Subject: RE: [EXTERNAL] RE: Re: Spark Kinesis Connector SSL issue

 

Hi,

 

The issue is that the KCL inside the Spark Streaming connector does not provide a way to pass KCL configuration in, which means we can’t supply configuration to disable SSL cert checks. In a typical (non-Spark Streaming) KCL app, we can instantiate the KCL via:

 

       Worker worker = new Worker.Builder()

                .config(kclConfig) , //we can specify to disable SSL certs here.

                .kinesisClient(kinesisClient)        

                .recordProcessorFactory(processorFactory)

                .build();

 

However with the Kinesis Spark Streaming consumer, we do not have the ability to set this. Our only interface with Kinesis is:

 

KinesisUtils.createStream(context, appName, streamName, serviceEndpoint, regionName,

        InitialPositionInStream.TRIM_HORIZON, checkpoint, StorageLevel.MEMORY_AND_DISK_2())

 

and so we can’t pass config anywhere for the KCL to read.

 

Regards,

 

Ben

 

From: yeikel valdes [mailto:[hidden email]]
Sent: 07 January 2019 12:21
To: Shashikant Bangera <[hidden email]>
Cc: [hidden email]; Ben Watson <[hidden email]>
Subject: [EXTERNAL] RE: Re: Spark Kinesis Connector SSL issue

 

CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected emails.

 

Any chance you can share a minimum example to replicate the issue?

 


---- On Mon, 07 Jan 2019 04:17:44 -0800 [hidden email] wrote ----

Hi Valdes,

 

Thank you for your response, to answer to your question. yes I can

 

@ben : correct me if I am wrong.

 

Cheers,

Shashi

 

Shashikant Bangera | DevOps Engineer

Payment Services DevOps Engineering

Email: [hidden email]

Group email: [hidden email]

Tel: +44 (0)

Mob: +44 (0) 7440783885

 

 

From: yeikel valdes [mailto:[hidden email]]
Sent: 07 January 2019 12:15
To: Shashikant Bangera <[hidden email]>
Cc: [hidden email]
Subject: [EXTERNAL] Re: Spark Kinesis Connector SSL issue

 

CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected emails.

 

Can you call this service with regular code(No Spark)?

 


---- On Mon, 07 Jan 2019 02:42:48 -0800 [hidden email] wrote ----

Hi team,

please help , we are kind of blocked here.

Cheers,
Shashi



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]