I have an Structured Streaming Application that reads from kafka, performs some aggregations and writes in S3 in parquet format.
Everything seems to work great except that from time to time I get a checkpoint error, at the beginning I thought it was a random error but it happened more than 3 times already in a few days
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://xxx/xxx/validation-checkpoint/offsets/.140.11adef9a-7636-4752-9e6c-48d605a9cca5.tmp
Does this happen to anyone else?
Thanks in advance.
This is the full error :
Structured Streaming is simply not working when checkpoint location is on S3 due to it's read-after-write consistency.
Please choose an HDFS compliant filesystem and it will work like a charm.
On Wed, Sep 16, 2020 at 4:12 PM German Schiavon <[hidden email]> wrote:
Makes sense, thanks a lot!
On Thu, 17 Sep 2020 at 11:51, Gabor Somogyi <[hidden email]> wrote:
|Free forum by Nabble||Edit this page|