Well the meta information is in the file so I am not surprised that it reads the file, but it should not read all the content, which is probably also not happening.
On 24. Oct 2017, at 18:16, Siva Gudavalli <[hidden email]> wrote:
I found a workaround, when I create Hive Table using Spark “saveAsTable”, I see filters being pushed down.
-> other approaches I tried where filters are not pushed down Is,
1) when I create Hive Table upfront and load orc into it using Spark SQL
2) when I create orc files using spark SQL and then create Hive External Table
If my understanding is correct, when I use saveAsTable spark is using & also registering Hive Metastore with its custom Serde and Is able to pushdown filters.
Please correct me.
When i am writing Orc to hive using “saveAsTable”, is there any way I can provide details about Orc Files.
for instance: stripe.size, can i create bloom filters etc…
|Free forum by Nabble||Edit this page|