Event Hubs properties kvp-value adds " to strings

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Event Hubs properties kvp-value adds " to strings

Magnus Nilsson
Hello all,

I have this peculiar problem where quote " characters are added to the beginning and end of my string values.

I get data using Structured Streaming from an Azure Event Hub using a Scala Notebook in Azure Databricks.

The Dataframe schema received contain a property of type Map named "properties" containing string/string key value pairs. Only one pair in my case.

I input data from a C# program where the "properties" property in the EventData schema is a Dictionary<string, object> where I input a string/string property pair.

If I input a string value of _Hello_ in C# I get back a string value of _"Hello"_ in my Dataframe.

In the body tag of the Schema I input a UTF-8 byte[] and get back a UTF-8 byte[] which I cast to string, this turns out correct, no " are added to the string value.

If I try to pass in a UTF-8 string as a byte[] as the value of the KvP I get a serialization exception so that's a no go.

Any idea on why or where the quote characters "" are added to the value of the string when I read them in Spark?

Any ideas would greatly be appreciated.

//Magnus