Reading only the first level of JSON

I have a Dataset that contains a 2 level JSON String. Here is an example of what my json looks like:

""field1":"AAA","field2":"subField1":"000","subField2":"111","value":100.0"


When I parse it using the read().json() function of Spark datasets, I get the following result:

read().json()

field1, field2, value
AAA, [000,111], 100.0


I want to find a way to setup my Dataset Reader so it only parses the first level of my JSON, giving my the following result :

field1, field2, value
AAA, "subField1":"000","subField2":"111", 100.0


How can I achieve this?

1 Answer
1

you can easily achieve this by defining the schema explicitly

val schema = StructType(Array(StructField("field1",StringType,true),StructField("field2",StringType,true),StructField("value",StringType,true)))
val df = spark.read.schema(schema).json("path.json")
df.show(false)


Thanks for contributing an answer to Stack Overflow!

But avoid …

To learn more, see our tips on writing great answers.

Required, but never shown

Required, but never shown

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.