Do the flat files need to have the same json format?

Question

You can see different responses from the API

illustrate: In the screenshot shared above, I compared two different files. On the left is the data Page number 1, and on the right, I get data for page number 1080. You can see that on page 1, I received the shipment in the form of array [ ], which is empty, while on page 1080, I received the shipment in the form of object {}. Furthermore, in this object, I get an array of pd-option [ ]

This will help you understand the arrays and objects shipped

So my question is, does the file format in ADF have to be the same as I want to convert 1700 files to flatten or not, then how can we provide different formats in Azure ADF to flatten the files?

Answer 1

• To flatten files, you need to make the files have the same format. You can use Azure Data Flows to flatten data or any data-related operations.

• If the data are in different formats, you can also use transformations to convert them into the same format.

• The following is an example of the same. Assume that file1.json has the following data:

{
    "id":"A1",
    "shipment":[]
}

• file2.json has the following data:

{
    "id":"A1",
    "shipment":{ "id": 171700, "order_print_process_id": 170541 }
}

• You cannot use the same logic to flatten the files here because they are of different types. When you read file1.json you can see the projection of the file:

• Due to different shipping attribute values, the projection of file2.json will be different from file1.json.

• You can flatten file1.json, but you will need to use different logic to extract shipments from file2.json. Use the selection transformation as shown below to extract the attributes.

• Therefore, you need to process these documents based on the shipment value. Using the same logic for both files won't work.