Intro:

Hi, It has been a while since my last story. This one isn’t going to tell anything special about infosec, in fact nothing with that topic in here. I will be presenting how I managed to automate a task with a malformed json data, especially updating repetitive values for the same key. Let’s start: I won’t go through fundamentals, but some specific parameters will be mentioned here. Yesterday, I got some json data that was like this:

  • JSON Data :
{
"id": "30",
"key2": "random value..",
"key3": "searched value1" 
},
{
"id": "31",
"key2": "random value..",
"key3": "searched value3"
},
…
…

The snippet above describes some json objects. When processing them directly with jq a lightweight json processor. See refs. We got into some errors. As neither it was a well constructed objects nor a complete array. So, using the basic . jq filter to list the data throws errors.

cat data | jq '.'

Initially with the above command we would print a pretty formatted json. but instead we hit the following parser error : parse error: Expected value before ‘,’ at line 5, column 10

Solve issue 1:

Digging more in documentation and json structure, decided to use it as arrays of object, so adding square brackets [].

[
..our data
{
...
}
]

With that the filter above works great. So will now extract all key3 values. We need to list what inside the array first, then filter only all keys we were interested in. Consider using the following filter:

cat data | jq '.[].key3'

All key3 values would be printed in between double quotes. So, using -r switch to have only raw values. so combining with something like this:

cat data | jq -r '.[].key3'

We got now our values, we could manipulate them the way you prefer, however for me, I have to replace these with some specific ones. So searched value1 will be for instance testing1.

Recap:

Untill now I had extracted all key3 values, save them into file let’s name it values.txt. Got new file with the replacement values within newValues.txt.

Resolve issue 2:

I need to replace now all the key3 values in data with those from newValues.txt.

After some search I didn’t find something interesting, but got some hints about using the select filter. Let’s say you want to search a specific value within a specific object. We need to select from all the objects the key that matches our identifier id. So, basically the following code would render our first object:

cat data | jq '.[] | select(.id=="30")'

The idea now is to update the key3 value filtered within the current object. To make this happen I would simply pipe it into another jq with the following : jq '.key3="newValue1"'.

Combining the two: extract the object, update the value :

cat data | jq '.[] | select(.id=="30")' | jq '.key3="newValue1"'

Resolve issue 3:

This is enough for one value, but what about other values in newValues.txt the idea is clear we need to loop into those with some scripting. I prefer since I’m in linux to use bash. So, the idea is :

  • Read line by line the new values from newValues.txt
  • Perform some filtering on data file using jq.
  • Passing each line as argument to update the key3 value.

Basic while loop reading line by line values from newValues.txt would do the job:

while IFS= read -r myline;
do 
     ...MAGIC HERE...
done < newValues.txt

To loop through all objects by id I have created a global variable named idv with the first object id value. So, idv initially will get idv=30 . This will be passed as argument to jq select filter. And we should increment its value as it passes through objects one by one.

The syntax is this : jq --arg argname value '...filtersHere', so we would have something as below:

cat data | jq --arg test $idv '.[] | select(.id==$test)'

jq argument takes value from our variable $idv, with the same logic, we can use values from inside newValues.txt. The syntax would be:

jq --arg val "$myline" '.key3=$val'

Note:

The double quotes in $myline, makes it act as literal string, as we don’t know how they are formatted.

The magic line is :

cat data | jq --arg test $idv '.[] | select(.id==$test)' | jq --arg val "$myline" '.key=$val' | tee -a NewData

Final result: (our wrapper)

The resulting script is : simo.sh (didn’t find the appealing name for this)

#! /bin/bash
idv=30
myfunction(){
while IFS= read -r myline;
do 
     cat data | jq --arg test $idv '.[] | select(.id==$test)' \
              | jq --arg val "$myline" '.key=$val' \
              | tee -a NewData
     idv=$((idv+1))
done < newValues.txt
}
myfunction
Refs:

jq documentation