yq refresher

 

Reading YAML in Bash with Examples

For YAML files, you'll want to use yq (the YAML equivalent of jq). 

Sample YAML File

Save this as config.yaml:

dev: 
   url: http://dev.com
test:
   url: http://test.com

Using yq 

Extract dev URL

$ yq '.dev.url' config.yaml

Output:

http://dev.com

Extract test URL

$ yq '.test.url' config.yaml

Output:

http://test.com

Extract both URLs

$ yq '.[] | .url' config.yaml

Output:

http://dev.com
http://test.com

Store in variables

$ DEV_URL=$(yq '.dev.url' config.yaml)
$ TEST_URL=$(yq '.test.url' config.yaml)

$ echo "Dev: $DEV_URL"
$ echo "Test: $TEST_URL"

Output:

Dev: http://dev.com
Test: http://test.com

Extract both conditionally (if they exist)

$ yq 'keys[] as $env | {env: $env, url: .[$env].url}' config.yaml

Output:

env: dev
url: http://dev.com
---
env: test
url: http://test.com


Load all environments into variables

#!/bin/bash

# Using yq
DEV_URL=$(yq '.dev.url' config.yaml)
TEST_URL=$(yq '.test.url' config.yaml)

echo "Dev URL: $DEV_URL"
echo "Test URL: $TEST_URL"

# Use them in commands
curl $DEV_URL
curl $TEST_URL


Loop through all environments
#!/bin/bash

yq 'keys[]' config.yaml | while read env; do
    url=$(yq ".${env}.url" config.yaml)
    echo "$env: $url"
done

Output:

dev: http://dev.com
test: http://test.com

Bottom line: Use yq if you can—it's much more reliable for YAML. The grep/awk method works for simple cases but breaks with complex YAML structures.


yq Array Tutorial with Examples & Output

yq is the YAML equivalent of jq. Works the same way but for YAML files. Let me show you all the main patterns.

Sample YAML Data

Simple list:

fruits:
  - apple
  - banana
  - cherry
  - date

List of objects:

students:
  - id: 1
    name: Alice
    score: 85
    dept: engineering
  - id: 2
    name: Bob
    score: 92
    dept: sales
  - id: 3
    name: Charlie
    score: 78
    dept: engineering
  - id: 4
    name: Diana
    score: 95
    dept: management

Nested arrays:

teams:
  - name: Engineering
    members:
      - Alice
      - Charlie
      - Eve
  - name: Sales
    members:
      - Bob
      - Diana

Basic Array Access

Access by index

Save as data.yaml:

numbers:
  - 10
  - 20
  - 30
  - 40
  - 50

$ yq '.numbers[0]' data.yaml

Output:

10

Last element

$ yq '.numbers[-1]' data.yaml

Output:

50

Slice (elements 1-3)

$ yq '.numbers[1:3]' data.yaml

Output:

- 20
- 30

Slice to end

$ yq '.numbers[2:]' data.yaml

Output:

- 30
- 40
- 50

Array length

$ yq '.numbers | length' data.yaml

Output:

5

Iterating Arrays

Save as students.yaml:

students:
  - id: 1
    name: Alice
    score: 85
  - id: 2
    name: Bob
    score: 92
  - id: 3
    name: Charlie
    score: 78

Iterate and output each element

$ yq '.students[]' students.yaml

Output:

id: 1
name: Alice
score: 85
---
id: 2
name: Bob
score: 92
---
id: 3
name: Charlie
score: 78

Get specific field from each object

$ yq '.students[].name' students.yaml

Output:

Alice
Bob
Charlie

Get multiple fields (one per line)

$ yq '.students[] | .name, .score' students.yaml

Output:

Alice
85
Bob
92
Charlie
78

Filtering Arrays with select

Filter by numeric comparison

$ yq '.students[] | select(.score > 80)' students.yaml

Output:

id: 1
name: Alice
score: 85
---
id: 2
name: Bob
score: 92

Filter by string value

$ yq '.students[] | select(.name == "Bob")' students.yaml

Output:

id: 2
name: Bob
score: 92

Filter with multiple conditions (AND)

$ yq '.students[] | select(.dept == "engineering" and .score > 80)' students.yaml

(Assuming your YAML has a dept field)

Output:

id: 1
name: Alice
score: 85
dept: engineering

Filter and return as array

$ yq '[.students[] | select(.score > 80)]' students.yaml

Output:

- id: 1
  name: Alice
  score: 85
- id: 2
  name: Bob
  score: 92

Transforming Arrays with map

Transform each element

numbers:
  - 1
  - 2
  - 3
  - 4
  - 5

$ yq '.numbers | map(. * 2)' numbers.yaml

Output:

- 2
- 4
- 6
- 8
- 10

Extract fields from each object

$ yq '.students | map(.name)' students.yaml

Output:

- Alice
- Bob
- Charlie

Create new objects with specific fields

$ yq '.students | map({name, score})' students.yaml

Output:

- name: Alice
  score: 85
- name: Bob
  score: 92
- name: Charlie
  score: 78

Add calculated field

$ yq '.students | map({name, score, grade: (if .score >= 90 then "A" elif .score >= 80 then "B" else "C" end)})' students.yaml

Output:

- name: Alice
  score: 85
  grade: B
- name: Bob
  score: 92
  grade: A
- name: Charlie
  score: 78
  grade: C

Combine filter and map

$ yq '.students | map(select(.score > 80) | {name, score})' students.yaml

Output:

- name: Alice
  score: 85
- name: Bob
  score: 92

Array Aggregation

Save as numbers.yaml:

numbers:
  - 10
  - 20
  - 30
  - 40
  - 50

Sum array values

$ yq '.numbers | add' numbers.yaml

Output:

150

Sum field values from objects

$ yq '.students | map(.score) | add' students.yaml

Output:

255

Average

$ yq '.numbers | (add / length)' numbers.yaml

Output:

30

Min and Max

$ yq '.numbers | min' numbers.yaml

Output:

10

$ yq '.numbers | max' numbers.yaml

Output:

50

Get unique values

numbers:
  - 1
  - 2
  - 2
  - 3
  - 3
  - 3
  - 4
  - 5
  - 5

$ yq '.numbers | unique' numbers.yaml

Output:

- 1
- 2
- 3
- 4
- 5

Count unique values

$ yq '.numbers | unique | length' numbers.yaml

Output:

5

Group by field

$ yq '.students | group_by(.dept)' students.yaml

Output:

- - id: 1
    name: Alice
    dept: engineering
  - id: 3
    name: Charlie
    dept: engineering
- - id: 2
    name: Bob
    dept: sales

Count occurrences

$ yq '.numbers | group_by(.) | map({value: .[0], count: length})' numbers.yaml

Output:

- value: 1
  count: 1
- value: 2
  count: 2
- value: 3
  count: 3
- value: 4
  count: 1
- value: 5
  count: 2

Array Manipulation

Add element to array

$ yq '.numbers += [60]' numbers.yaml

Output:

numbers:
  - 10
  - 20
  - 30
  - 40
  - 50
  - 60

Append array to array

$ yq '.numbers += [60, 70]' numbers.yaml

Output:

numbers:
  - 10
  - 20
  - 30
  - 40
  - 50
  - 60
  - 70

Remove duplicates

$ yq '.numbers | unique' numbers.yaml

Output:

- 1
- 2
- 3
- 4
- 5

Flatten nested arrays

nested:
  - - 1
    - 2
  - - 3
    - 4
  - - 5

$ yq '.nested | flatten' nested.yaml

Output:

- 1
- 2
- 3
- 4
- 5

Sort array

$ yq '.numbers | sort' numbers.yaml

Output:

- 1
- 1
- 2
- 3
- 4
- 5
- 6
- 9

Sort objects by field

$ yq '.students | sort_by(.score)' students.yaml

Output:

- id: 3
  name: Charlie
  score: 78
- id: 1
  name: Alice
  score: 85
- id: 2
  name: Bob
  score: 92

Sort descending

$ yq '.students | sort_by(.score) | reverse' students.yaml

Output:

- id: 2
  name: Bob
  score: 92
- id: 1
  name: Alice
  score: 85
- id: 3
  name: Charlie
  score: 78

Reverse array

$ yq '.numbers | reverse' numbers.yaml

Output:

- 50
- 40
- 30
- 20
- 10

Working with Nested Arrays

Save as teams.yaml:

teams:
  - name: Engineering
    members:
      - Alice
      - Charlie
      - Eve
  - name: Sales
    members:
      - Bob
      - Diana

Access nested array element

$ yq '.teams[0].members[1]' teams.yaml

Output:

Charlie

Get all members from all teams

$ yq '.teams[] | .members[]' teams.yaml

Output:

Alice
Charlie
Eve
Bob
Diana

Count total members

$ yq '.teams[] | .members | length' teams.yaml

Output:

3
2

Flatten all members into single array

$ yq '.teams | map(.members) | flatten' teams.yaml

Output:

- Alice
- Charlie
- Eve
- Bob
- Diana

Create list of team and member pairs

$ yq '.teams[] | .name as $team | .members[] | {team: $team, member: .}' teams.yaml

Output:

team: Engineering
member: Alice
---
team: Engineering
member: Charlie
---
team: Engineering
member: Eve
---
team: Sales
member: Bob
---
team: Sales
member: Diana

Common Array Patterns

Check if array contains value

$ yq '.fruits | contains(["apple", "cherry"])' data.yaml

Output:

true

Check if value is in array

$ yq '. as $array | "apple" | IN($array[])' data.yaml

Output:

true

Find index of element

$ yq '.fruits | index("cherry")' data.yaml

Output:

2

Check if any element matches condition

$ yq '.students | any(.score > 90)' students.yaml

Output:

true

Check if all elements match condition

$ yq '.students | all(.score > 70)' students.yaml

Output:

true

Practical Real-World Examples

Extract top 3 students by score:

$ yq '.students | sort_by(.score) | reverse | .[0:3] | map({name, score})' students.yaml

Output:

- name: Bob
  score: 92
- name: Alice
  score: 85
- name: Charlie
  score: 78

Convert array to comma-separated string:

$ yq '.fruits | join(", ")' data.yaml

Output:

apple, banana, cherry, date

Split string into array:

$ yq '"apple,banana,cherry" | split(",")'

Output:

- apple
- banana
- cherry

Generate summary report:

$ yq '.students | {total: length, average_score: (map(.score) | add / length), top_student: (sort_by(.score) | reverse | .[0].name)}' students.yaml

Output:

total: 3
average_score: 85
top_student: Bob

Filter and output as CSV:

$ yq -o=csv '.students | map(select(.score > 80)) | map([.id, .name, .score])' students.yaml

Output:

1,Alice,85
2,Bob,92

Key Differences from jq

Feature jq yq
Input JSON YAML
Output format -r for raw -o=csv, -o=json, -o=yaml
Iterating .[] .[] (same)
Filtering select() select() (same)
Multiple files jq file1 file2 yq file1 file2

Useful yq Flags

yq -r              # Raw output
yq -o=json         # Output as JSON
yq -o=csv          # Output as CSV
yq -o=tsv          # Output as TSV
yq -i              # In-place edit (modify file)
yq -s              # Slurp (read whole file as array)
yq '.field | @base64'  # Encode as base64
yq '.field | @uri'     # Encode as URI

Key Takeaway: yq syntax is almost identical to jq, just swap JSON for YAML. All the array patterns work the same way!

What array operation in YAML do you need to do?


Comments

Post a Comment

Popular posts from this blog

Windows SSH: Permissions for 'private-key' are too open

Image
Due to a security permission constraints on your folder,  when using ssh via pem file, it needs to be secured. To do what, place your file in a folder and then configure the necessary permission.  right click on the folder, right click on properties -> security -> Advance tab (as shown in figure below). Once you click on Apply, try to re-connect again.
Read more

NodeJS: Error: spawn EINVAL in window for node version 20.20 and 18.20

Encounter this issue when running autorest (that uses nodejs).  I am using node version:  v18.20.2 After i revert to use node v18.12.0, then i was able to get my app to run without this error. Since i am working with Azure devops yaml, i was able to use the following to resolve my issue.  - task : NodeTool@0   inputs :     versionSource : 'spec'     versionSpec : '18.18.0'
Read more