Linux / Bash Text Processing Stream Editing with sed and Field Processing with awk

Linux / Bash Beginner 1 min read

Stream Editing with sed and Field Processing with awk

The `sed` (stream editor) command performs text transformations on an input stream. Its most common use is the substitution command `s/pattern/replacement/flags`, which replaces text matching a regex. The `g` flag makes substitutions global (all occurrences on a line). With `-i` it edits files in place, and `-n` with `p` prints only matching lines like a filter. `awk` is a full pattern-action language designed for structured text processing. It reads input line by line, automatically splitting each line into fields separated by the field separator (default: whitespace). Fields are accessed as `$1`, `$2`, etc., with `$0` being the full line and `NF` the number of fields. Built-in variables like `NR` (record number) and `FS` (field separator) make `awk` ideal for processing tabular data like `/etc/passwd` or CSV files. Together, `sed` and `awk` are indispensable for configuration management, log parsing, and data transformation in shell pipelines. Many tasks that would require a Python script can be accomplished with a single `awk` one-liner once you understand the pattern-action model.

Example

# ---- sed examples ----

# Replace first occurrence on each line
sed 's/foo/bar/' file.txt

# Replace all occurrences on each line (global flag)
sed 's/foo/bar/g' file.txt

# Case-insensitive substitution
sed 's/error/ERROR/Ig' logfile.txt

# Edit file in place (create backup with .bak)
sed -i.bak 's/old_host/new_host/g' config.conf

# Delete lines matching a pattern
sed '/^#/d' config.conf         # Remove comment lines
sed '/^$/d' file.txt            # Remove blank lines

# Print only lines 10-20
sed -n '10,20p' file.txt

# Insert a line before line 5
sed '5i\New line here' file.txt

# ---- awk examples ----

# Print the first field (column) of every line
awk '{print $1}' /etc/passwd

# Use : as field separator, print username and shell
awk -F: '{print $1, $7}' /etc/passwd

# Print lines where field 3 > 1000 (non-system users)
awk -F: '$3 >= 1000 {print $1, $3}' /etc/passwd

# Sum the sizes from ls -l output
ls -l | awk '{sum += $5} END {print "Total bytes:", sum}'

# Print line numbers and lines longer than 80 chars
awk 'length($0) > 80 {print NR": "$0}' file.txt

# Print fields in reverse order
awk '{for(i=NF;i>=1;i--) printf $i" "; print ""}' file.txt

# Count occurrences of each unique value in column 1
awk '{count[$1]++} END {for (k in count) print k, count[k]}' access.log

HTML

Full Editor

Related Resources

Linux / Bash Reference

Complete tag & property list

Linux / Bash How-To Guides

Step-by-step practical guides

Linux / Bash Exercises

Practice what you've learned