Transforming text

This topic covers the commands that reshape text: putting lines in order, dropping duplicates, swapping characters, reflowing paragraphs to a width, counting what is there. Its companion, Viewing and joining text, covers printing and combining files; this topic is about changing the text itself.

The commands #

Ordering lines

Filtering lines

Substituting characters

Whitespace

Reflowing and paginating

Indexing and counting

Two commands that need sorted input #

A theme worth knowing before you start: uniq only collapses adjacent duplicate lines, so duplicates scattered through a file are not caught unless the file is sorted first. sort and uniq are almost always used together — and sort -u does both jobs in one step.

Where to start #

sort is the workhorse of the topic and the one with the most depth — start there.

sort reads lines and writes them out in order.

sort [options] [file...]

$ sort names.txt

With no file, sort reads standard input. Given several files, it sorts all of their lines together as one stream. By default it compares lines as plain text, character by character.

How lines are compared #

The default text comparison is often not the order you want — 10 sorts before 9, and case matters. These options change the comparison:

And these adjust what is compared:

Sort order options #

Sorting by a field #

By default sort compares whole lines. -k sorts on a key — a chosen part of each line — which is what you need for tabular data.

A line is split into fields (by default at whitespace). A key is written FIELD[.CHAR][OPTIONS][,FIELD[.CHAR]] — a start field, an optional start character within it, and an optional end. The single-letter comparison options above can be appended to a key to apply only to it.

$ sort -k 2 -n scores.txt        # sort on the 2nd field, numerically
$ sort -t , -k 3 people.csv      # sort a CSV on the 3rd field

Checking and merging #

Output and resources #

Exit status #

uniq deals with repeated lines: it can collapse a run of identical lines into one, count them, or print only the ones that repeated.

uniq [options] [input [output]]

$ uniq events.txt

input and output may be given as operands; with neither, uniq reads standard input and writes standard output.

uniq only sees adjacent duplicates #

This is the one thing to know about uniq: it only compares each line with the one before it. It collapses adjacent repeats. Duplicate lines scattered through a file, with other lines between them, are not caught.

So uniq is almost always paired with sort, which brings every copy of a line together first:

$ sort events.txt | uniq

If collapsing duplicates is all you need, sort -u does both jobs in one command. Use uniq itself when you want what sort -u cannot do — counting repeats, or printing only the duplicated lines.

What to output #

By default uniq prints every line, with each run of adjacent duplicates reduced to a single copy. These options change that:

What counts as "the same" #

Other options #

Exit status #

tr — "translate" — works on text one character at a time. It can replace characters with other characters, remove characters, or collapse runs of a character into one.

tr [options] set1 [set2]

tr reads standard input and writes standard output — it has no file arguments, so it always sits in a pipeline.

$ tr 'a-z' 'A-Z' < notes.txt        # lower-case to upper-case
$ tr -d ' ' < spaced.txt            # delete every space
$ tr -s ' ' < padded.txt            # collapse runs of spaces to one

The three jobs #

What tr does depends on the options and how many sets you give it.

• Translate — give two sets. Each character in set1 is replaced by the character at the same position in set2. tr 'abc' 'xyz' turns every a into x, b into y, c into z.
• Delete (-d) — give one set. Every character in set1 is removed.
• Squeeze (-s) — each run of a repeated character that appears in the relevant set is collapsed to a single occurrence.

-d and -s can be combined: delete one set of characters, then squeeze another.

Writing a set #

A set is a string of characters, with a few shorthands:

Options #

Exit status #

expand converts tab characters into spaces, so that text lines up the same way no matter what tab width something is viewed with.

expand [options] [file...]

$ expand source.txt > source-spaces.txt

A tab is not a fixed number of spaces — it jumps to the next tab stop, and where those stops are is a display setting. expand removes the ambiguity: it replaces each tab with exactly the number of spaces needed to reach the same column, baking the layout in. With no file, it reads standard input.

unexpand is the reverse operation.

Options #

Exit status #

unexpand is the reverse of expand: it converts runs of spaces back into tab characters.

unexpand [options] [file...]

$ unexpand spaced.txt > tabbed.txt

Where a run of spaces reaches a tab stop, unexpand replaces it with a tab. By default it only converts the leading spaces on each line — the indentation — and leaves spacing within the text alone. With no file, it reads standard input.

Options #

Exit status #

fmt reflows paragraphs: it rejoins the lines of a paragraph and re-breaks them so each comes out close to a target width. Words move freely between lines to make the result fit.

fmt [options] [file...]

$ fmt -w 72 draft.txt

fmt works on paragraphs — runs of non-blank lines separated by blank lines — and treats word boundaries as places it may break. The result reads naturally. This is what distinguishes it from fold, which simply cuts every line at a fixed column without regard to words.

With no file, fmt reads standard input.

Setting the width #

Controlling the reflow #

Selecting which lines to format #

Exit status #

fold breaks long lines so that none exceeds a fixed width. Every line longer than the limit is cut into pieces.

fold [options] [file...]

$ fold -w 80 wide.txt

By default fold wraps at 80 columns, and it cuts at exactly that column — even in the middle of a word. That hard cut is the difference between fold and fmt: fmt reflows paragraphs and breaks between words; fold just enforces a width, mechanically. Use fold when you need a guarantee that no line is wider than N; use fmt when you want the result to read well.

With no file, fold reads standard input.

Options #

Exit status #

pr prepares text for printing on paper. It breaks a file into pages, puts a header and a trailer on each, and can arrange the text into columns.

pr [options] [file...]

$ pr report.txt

Run plainly, pr divides the input into 66-line pages, each beginning with a five-line header — the date, the file name, and a page number — and ending with a trailing margin. With no file, it reads standard input.

Columns #

pr can lay text out in several columns per page.

The header and the page #

Selecting and numbering #

Other options #

Exit status #

ptx produces a permuted index — a concordance. For each significant word in the input, it prints a line showing that word together with the text around it, so the same passage appears once per keyword it contains.

ptx [options] [input...]

$ ptx manual.txt

A permuted index is the kind of index found at the back of a reference book: every keyword, in alphabetical order, each shown in its context. ptx builds one from plain text. With no file, it reads standard input.

Choosing the keywords #

Output format #

Exit status #

tsort performs a topological sort. Given a set of "this must come before that" rules, it produces an order in which everything appears after the things it depends on.

tsort [file]

$ tsort dependencies.txt

With no file, tsort reads standard input.

The input #

tsort reads its input as a flat sequence of whitespace-separated tokens, taken in pairs. Each pair A B means "A must come before B" — an edge in a dependency graph.

compiler  linker
linker    installer
compiler  installer

That input says the linker depends on the compiler, the installer on the linker, and the installer on the compiler. tsort reads the pairs and prints the items in a valid order:

compiler
linker
installer

A token paired with itself (A A) simply introduces A with no dependency — a way to list an item that nothing else mentions.

Cycles #

A topological order only exists if the dependencies form no cycle. If A must come before B and B must come before A, there is no valid order. tsort detects this, reports the loop it found, and exits with a failure status — it still prints an order, but the cycle means that order cannot satisfy every rule.

Where it is used #

tsort answers "what order do I do these in?" — building components in dependency order, scheduling tasks, sequencing anything where some steps must precede others. It is the dependency-aware counterpart to sort, which orders by value rather than by dependency.

Exit status #

wc — "word count" — counts what is in a file: its lines, its words, and its bytes.

wc [options] [file...]

$ wc report.txt
  214  1832 12044 report.txt

By default wc prints three numbers per file — lines, words, bytes — followed by the file name. Given several files, it adds a total line. With no file, it reads standard input.

Choosing what to count #

Pass any of these and wc prints only the counts you ask for, in the fixed order lines, words, characters/bytes, longest-line:

$ wc -l report.txt        # just the line count
214 report.txt

Other options #

Exit status #

shuf shuffles: it outputs its input lines in a random order. Every possible ordering is equally likely.

shuf [options] [file]
shuf -e [options] arg...
shuf -i lo-hi [options]

$ shuf playlist.txt

shuf is the opposite of sort — instead of imposing an order, it removes one. With no file, it reads standard input.

Where the lines come from #

shuf has three ways of getting its input, one per usage form:

$ shuf -e red green blue          # shuffle three given words
$ shuf -i 1-100                   # the numbers 1..100 in random order

Shaping the output #

shuf -n 1 is a common idiom — pick one random line.

Reproducible shuffles #

A shuffle is random by default. To get the same shuffle every time — for a repeatable test, say — fix the randomness:

Exit status #