Viewing and joining text

Much of the work at a command line is reading text — looking at a file, checking the start or end of one, pulling a column out, stitching two files together. This topic covers the commands for that: viewing files, showing parts of them, and joining or splitting text by line and by column.

This page names every command in the topic. The companion topic, Transforming text, covers the commands that reshape text — sorting, de-duplicating, substituting characters.

The commands #

Viewing a whole file

Viewing part of a file

Cutting and combining

Splitting a file into files

A note on input #

Almost every command here follows the same convention for where its text comes from. Name one or more files and it reads those; name none and it reads from standard input, so it can sit in the middle of a pipeline. Where a command accepts files, the name - stands for standard input, so you can mix files and piped input in one invocation.

Where to start #

To simply print a file, or join a few together, read cat — the most-used command in the topic.

cat prints the contents of files. Its name is short for "concatenate" — given several files, it prints them one after another, as a single stream.

cat [options] [file...]

$ cat notes.txt
$ cat part1.txt part2.txt part3.txt > whole.txt

With no file — or with the name - — cat reads standard input, which is what makes it useful in a pipeline or for capturing typed input into a file.

Plain use #

Most of the time cat is run with no options at all: it copies its input to its output, byte for byte, unchanged. The options exist for the times you want to see something about the text that is normally invisible, or to adjust spacing and numbering.

Numbering lines #

Making invisible characters visible #

These options reveal characters that normally print as nothing, or as whitespace — useful when a file is not behaving and you suspect a stray tab or control character.

Adjusting spacing #

Exit status #

tac prints a file with its lines in reverse — the last line first, the first line last. The name is cat spelled backwards, which is exactly what it does.

tac [options] [file...]

$ tac events.log

Reversing a log file so the newest entries come first is the everyday use. With no file, tac reads standard input.

Reversing on something other than lines #

tac does not have to split on newlines. You can tell it to treat some other string as the separator, and it will reverse the order of the pieces between those separators.

$ tac -s ', ' -r names.csv

Exit status #

head prints the beginning of a file — by default, the first 10 lines.

head [options] [file...]

$ head config.toml

It is the quick way to glance at the top of a file without printing the whole thing — a header, the first few records, the start of a log. With no file, head reads standard input.

Choosing how much #

NUM accepts a unit suffix (K, M, …) when counting bytes.

Counting from the end #

If NUM is given a leading -, the meaning flips: head prints everything except the last NUM.

$ head -n -5 report.txt    # all but the final 5 lines

Headers for multiple files #

When given more than one file, head prints a header line before each one so you can tell them apart. Two options override that:

Other options #

Exit status #

tail prints the end of a file — by default, the last 10 lines.

tail [options] [file...]

$ tail server.log

It is the counterpart of head: the quick way to see the most recent lines of a file. With no file, tail reads standard input.

Choosing how much #

If NUM is given a leading +, the meaning flips: tail prints everything from line NUM onward.

$ tail -n +20 report.txt    # from line 20 to the end

Following a growing file #

tail's most useful trick is -f. Instead of printing the end and exiting, it stays running and prints new lines as they are appended — the standard way to watch a log file live.

Press Ctrl-C to stop a tail -f.

Headers for multiple files #

As with head, when given more than one file tail prints a header before each.

Other options #

Exit status #

more displays a file one screenful at a time. Where cat prints a whole file at once — and a long file scrolls straight past — more shows the first screen, then waits for you before showing the next.

more [options] file...

$ more long-report.txt

Moving through the file #

While more is paused, it is waiting for a keypress:

more moves forward through a file. It is the simple, always-available pager — enough for reading something through once.

Options #

Exit status #

nl prints a file with its lines numbered. cat -n also numbers lines; nl is the command for when you need control over the numbering — which lines count, how the number looks, where it restarts.

nl [options] [file...]

$ nl chapter.txt
     1  Once the system is installed,
     2  the first boot brings up
       
     3  the configuration service.

Which lines get a number #

By default nl numbers only the non-empty lines. -b sets the rule:

How the number looks #

Logical pages #

nl can treat one file as a sequence of logical pages, each with a header, a body, and a footer, and number the three parts by different rules. Pages are separated by special delimiter lines in the input.

The styles for -h and -f are the same a / t / n / pBRE set as -b. For an ordinary file with no delimiter lines, the whole file is one body and only -b matters.

Exit status #

cut extracts columns from text. For each line of input it prints only the parts you select, dropping the rest.

cut option... [file...]

$ cut -d , -f 1,3 people.csv     # the 1st and 3rd comma-separated field
$ cut -c 1-8 log.txt             # the first 8 characters of each line

Every run of cut needs two decisions: what counts as a column, and which columns to keep.

What counts as a column #

Choose exactly one mode:

-b and -c cut by position. -f cuts by field, which is what you want for tabular data — a CSV, a table, the columns of a report.

Which columns to keep #

The mode option takes a sequence: numbers and inclusive ranges, separated by commas.

Field mode: the delimiter #

In field mode, cut needs to know what separates the fields.

Other options #

Exit status #

paste joins files side by side. It takes the first line of each file and prints them on one line, then the second line of each, and so on — merging files into columns.

paste [options] [file...]

$ paste names.txt ages.txt
Ada      36
Grace    41
Linus    29

By default the merged pieces are separated by a tab. paste is the column-wise counterpart of cat, which joins files end to end.

Pasting one file at a time #

$ paste -s -d , names.txt
Ada,Grace,Linus

Choosing the separator #

paste and join #

paste merges by position — line 1 with line 1, line 2 with line 2, blind to content. When you want to merge by a matching value — pairing rows that share a key — that is join.

Exit status #

join merges two files on a shared field. For every pair of lines — one from each file — that have the same value in their join field, it prints a combined line. It is the command-line form of a database join.

join [options] file1 file2

$ join employees.txt salaries.txt

If employees.txt has 101 Ada and salaries.txt has 101 80000, join pairs them on the shared 101 and prints 101 Ada 80000.

The input must be sorted #

join works by stepping through both files together, so both files must be sorted on the join field. If they are not, join will miss matches. Sort them first with sort, on the same field join will use.

join checks the ordering as it goes and reports a file that is out of order; --nocheck-order turns that check off, and --check-order forces it on.

Choosing the join field #

By default join joins on the first field of each file, and fields are separated by whitespace.

Unmatched lines #

By default join prints only the lines that paired. A line with no match on the other side is dropped. These options bring unmatched lines back:

Shaping the output #

Exit status #

comm compares two sorted files and tells you which lines are unique to each and which are common to both.

comm [options] file1 file2

By default it prints three columns:

$ comm list-a.txt list-b.txt
            apple
banana
                        cherry
date

• Column 1 — lines only in file1.
• Column 2 — lines only in file2.
• Column 3 — lines in both.

Each column is indented past the one before, so a line's column position tells you where it was found.

The input must be sorted #

comm steps through both files together, so both must be sorted. On unsorted input the comparison is meaningless. Sort the files first with sort.

comm checks the ordering and reports a file that is out of order. --check-order forces the check; --nocheck-order disables it.

Showing only the columns you want #

Each column can be switched off. The remaining columns close up to fill the gap.

These combine to answer specific questions. comm -12 shows only the common lines — the intersection of the two files. comm -23 shows lines in file1 that are not in file2 — the difference.

Other options #

Exit status #

split breaks one file into several smaller files of roughly equal size.

split [options] [input [prefix]]

$ split -l 1000 big.log
$ ls
xaa  xab  xac  xad

By default split writes 1000 lines per piece, names the pieces xaa, xab, xac, … — a prefix of x followed by a two-letter suffix — and reads its input from a named file or from standard input.

To put the file back together, concatenate the pieces in order: cat xaa xab xac … > original.

Each piece is a newly created file, and a new file needs a security descriptor; each piece inherits one from the directory it is created in.

How big each piece is #

Choose one of these to set the piece size:

A SIZE accepts a unit suffix: K, M, G, … as powers of 1024, or KB, MB, … as powers of 1000.

Fixed number of chunks (-n) #

-n divides the input into a set number of pieces rather than fixing each piece's size. CHUNKS may be:

Naming the pieces #

Other options #

Exit status #

csplit — "context split" — breaks a file into pieces at points you choose: a line number, or a line that matches a pattern. Where split cuts a file into equal sizes, csplit cuts it at meaningful boundaries.

csplit [options] file pattern...

$ csplit report.txt '/^Chapter/' '{*}'

That splits report.txt into one piece per chapter, cutting at every line beginning with Chapter. The pieces are written to xx00, xx01, xx02, … and csplit prints the byte size of each.

Each piece is a newly created file, and a new file needs a security descriptor; each piece inherits one from the directory it is created in.

Patterns #

Each pattern argument marks a cut point. csplit copies everything up to that point into the next output file, then continues.

A pattern that finds no match is an error, and by default csplit removes the pieces it had already written. -k keeps them.

Naming the pieces #

Other options #

Exit status #

od — short for "octal dump" — prints the raw bytes of a file. Where cat shows you a file as text, od shows you the actual bytes underneath: each one rendered as a number, a character, or a floating-point value, in whatever base you ask for. It is the tool for looking at binary files, checking exactly which bytes a text file contains, or finding a stray control character that isn't visible on screen.

od [options] [file...]

$ od hello.txt
0000000 062510 066154 020157 067527 066162 020144 000012
0000015

With no file — or with the name - — od reads standard input, so it can sit at the end of a pipeline. Given several files, it reads them one after another as a single continuous stream, and the byte offsets run straight through from one file into the next.

How the output is laid out #

Every line of output has two parts:

• An offset on the left: the position, counted in bytes from the start of the input, of the first byte on that line. By default it is printed in octal (see -A to change the base).
• One or more format columns on the right: the bytes of that line rendered in the format(s) you chose.

With no format option, od prints the input as octal, two bytes at a time — the traditional default. The very last line of output is a lone offset marking the total length of the input.

If you ask for more than one format at once, each format is printed on its own line, stacked under a single shared offset. Only the first line of the group carries the offset; the rest are indented to line up beneath it.

Choosing what the bytes look like #

There are two ways to say how bytes should be rendered: a set of single-letter named shortcuts for the common cases, and the general -t / --format option for full control. You can give several at once, and they are applied in the order they appear on the command line.

Named format shortcuts #

Each of these selects one output format. Combine them freely (od -cx prints characters and hex together).

Type specifications (-t, --format) #

-t takes a type specification and gives you every combination the named shortcuts cover and more. Repeat -t (or list several specs in one argument) to print multiple formats.

A type specification is a type letter, an optional size, and an optional z suffix:

For the numeric types (d, u, o, x, f) a size says how many bytes make up one unit. It can be a number, or a letter:

If you leave the size off, integer types default to 4 bytes and floating point defaults to 8 bytes. For the character types a and c, a size is not allowed.

Add a z at the end of a spec to append an ASCII dump — the bytes of that line shown as text, . for anything non-printable — to the right of the numbers. For example, od -t x1z gives the familiar hex-plus-text layout:

$ od -A x -t x1z file.bin
000000 48 65 6c 6c 6f 2c 20 77 6f 72 6c 64 21 0a           >Hello, world!.<

Examples: od -t d2 is signed decimal in 2-byte units; od -t x1 is hex byte-by-byte; od -t fD is double-precision floats; od -t c is characters. od -t x1 -t c prints hex and characters as two stacked lines.

Byte order (--endian) #

For multi-byte numeric formats, --endian sets the byte order used to assemble each unit. Without it, od uses the machine's native byte order. Use --endian=big or --endian=little to force one regardless of the host.

The offset column #

Only the first character of the value is examined, so od -A n and od -Anone mean the same thing.

Which bytes to read #

By default od dumps the whole input. Two options narrow that to a window:

In both, BYTES is decimal by default, octal if it starts with 0, or hexadecimal if it starts with 0x. A suffix scales it: b = ×512, KB/K = ×1000 / ×1024, MB/M = ×1000² / ×1024², GB/G = ×1000³ / ×1024³.

Collapsing repeated lines #

When several output lines in a row would be identical, od prints the first one, then a single line containing just * to stand for the run, and resumes at the next line that differs. This keeps a dump of a large block of identical bytes (a file full of zeros, say) short.

Output width #

The width must be a whole multiple of the size of the largest format unit in play; if it isn't, od warns and rounds it to a usable value.

Finding printable strings #

This turns od into a quick way to pull the human-readable text out of a binary file. The offset in front of each string honours -A (and is dropped entirely under -A n). -j and -N still apply, so you can restrict the scan to a window.

The traditional offset form #

od also accepts the historical calling convention, where an offset (and, in --traditional mode, a label) is given as a bare operand rather than an option:

od [named-format-options] [file] [[+]offset[.][b]]
od --traditional [options] [file] [[+]offset[.][b] [[+]label[.][b]]]

The offset says where in the input to begin, exactly like -j. It is read as octal by default, hexadecimal if prefixed with 0x, or decimal if it carries a trailing .; a trailing b multiplies it by 512. The optional leading + is allowed everywhere and is required when there is no filename (so od +20 reads standard input starting at octal offset 20).

The label (only in --traditional mode) is a second such number: the dump still begins at offset, but the offset column is displayed as if it started at label — useful for making a partial dump's offsets match the original file.

od only reads an operand as an offset in the plain form when none of -A, -j, -N, -t, -v, or -w are present. So if a real filename happens to look like a number, add a harmless option such as -j0 to force it to be treated as a filename.

A note on reading files #

od has no access rules of its own — it simply opens a file and reads its bytes. That read is subject to the same check as every read on Peios: whether you may open the file for reading is decided by the file's security descriptor (see Security descriptors). od never reveals bytes you could not already read another way; it only presents the bytes you can read in a different form.

Exit status #