Hashing and encoding

This topic covers two related jobs. Hashing reduces a file to a short fixed-size value — a checksum — that can be used to tell whether the file has changed. Encoding rewrites binary data as plain text so it can travel safely through channels that only handle text.

The commands #

Checksums and hashes

Encoding

What these commands are not #

It is worth being clear up front, because both jobs are easy to mistake for something they are not.

A hash is not encryption. Hashing is one-way: a checksum tells you whether data has changed, but the original cannot be recovered from it. Comparing it with a trusted expected hash can detect corruption or tampering; a hash supplied alongside an untrusted file does not establish who published that file. Hashing does not keep anything secret.

Encoding is not encryption either. base64 rewrites data into a text-safe form, but anyone can decode it straight back — it is a change of representation, not of secrecy. Encoding something does not protect it.

Neither hashing nor encoding hides data. They are about integrity and transport, not confidentiality.

Where to start #

To verify a file you downloaded against a published checksum, read the checksum commands. To turn binary data into something safe to paste into text, read base32 and base64.

This page covers a family of commands that all work the same way and differ only in which hash algorithm they use:

sha256sum [options] [file...]

Everything below is written with sha256sum, but applies to every command in the table.

Computing a checksum #

Run plainly, the command prints the hash of each file, followed by the file name:

$ sha256sum installer.iso
9f86d0818884...b1a5  installer.iso

With no file, it reads standard input. The hash is a fingerprint of the file's contents: change a single byte and the hash changes completely.

Verifying with a checksum #

The everyday use is verification — confirming a file is exactly what it should be, usually a download.

First, the file's publisher computes a checksum and publishes it, often in a file:

$ sha256sum installer.iso > installer.iso.sha256

Obtain the publisher's checksum file through a trusted source. Computing a new checksum from your own download only records its current contents; it does not verify them against the publisher's copy.

Use the complete digest-and-filename lines, not a bare hash or the abbreviated display example above. Run the check from the directory against which the manifest's file names were written. Giving an absolute path to the checksum file does not change that working directory.

For example, with both files in ~/Downloads:

$ cd ~/Downloads
$ sha256sum --check --strict installer.iso.sha256
installer.iso: OK

-c reads each hash filename line, recomputes the hash, and reports OK or FAILED. A checksum mismatch means the file differs from the one the checksum describes. File-access failures and malformed input can also prevent verification; read the diagnostic and check the exit status before using the download.

Keep manifests newline-delimited. --zero output is not a NUL-delimited input format for --check.

Without --strict, malformed lines can be skipped while the command still succeeds. A checksum input with no properly formatted lines fails in either mode.

Avoid --ignore-missing when verifying that a complete download set is present. In the current implementation it skips file-open failures, including errors other than a missing file. It still fails if no file in a checksum input verifies successfully.

Output format #

These dedicated commands produce plain, untagged hash file output by default. Unlike cksum, they do not accept --untagged.

b2sum additionally accepts -l, --length=BITS to produce a shorter BLAKE2b digest.

A note on choosing an algorithm #

These commands detect change — but not all of them resist a deliberate attempt to forge a match. MD5 and SHA-1 can be defeated by an attacker who wants two different files to share a checksum. They are still fine for catching accidental corruption, but for verifying that a file has not been tampered with, use a SHA-2 command (sha256sum and up) or b2sum.

Exit status #

The flag and verification behavior above is checked against peiosutils source at ec00f31 and its checksum validator. It was not tested on an installed Peios release. Treat any nonzero status as failure and use the diagnostics to distinguish a mismatch from an input or access error.

cksum is the general checksum command. Where each of the checksum commands is fixed to one algorithm, cksum does any of them — the algorithm is an option.

cksum [options] [file...]

$ cksum installer.iso
3915528286 372736000 installer.iso
$ cksum -a sha256 --untagged installer.iso
9f86d0818884...b1a5  installer.iso

With no -a, cksum computes a CRC checksum and prints the CRC value, the file's byte count, and the name. With -a, it uses the selected algorithm. The SHA-256 example adds --untagged to request the plain hash file output; without it, SHA-256 output is tagged by default.

Choosing the algorithm #

NAME may be:

crc, crc32b, sha3, and sm3 are available only through cksum — there is no dedicated command for them.

Verifying #

Choose the check command to match the publisher's manifest format. Run it from the directory against which the manifest's file names were written; the manifest's own location does not set that directory. The checksum workflow covers trusted expected values, complete manifest lines and exit-status checks.

For tagged lines such as SHA256 (installer.iso) = ..., the input identifies the algorithm:

cksum --check --strict CHECKSUMS

For untagged SHA-256 lines such as hash installer.iso, supply the algorithm:

cksum --algorithm=sha256 --check --strict SHA256SUMS

The dedicated sha256sum --check --strict SHA256SUMS command already selects SHA-256. With cksum, an explicitly selected algorithm must agree with any tagged input. Do not add --tag, --binary or --text to a check command: those output-mode options are rejected during verification.

You cannot combine --check with --algorithm=crc, crc32b, bsd or sysv. The default CRC output is not a checkable checksum manifest. Use normal newline-delimited checksum output for manifests; --zero output is not read as NUL-delimited check input.

Without --strict, malformed lines can be skipped while a check still succeeds. A checksum input with no properly formatted lines fails in either mode. Use --ignore-missing only for an intentional subset: the current implementation skips file-open failures beyond missing files, so success does not establish that the whole set was available. It still fails if no file in a checksum input verifies successfully.

Output format #

Exit status #

These verification rules are checked against current command handling and manifest validation, not tested on an installed Peios release. Treat any nonzero status as failure.

sum computes a small, old-style checksum of a file and reports it along with the file's size in blocks.

sum [options] [file...]

$ sum archive.tar
12345 84 archive.tar

The output is the checksum, the block count, and the file name. With no file, sum reads standard input.

Which algorithm #

sum predates the modern hashes and offers two historical algorithms:

When to use it #

sum exists for compatibility — for reading checksums produced by old tools, and for scripts that still expect its output. Its checksum is short and weak: it catches some accidental corruption, but it is easily fooled and gives no protection against tampering.

For anything new, do not use sum. Use a checksum command such as sha256sum, or cksum — both of which cksum can still produce, via cksum -a bsd and cksum -a sysv, if you need the legacy values.

Exit status #

base32 and base64 encode binary data as plain text — and decode it back. They are the same command with one difference: the alphabet they use.

base32 [options] [file]
base64 [options] [file]

$ echo Peios | base64
UGVpb3MK
$ echo UGVpb3MK | base64 -d
Peios

What encoding is for #

Some channels only carry text safely — they mangle or reject arbitrary bytes. Encoding rewrites binary data using a small, safe set of characters, so it can pass through unharmed; decoding reverses it exactly. base64 uses 64 characters and is the more compact; base32 uses 32. Preserve the encoded text exactly, including letter case: the current Base32 decoder expects uppercase letters. With no file, both read standard input.

Encoding is not encryption — anyone can decode the result. See the overview.

Options #

By default these commands encode. The options below apply identically to both.

When decoding, line-feed and carriage-return characters in the input are tolerated; -i is for other stray characters. Use it only when those characters are known transport noise. It discards characters outside the alphabet; it does not repair a damaged payload or normalize letter case.

Decode before using the result #

A decoder can write part of the result before reporting invalid input. A nonempty output file is therefore not proof that decoding succeeded. Decode to a separate file, check the exit status, and use the result only on success. Do not redirect over your input or an existing file you need: the shell opens and truncates the output before decoding starts.

For example, in a directory you control, choose a new output name and run:

if base64 --decode payload.b64 > payload.decoded; then
    printf '%s\n' 'Decoding succeeded; payload.decoded is ready to inspect.'
else
    printf '%s\n' 'Decoding failed; do not use payload.decoded.' >&2
    false
fi

In a script, put any processing of the decoded file inside the success branch. The failure branch returns a nonzero status and deliberately leaves any partial file for inspection; remove it before treating that name as a completed result. The same status check applies to base32 and basenc. Decoding success establishes a valid encoding, not the origin or safety of the decoded content.

This failure behavior is checked against current peiosutils source, not tested on an installed Peios release.

Exit status #

basenc is the general encoding command. Where base32 and base64 each do one encoding, basenc does many — the encoding is chosen by an option.

basenc encoding [options] [file]

$ echo Peios | basenc --base16
5065696F730A

Every run of basenc must name an encoding. With no file, it reads standard input.

The encodings #

Options #

By default basenc encodes. These options apply to whichever encoding you chose:

Check decoding before using the output #

Select the same encoding used by the sender. Decoding can write a partial result before failing, so check the exit status before using the output. Follow the separate-output example, replacing base64 --decode with, for example, basenc --base64url --decode for URL-safe Base64 input. Do not use --ignore-garbage to guess your way past a damaged payload: it only discards characters outside the selected alphabet.

basenc and the dedicated commands #

For plain base64 or base32, base64 and base32 are the shorter way to ask. Use basenc when you need an encoding the dedicated commands do not offer — hex, URL-safe base64, the bit-string forms, z85, or base58.

Exit status #