encoding-formats（编码格式）

Encoding & Formats

Encode, decode, and inspect data in common formats. Covers Base64, URL encoding, hex, Unicode, JWTs, hashing, checksums, and serialization formats.

When to Use

• Decoding a Base64 string from an API response or config
• URL-encoding parameters for HTTP requests
• Inspecting hex dumps of binary data
• Decoding JWT tokens to see claims
• Computing or verifying file checksums
• Converting between character encodings (UTF-8, Latin-1, etc.)
• Understanding wire formats (protobuf, MessagePack)

Base64

Encode and decode

# Encode string
echo -n "Hello, World!" | base64
# SGVsbG8sIFdvcmxkIQ==

# Decode string
echo "SGVsbG8sIFdvcmxkIQ==" | base64 -d
# Hello, World!

# Encode a file
base64 image.png > image.b64
cat file.bin | base64

# Decode a file
base64 -d image.b64 > image.png

# Base64url (URL-safe variant: + → -, / → _, no padding)
echo -n "Hello" | base64 | tr '+/' '-_' | tr -d '='
# Base64url decode
echo "SGVsbG8" | tr '-_' '+/' | base64 -d

In code

// JavaScript (browser + Node.js 16+)
btoa('Hello');                    // "SGVsbG8="
atob('SGVsbG8=');                 // "Hello"

// Node.js Buffer
Buffer.from('Hello').toString('base64');           // "SGVsbG8="
Buffer.from('SGVsbG8=', 'base64').toString();      // "Hello"

// Binary data
Buffer.from(binaryData).toString('base64');
Buffer.from(b64String, 'base64');

# Python
import base64

base64.b64encode(b"Hello").decode()     # "SGVsbG8="
base64.b64decode("SGVsbG8=")            # b"Hello"

# URL-safe Base64
base64.urlsafe_b64encode(b"Hello").decode()
base64.urlsafe_b64decode("SGVsbG8=")

URL Encoding

Encode and decode

# Python one-liner
python3 -c "from urllib.parse import quote; print(quote('hello world & foo=bar'))"
# hello%20world%20%26%20foo%3Dbar

# Decode
python3 -c "from urllib.parse import unquote; print(unquote('hello%20world%20%26%20foo%3Dbar'))"
# hello world & foo=bar

# curl does it automatically for --data-urlencode
curl -G --data-urlencode "q=hello world & more" https://api.example.com/search

In code

// JavaScript
encodeURIComponent('hello world & foo=bar');
// "hello%20world%20%26%20foo%3Dbar"

decodeURIComponent('hello%20world%20%26%20foo%3Dbar');
// "hello world & foo=bar"

// encodeURI vs encodeURIComponent:
encodeURI('https://example.com/path?q=hello world');
// "https://example.com/path?q=hello%20world" (preserves URL structure)
encodeURIComponent('https://example.com/path?q=hello world');
// "https%3A%2F%2Fexample.com%2Fpath%3Fq%3Dhello%20world" (encodes everything)

from urllib.parse import quote, unquote, urlencode

quote('hello world')            # 'hello%20world'
unquote('hello%20world')        # 'hello world'
urlencode({'q': 'hello world', 'page': 1})  # 'q=hello+world&page=1'

Hex

View and convert

# File hex dump
xxd file.bin | head -20
xxd -l 64 file.bin          # First 64 bytes only

# Hex dump (compact, no ASCII)
xxd -p file.bin

# Convert hex to binary
echo "48656c6c6f" | xxd -r -p
# Hello

# od (alternative)
od -A x -t x1z file.bin | head -20

# hexdump
hexdump -C file.bin | head -20

# Python
python3 -c "print(bytes.fromhex('48656c6c6f').decode())"  # Hello
python3 -c "print('Hello'.encode().hex())"                 # 48656c6c6f

In code

// JavaScript
Buffer.from('Hello').toString('hex');           // "48656c6c6f"
Buffer.from('48656c6c6f', 'hex').toString();    // "Hello"

// Number to hex
(255).toString(16);     // "ff"
parseInt('ff', 16);     // 255

# Python
"Hello".encode().hex()                  # '48656c6c6f'
bytes.fromhex('48656c6c6f').decode()    # 'Hello'
hex(255)                                # '0xff'
int('ff', 16)                           # 255

Unicode

Inspect characters

# Show Unicode code points
echo -n "Hello 世界" | python3 -c "
import sys
for char in sys.stdin.read():
    print(f'U+{ord(char):04X}  {char}  {char.encode(\"utf-8\").hex()}')"
# U+0048  H  48
# U+0065  e  65
# ...
# U+4E16  世  e4b896
# U+754C  界  e7958c

# Convert Unicode escape to character
printf '\H\e\l\l\o'   # Hello
echo -e '\ä\¸\–\ç\•\Œ'       # 世界

# File encoding detection
file -bi document.txt
# text/plain; charset=utf-8

Encoding conversion

# Convert between encodings
iconv -f ISO-8859-1 -t UTF-8 input.txt > output.txt
iconv -f UTF-16 -t UTF-8 input.txt > output.txt

# List available encodings
iconv -l

# Python
python3 -c "
with open('latin1.txt', 'r', encoding='iso-8859-1') as f:
    content = f.read()
with open('utf8.txt', 'w', encoding='utf-8') as f:
    f.write(content)
"

Common Unicode issues

BOM (Byte Order Mark):
  UTF-8 BOM: EF BB BF at start of file
  Remove: sed -i '1s/^\ï\»\¿//' file.txt

Normalization (NFC vs NFD):
  "é" can be U+00E9 (one char) or U+0065 U+0301 (e + combining accent)
  Python: import unicodedata; unicodedata.normalize('NFC', text)

Mojibake (wrong encoding):
  "café" appears as "café" → file is UTF-8 but read as Latin-1
  Fix: re-read with correct encoding

JWT (JSON Web Tokens)

Decode a JWT

# JWT has 3 parts separated by dots: header.payload.signature
# Each part is Base64url-encoded

# Decode header and payload
TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"

# Decode header
echo "$TOKEN" | cut -d. -f1 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
# {"alg":"HS256","typ":"JWT"}

# Decode payload
echo "$TOKEN" | cut -d. -f2 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
# {"sub":"1234567890","name":"John Doe","iat":1516239022}

# One-liner function
jwt_decode() {
    echo "$1" | cut -d. -f2 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
}
jwt_decode "$TOKEN"

In code

// JavaScript (no library needed for decoding)
function decodeJWT(token) {
    const [header, payload] = token.split('.').slice(0, 2)
        .map(part => JSON.parse(atob(part.replace(/-/g, '+').replace(/_/g, '/'))));
    return { header, payload };
}

// Check expiry
function isJWTExpired(token) {
    const { payload } = decodeJWT(token);
    return payload.exp && payload.exp < Math.floor(Date.now() / 1000);
}

# Python
import json, base64

def decode_jwt(token):
    parts = token.split('.')
    # Add padding
    def pad(s): return s + '=' * (4 - len(s) % 4)
    header = json.loads(base64.urlsafe_b64decode(pad(parts[0])))
    payload = json.loads(base64.urlsafe_b64decode(pad(parts[1])))
    return header, payload

header, payload = decode_jwt(token)

Hashing

Common hash functions

# MD5 (not for security — only for checksums/dedup)
echo -n "Hello" | md5sum        # Linux
echo -n "Hello" | md5           # macOS

# SHA-256 (standard for integrity)
echo -n "Hello" | sha256sum
echo -n "Hello" | shasum -a 256

# SHA-1 (deprecated for security, still used in git)
echo -n "Hello" | sha1sum

# SHA-512
echo -n "Hello" | sha512sum

# Hash a file
sha256sum file.bin
md5sum file.bin

# openssl (works everywhere)
echo -n "Hello" | openssl dgst -sha256
openssl dgst -sha256 file.bin

In code

// Node.js
const crypto = require('crypto');
crypto.createHash('sha256').update('Hello').digest('hex');
// "185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969"

// File hash
const fs = require('fs');
const hash = crypto.createHash('sha256');
hash.update(fs.readFileSync('file.bin'));
console.log(hash.digest('hex'));

import hashlib
hashlib.sha256(b"Hello").hexdigest()
# "185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969"

# File hash
with open("file.bin", "rb") as f:
    print(hashlib.sha256(f.read()).hexdigest())

Checksums for file integrity

# Generate checksum file
sha256sum *.tar.gz > checksums.sha256

# Verify checksums
sha256sum -c checksums.sha256

# Compare two files without reading content
sha256sum file1.bin file2.bin
# or
cmp file1.bin file2.bin && echo "Identical" || echo "Different"

Serialization Formats

JSON ↔ other formats

# JSON to YAML
python3 -c "import json, yaml, sys; yaml.dump(json.load(sys.stdin), sys.stdout)" < data.json

# YAML to JSON
python3 -c "import json, yaml, sys; json.dump(yaml.safe_load(sys.stdin), sys.stdout, indent=2)" < data.yaml

# JSON to CSV
jq -r '.[] | [.id, .name, .email] | @csv' data.json > data.csv

# CSV to JSON
python3 -c "
import csv, json, sys
reader = csv.DictReader(open(sys.argv[1]))
print(json.dumps(list(reader), indent=2))
" data.csv

# JSON to TOML
python3 -c "import json, tomli_w, sys; tomli_w.dump(json.load(sys.stdin), sys.stdout.buffer)" < data.json

# Pretty-print JSON
jq '.' data.json
python3 -m json.tool data.json

Binary formats (inspection)

# MessagePack → JSON
python3 -c "
import msgpack, json, sys
data = msgpack.unpackb(sys.stdin.buffer.read(), raw=False)
print(json.dumps(data, indent=2))
" < data.msgpack

# Protobuf (decode without schema — shows field numbers)
protoc --decode_raw < data.pb

# CBOR → JSON
python3 -c "
import cbor2, json, sys
data = cbor2.loads(sys.stdin.buffer.read())
print(json.dumps(data, indent=2, default=str))
" < data.cbor

Quick Decode Script

#!/bin/bash
# decode.sh — Auto-detect and decode common encoded strings
INPUT="${1:-$(cat)}"

# Try Base64
B64_DECODED=$(echo "$INPUT" | base64 -d 2>/dev/null)
if [[ $? -eq 0 && -n "$B64_DECODED" ]]; then
    echo "Base64 → $B64_DECODED"
fi

# Try URL encoding
if echo "$INPUT" | grep -q '%[0-9A-Fa-f]\{2\}'; then
    URL_DECODED=$(python3 -c "from urllib.parse import unquote; print(unquote('$INPUT'))" 2>/dev/null)
    echo "URL   → $URL_DECODED"
fi

# Try JWT
if echo "$INPUT" | grep -qP '^eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.'; then
    echo "JWT header:"
    echo "$INPUT" | cut -d. -f1 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
    echo "JWT payload:"
    echo "$INPUT" | cut -d. -f2 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
fi

# Try hex
if echo "$INPUT" | grep -qP '^[0-9a-fA-F]+$' && [[ $((${#INPUT} % 2)) -eq 0 ]]; then
    HEX_DECODED=$(echo "$INPUT" | xxd -r -p 2>/dev/null)
    if [[ -n "$HEX_DECODED" ]]; then
        echo "Hex   → $HEX_DECODED"
    fi
fi

Tips

• Base64 increases data size by ~33%. Use it for embedding binary data in text formats (JSON, XML, email), not for compression or encryption.
• Base64url (RFC 4648) uses - and _ instead of + and /, and omits padding =. JWTs and URL parameters use this variant.
• SHA-256 is the standard for integrity checks. MD5 is fine for dedup and non-security checksums but broken for cryptographic use.
• JWTs are signed, not encrypted. Anyone can decode the header and payload. Only the signature verifies authenticity. Never put secrets in JWT claims.
• When files display garbled text (mojibake), the problem is almost always wrong encoding assumption. Check with file -bi and re-read with the correct encoding.
• xxd -p (plain hex) and xxd -r -p (reverse) are the fastest way to convert between binary and hex on the command line.
• URL-encode with encodeURIComponent (JavaScript) or urllib.parse.quote (Python), not by hand. Manual encoding misses edge cases.