File Formats:SAVF

A SAVF (save file) is the container OS/400 and IBM i write with SAVOBJ and SAVLIB, and read back with RSTOBJ and RSTLIB. It holds Machine Interface objects — programs, files, data areas, libraries — in the form they have on disk, together with a catalogue describing what was saved and from which machine.

Understanding the format is useful well beyond restoring a save file on a machine. It is the only practical way to get at the bytes of an MI object offline: the same byte stream also appears inside optical images and tape images, so a distribution medium, a PTF, or a backup is an extractable archive of real MI objects. What can be done with one once it is extracted is described in File Formats:PGM for programs.

Everything below is big endian, and every text field is EBCDIC.

Three layers

It helps to separate three things that are often run together:

• the chunk layer, which wraps the save stream in 528-byte units with a checksum on each;
• the item layer, which is a chain of saved objects, each a 512-byte descriptor followed by that object's segments;
• the catalogue, a single special item per save that lists the library and its objects.

Only the chunk layer is specific to a save file held in QSYS. When the same save is carried on optical or tape media, the chunk wrapper is absent and the item chain sits in the medium directly. A reader that separates the layers handles all three media with one item parser.

The chunk layer

A save file is exactly chunk count × 528 bytes. Chunk n, counting from zero, begins at file offset n × 528 and consists of 512 bytes of save-stream data followed by a 16-byte checksum. The logical save stream is the concatenation of the data halves, so translating a stream offset to a file offset is

file_offset = (stream_offset / 512) * 528 + (stream_offset % 512)

File header

The first 512 bytes of the stream — which are also the data half of chunk 1 — describe the container.

Offset	Size	Field	Notes
+0x00	8	MI creation timestamp	Also the seed for every chunk checksum.
+0x08	4	Chunk count	32-bit form.
+0x10	8	Machine type and model	EBCDIC, model first: 42A 8286 is an 8286-42A.
+0x30	4	Release level
+0x34	4	Header length	The first item starts here. Zero means 512.
+0x38	8	Chunk count	64-bit form, used when the 32-bit field would overflow.
+0x40	16	Global checksum backup	See below.

A valid file satisfies chunk count == file size / 528; anything else means truncation. Because the header is itself chunk 1, verifying its checksum is the cheapest reliable test of whether a file is a save file at all.

The global checksum displaces real data

This is the trap worth knowing about before writing any extractor. The last 16 bytes of the logical stream, at chunk count × 512 − 16, are overwritten with a whole-file checksum. The data bytes they replaced are preserved in the file header at +0x40, and a reader must put them back. A reader that does not will silently corrupt the tail of the last object it extracts, in a way that no chunk checksum will catch, because the chunk checksums are computed over the overwritten bytes.

The chunk checksum

Each chunk's 16-byte trailer is the chunk number, counting from one, as a 4-byte big-endian integer, followed by a 12-byte digest. The digest is two folding passes built from a single primitive:

chkadd(a, b):          # b is exactly one byte longer than a
    b[1:] += a << 1    # big-endian magnitudes, carry propagated
    b[0]  += carry + (the bit shifted out of the top of a)

The first pass takes the chunk number followed by the 512 data bytes and cuts them into eight parts of 61, 62, 63, 64, 65, 66, 67 and 68 bytes, then folds them in ascending order: chkadd of the 61-byte part into the 62-byte part, that into the 63-byte part, and so on. The second pass takes the resulting 68 bytes, cuts them into parts of 5, 6, 7, 8, 9, 10, 11 and 12 bytes, and folds them the same way, leaving 12 bytes. The first 8 of those are then XORed with the file header's MI timestamp.

There is one non-obvious detail. In both passes the parts are filled from the input in the order 61, 62, 63, 64, 65, 66, 68, 67 — and 5, 6, 7, 8, 9, 10, 12, 11 — while being folded in ascending order. An implementation that fills them in plain ascending order produces a plausible-looking wrong answer.

The construction is a checksum, not a cryptographic hash. It detects media damage. It is not a tamper seal, and it carries no secret beyond a timestamp printed in the same header.

The item layer

Items are chained: parse the descriptor at the header length, add the item's computed length, repeat, and stop when the next item would begin past (chunk count − 1) × 512. Every descriptor begins with FFFFFFFF, which makes resynchronising after damage feasible.

Item descriptor

Offset	Size	Field	Notes
+0x00	4	FFFFFFFF	Eyecatcher.
+0x04	10	Object name	EBCDIC. Replaced by a file ID for integrated file system objects.
+0x0E	20	Member name	Empty unless the object has members.
+0x22	2	MI object type	The same code space used throughout the machine: 0x0201 *PGM, 0x0401 *LIB, 0x1901 *FILE, 0x19DB *SRDS.
+0x24	1	Flags	Bit 0x40 or 0x80 means the item is an IFS object.
+0x25	3	Catalogue offset	Where this object's entry sits inside the catalogue.
+0x44	4	Header occupation	Bytes of the descriptor page actually used.
+0x48	4	Stored length	In 512-byte units, minus one.
+0x54	2	Release created for	Governs the section-entry layout below.
+0x56	4	Metadata length	For a catalogue item, the size of the catalogue.
+0x64	4	Section count
+0xC0	8	Save timestamp
+0xCC	4	Total length	In 512-byte units.
+0xD4	4	Data length	In 512-byte units. Always the total length less 4096 bytes.
+0x100	14 × 8	Pointer array	See below.
+0x170	8	Packed length	Stored as the length less 512.

The pointer array holds single-level store addresses, but only the low 24 bits matter: they are an offset into the item, biased by 4096, so (pointer & 0xFFFFFF) − 4096 gives an offset within the descriptor page. Index 1 locates the section length table and index 8 the section type table.

Sections

A saved MI object is a set of segments, and each segment is one section of the item. Sections start at the first 4096-byte boundary after the descriptor and run consecutively, which is why the data length is always the total length less one page.

The length table has one entry per section. On any modern release — when the release created for field exceeds 32 — an entry is 16 bytes: capacity, length, and the segment's single-level store address. Older saves use a 10-byte entry with a shifted 6-byte address and no separate length.

The type table has one 8-byte entry per section, of which the first halfword is the segment type. This is the same value as the first halfword of the segment's own segment header, which is a useful consistency check when extracting: the item's idea of what a section is must agree with the segment's own.

Item length and compression

An uncompressed item occupies its stored length. A packed item records a non-zero packed length, and the writer then realigns the following item on the enclosing 512-byte boundary, so the step from one item to the next is

if packed == 512:        step = stored
elif packed > stored:    step = packed
else:
    slack  = stored - packed
    within = item_offset % 512
    if within == 0:      step = stored
    elif within < slack: step = packed + slack - within
    else:                step = stored + 512 - within

The compression method is identifiable from the first six bytes of the packed data, which are EBCDIC eyecatchers:

Bytes	EBCDIC	DTACPR
D3C46DE3D9E2	LD_TRS	*MEDIUM, the TERSE algorithm
D3C46DD3E9F1	LD_LZ1	*HIGH, IBM's LZ1
D3C46DE9D3C2	LD_ZLB	*ZLIB; the deflate stream begins 12 bytes in
none		*LOW, an SNA-style run-length scheme, recognisable only by decoding it

Note that these are not the compression used for a compressed LID payload; see Data Structures:LID, where all four of these were tried and rejected.

The catalogue

Every save writes one save/restore descriptor space item, MI type 0x19DB, named QSRDSSPC.n, ahead of the objects it describes. Items named QSR.ADDITIONAL_INFO.* or QSRDSSPC.EA SRD* are continuation blocks belonging to the preceding descriptor, not new saves.

The catalogue itself is the last metadata length bytes of that item's data. Its first byte is the save command, and that selects the shape: a library save has the layout below, while a save of the integrated file system produces a different, tree-structured descriptor.

Offset	Size	Field
+0x00	1	Save command
+0x01	1	Target release
+0x02	30	Saved library name
+0x20	2	Base object type, 0x0401
+0x22	8	Save timestamp
+0x2A	4	Object count
+0x34	8	System serial number
+0x3C	2	ASP number
+0x3E	10	Create authority
+0x4C	1	Save-while-active indicator
+0x50	10	ASP name, target release 66 and later only

Object descriptors follow, at +0x50 or at +0x70 when the ASP name is present. Each is 151 bytes on target release 49 and later, and 87 bytes before that. The first fields are the object name at +0x00 for 30 bytes, the MI object type at +0x1E, and the owning user profile at +0x20 for 10 bytes. The size is a 4-byte field at +0x43, multiplied by a second 4-byte field at +0x79 when that is non-zero. The remainder of the descriptor holds offsets, into the same buffer, of chains of length-prefixed records carrying the object's text description, service description, usage information, work-unit references and user information.

Worth noticing for its own sake: the catalogue records the serial number of the machine that produced the save. A save file is not anonymous.

Practical consequences

• Any MI object can be lifted out of a save file byte for byte, with its segments separated and their segment identifiers intact. That is what makes offline analysis of a program object possible without console access to a machine and without printing a storage dump.
• Verifying the chunk checksums is a genuine integrity check on an archived save file, and can be done without a machine.
• The item chain and the catalogue are independent descriptions of the same set of objects. Where they disagree, the file has been damaged or truncated.
• Because the chunk layer is only present for a save file held in QSYS, a save carved out of an optical or tape image is parsed by the same code with the chunk translation switched off.

Scope of this description

The layout was mapped with reference to jSAVF, an independently developed third-party reader, and then verified field by field against a real V7R4 save file. Where an object in that file also had a printed Display/Alter/Dump available, the bytes extracted here matched the printout exactly. The chunk checksum was reimplemented from scratch and verifies every chunk of the test file.

Not established: the meaning of the 4-byte field at +0x0C of the file header, the layout of the integrated file system catalogue, and the LZ1 and TERSE decompressors.

Weblinks

• jSAVF — a maintained third-party reader for save files, and the starting point for the layout described here.