Write-Ahead Logging in Apache Cloudberry

Cloudberry adds its own WAL resource managers — for AO storage, the bitmap index and the distributed log. They are the reason a segment can be recovered at all.

Overview & Recovery Foundations

20.1.1 Two failure domains, one write-ahead log

Cloudberry eliminates single points of failure on two independent planes. The control plane is the coordinator plus a hot standby coordinator; the data plane is every primary segment paired with a mirror. Both planes ride the same PostgreSQL write-ahead log, replicated by streaming replication — but Cloudberry layers two MPP-specific pieces on top: a cluster-wide fault detector (FTS) that decides when to fail over, and a recovery path that understands distributed (two-phase) transactions, not just local ones.

20.1.2 The failure model: what can actually fail

Recovery is only definable against a model of what can fail. A single PostgreSQL node faces three classical failure classes; Cloudberry’s MPP shape adds two more — and each pillar of this chapter answers one of them:

Transaction failure — A statement errors or the client issues ROLLBACK. Nothing durable is lost — the transaction simply never commits. Handled by ordinary abort, not recovery.

Process / system crash — A backend, the postmaster, or the whole OS dies. In-memory state (shared buffers, unflushed WAL) is lost; on-disk files survive but may be inconsistent with the last acknowledged commits. This is what crash recovery (§20.4) repairs.

Media failure — A disk corrupts or dies — the bytes are gone, so local replay cannot help. The answer is a second copy on another machine: a mirror or standby, kept current by replication (§20.3).

Segment / node failure (MPP) — A whole segment host disappears. The cluster must detect it (FTS, §20.5) and promote that segment’s mirror, so the cluster stays available even though the failed node never recovered.

Coordinator failure (MPP) — The coordinator is itself a single point of failure; a hot standby coordinator, fed by WAL streaming, is promoted to replace it (§20.4.7).

20.1.3 Why a live database cannot trust its own data files

For speed, every change first lands in shared buffers in RAM; dirty pages are written back lazily by the background writer and checkpointer — often long after the transaction that made them has committed, sometimes before it commits at all. Two buffer-management policies decide what state a crash can leave on disk:

STEAL — Cloudberry may evict and flush a dirty page belonging to an uncommitted transaction to reclaim a buffer. So after a crash the data files can contain changes that were never committed.

NO-FORCE — A commit does not force its data pages to disk — only its WAL. So after a crash the data files can be missing changes that were already acknowledged to the client as committed.

STEAL + NO-FORCE is the fast combination, but it opens a gap in both directions: uncommitted junk may be on disk, and committed truth may be absent from it. Closing that gap at restart is precisely the job of recovery — and the write-ahead log is the only record that spans it.

20.1.4 WAL, checkpoints, and the redo-only model

The write-ahead log (§20.2) resolves NO-FORCE: because every change is logged before its data page may be written, replaying the log after a crash re-applies any committed change that never reached the data files. Redo is idempotent — each page stores the LSN of the last WAL record applied to it, so replay simply skips any record already reflected on the page. A checkpoint (§20.2.6) bounds how far back replay must begin: everything logically before the checkpoint’s REDO point is guaranteed already on disk, so older WAL can be discarded.

Textbook ARIES recovery then runs three phases — analysis, redo, and undo, the last rolling back the uncommitted changes that STEAL allowed onto disk. Cloudberry, like PostgreSQL, does redo only and never physically undoes anything. Uncommitted changes left on disk are harmless because MVCC never marks their transaction committed in pg_clog, so no snapshot can ever see them; VACUUM reclaims the dead row versions later. The one deliberate exception is a prepared (phase-1) two-phase transaction, which is kept across recovery so distributed recovery can resolve it (§20.4.4, §20.4.5).

20.1.5 Roadmap: the four pillars

The rest of the chapter implements this theory in four pillars, in dependency order — each builds on the one before, and the bridge at the end of every section hands off to the next: (1) WAL (§20.2) — the durable redo record everything is rebuilt from, including Cloudberry’s distributed-commit records; (2) streaming replication (§20.3) — keeping a second copy so a media or node failure loses nothing, and turning synchronous replication off when a mirror dies; (3) system recovery (§20.4) — single-node redo plus the distributed-transaction reconciliation layer; and (4) FTS (§20.5) — turning a detected failure into a role change. Every claim below was read and confirmed against the source; file:line references point at the working tree.

Write-Ahead Logging (WAL)

20.2.1 The WAL invariant and the insert path

Cloudberry inherits PostgreSQL’s WAL machinery essentially unchanged at its core. Every modification first becomes a WAL record, built through a fixed three-call sequence:

1. XLogBeginInsert()
2. XLogRegisterData() / XLogRegisterBuffer()
3. XLogInsert(rmid, info)

Anatomy of a WAL record

XLogRecord — length · xid · prev-LSN · CRC · rmid + info

block data — block refs + full-page images

main data — resource-manager payload

Two invariants govern that path:

No WAL during recovery — XLogBeginInsert refuses to start a record while replaying — it errors with “cannot make new WAL entries during recovery.”

WAL before data — XLogInsert enforces the write-ahead rule: a record reaches durable storage before the dirty data page it describes may be written back.

The info byte splits in two: its high 4 bits are the resource-manager opcode (the xact manager masks them with XLOG_XACT_OPMASK = 0xF0), and the low bits are record-specific flags.

Source — insert path src/backend/access/transam/xloginsert.c:154,355,467; opcode mask src/include/access/xact.h:192.

20.2.2 Resource managers and common record kinds

Every WAL record carries an xl_rmid byte naming the resource manager (rmgr) that knows how to describe and replay it. At redo time the startup process dispatches on that byte through a fixed table, RmgrTable[], indexed by RmgrId. Each entry is a RmgrData struct of function pointers; the two that matter most for this chapter are rm_redo (replay the record) and rm_desc/rm_identify (render it for pg_waldump).

src/include/access/xlog_internal.h:360

typedef struct RmgrData
{
    const char *rm_name;
    void        (*rm_redo) (XLogReaderState *record);
    void        (*rm_desc) (StringInfo buf, XLogReaderState *record);
    const char *(*rm_identify) (uint8 info);
    void        (*rm_startup) (void);
    void        (*rm_cleanup) (void);
    void        (*rm_mask) (char *pagedata, BlockNumber blkno);
    void        (*rm_decode) (struct LogicalDecodingContext *ctx,
                              struct XLogRecordBuffer *buf);
} RmgrData;

The table is defined by an X-macro: rmgr.c redefines PG_RMGR to emit one initializer per manager, then #includes rmgrlist.h. The list order is load-bearing — it fixes each manager’s numeric ID, which is written into every WAL record, so new managers may only be appended.

src/backend/access/transam/rmgr.c:43

#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,decode) \
    { name, redo, desc, identify, startup, cleanup, mask, decode },

RmgrData    RmgrTable[RM_MAX_ID + 1] = {
#include "access/rmgrlist.h"
};

The standard upstream managers cover the core engine: RM_XLOG_ID (checkpoints, switches, FPI), RM_XACT_ID (commit/abort/prepare — §20.2.5), RM_SMGR_ID, RM_CLOG_ID, RM_HEAP_ID/RM_HEAP2_ID, the index managers RM_BTREE_ID/RM_HASH_ID/RM_GIN_ID/RM_GIST_ID/RM_SPGIST_ID/RM_BRIN_ID, plus RM_SEQ_ID, RM_STANDBY_ID, RM_RELMAP_ID, and so on. Cloudberry appends three of its own after the upstream list:

src/include/access/rmgrlist.h:50

/* Cloudberry-specific resource managers */
PG_RMGR(RM_BITMAP_ID, "Bitmap", bitmap_redo, bitmap_desc, bitmap_identify, NULL, NULL, NULL, NULL)
PG_RMGR(RM_DISTRIBUTEDLOG_ID, "DistributedLog", DistributedLog_redo, DistributedLog_desc, DistributedLog_identify, NULL, NULL, NULL, NULL)
PG_RMGR(RM_APPEND_ONLY_ID, "Appendonly", appendonly_redo, appendonly_desc, appendonly_identify, NULL, NULL, NULL, appendonly_decode)

• RM_BITMAP_ID — WAL for the on-disk bitmap index access method.
• RM_DISTRIBUTEDLOG_ID — the distributed-transaction commit log (local XID → distributed XID), redo via DistributedLog_redo.
• RM_APPEND_ONLY_ID — WAL for Append-Optimized / AOCO storage, redo via appendonly_redo; even supplies a logical-decoding callback.

At redo, GetRmgr(rmid) bounds-checks the ID and RmgrNotFound (an ERROR) fires for an unregistered manager. rmgr.c also exposes RegisterCustomRmgr for extension-supplied managers in a reserved ID range.

The header of rmgrlist.h warns that entry order defines the persisted IDs and that changes “possibly need an XLOG_PAGE_MAGIC bump” — reordering or removing a manager breaks on-disk WAL compatibility.

20.2.3 Writing WAL: buffers, flush, and the WAL writer

WAL lives first in a ring of shared WAL buffers (wal_buffers pages of XLOG_BLCKSZ). Getting a record to durable disk is a three-stage pipeline tracked by two pairs of LSNs in shared memory — a request (how far we want to be) and a result (how far we are):

src/backend/access/transam/xlog.c:406

typedef struct XLogwrtRqst { XLogRecPtr Write; XLogRecPtr Flush; } XLogwrtRqst;
typedef struct XLogwrtResult { XLogRecPtr Write; XLogRecPtr Flush; } XLogwrtResult;

So there are three frontiers: the insert position (head of reserved space, Insert->CurrBytePos), the write position (copied out of the buffers via write()), and the flush position (made durable via fsync). The WAL rule: a data page may not be written back until WAL up to its LSN has reached the flush frontier.

XLogInsertRecord is the low-level insert: under an insertion lock it reserves space (ReserveXLogInsertLocation), fills xl_prev, computes the CRC, copies bytes into the buffer pages — and deliberately does not write or flush. XLogWrite (xlog.c:2160) drains buffer pages to the kernel, opening segment files as it crosses boundaries; when it finishes a segment it fsyncs it, wakes walsenders, and notifies the archiver:

src/backend/access/transam/xlog.c:2347

if (finishing_seg)
{
    issue_xlog_fsync(openLogFile, openLogSegNo, tli);
    WalSndWakeupRequest();
    LogwrtResult.Flush = LogwrtResult.Write;     /* end of page */
    if (XLogArchivingActive())
        XLogArchiveNotifySeg(openLogSegNo, tli);

XLogFlush (xlog.c:2607) is the durability entry point callers use to make a specific LSN durable — e.g. before reporting a synchronous commit. It quick-exits if record <= LogwrtResult.Flush (someone else’s fsync already covered it — the always-on form of group commit), otherwise takes WALWriteLock and pushes the write request forward so one fsync covers as much as possible. The classic opt-in group-commit delay stacks on top: with commit_delay > 0 and enough active backends it sleeps to let more followers join the same fsync:

src/backend/access/transam/xlog.c:2711

if (CommitDelay > 0 && enableFsync &&
    MinimumActiveBackends(CommitSiblings))
{
    pg_usleep(CommitDelay);
    insertpos = WaitXLogInsertionsToFinish(insertpos);
}

The WAL writer process (walwriter.c) keeps ordinary backends off the fsync path and bounds asynchronous-commit latency: its loop calls XLogBackgroundFlush() (xlog.c:2797) once per wal_writer_delay, flushing at least that often or once wal_writer_flush_after blocks accumulate. Backends may still write/fsync themselves when the WAL writer can’t keep up.

Insert vs Write vs Flush. A returned end-LSN from XLogInsertRecord only proves the record is in the WAL buffers. Crash safety requires it to reach the Flush frontier — exactly what a synchronous commit waits on via XLogFlush, and what the WAL writer guarantees within a bounded delay for async commits.

20.2.4 WAL file operations: segments, recycling, archiving

A WAL segment file name is 24 hex digits

TimelineID — 8 hex digits

log id (segno high 32 bits) — 8 hex digits

segment (segno low 32 bits) — 8 hex digits

pg_wal holds the physical log as fixed-size segment files. Segment size is wal_segment_size, set at initdb, a power of two between 1 MB and 1 GB. Cloudberry changes the default from PostgreSQL’s 16 MB to 64 MB (DEFAULT_XLOG_SEG_SIZE) — which also affects checkpoint frequency. A 64-bit XLogRecPtr maps to a 64-bit XLogSegNo by division; the 24-hex-digit file name is the timeline followed by the segment number. pg_wal/archive_status holds the archiver’s .ready/.done markers.

New segments are created by XLogFileInit (xlog.c:3185) → XLogFileInitInternal (xlog.c:3015): build under a temp name xlogtemp.<pid>, zero-fill (when wal_init_zero), fsync, then atomically install. InstallXLogFileSegment (xlog.c:3375) does the atomic placement under ControlFileLock, scanning forward for a free future slot (find_free) via a crash-safe durable_rename:

src/backend/access/transam/xlog.c:3398

/* Find a free slot to put it in */
while (stat(path, &stat_buf) == 0)
{
    if ((*segno) >= max_segno) { LWLockRelease(ControlFileLock); return false; }
    (*segno)++;
    XLogFilePath(path, tli, *segno, wal_segment_size);
}

At checkpoint, RemoveOldXlogFiles (xlog.c:3640) reclaims segments older than the still-needed point — but only those already archived (XLogArchiveCheckDone). RemoveXlogFile (xlog.c:3783) embodies the recycle-vs-remove decision: if wal_recycle is on and future segments are still wanted, rename the file forward into a future slot (no zero-fill cost); otherwise durable_unlink it:

src/backend/access/transam/xlog.c:3800

if (wal_recycle && *endlogSegNo <= recycleSegNo &&
    XLogCtl->InstallXLogFileSegmentActive &&
    InstallXLogFileSegment(endlogSegNo, path, true, recycleSegNo, insertTLI))
{
    CheckpointStats.ckpt_segs_recycled++;
    (*endlogSegNo)++;
}
else { durable_unlink(path, LOG); CheckpointStats.ckpt_segs_removed++; }

Archiving is marker-file based (xlogarchive.c). When a segment is fully written, XLogArchiveNotify (xlogarchive.c:453) creates an empty <seg>.ready in archive_status and wakes the archiver; the archiver copies the segment then renames the marker to <seg>.done. A segment may only be deleted/recycled once that .done exists.

pg_wal layout. Segment files (24 hex chars) sit directly under pg_wal/; pg_wal/archive_status/ holds the .ready/.done handshake. Recycling renames an old segment forward instead of unlink-then-recreate, so steady-state WAL churn does almost no zero-fill I/O.

20.2.5 Transaction records: PREPARE / COMMIT / ABORT and their redo

A commit record: a fixed header followed by optional, flagged sub-records

xl_xact_commit — xact_time · tablespace

xinfo — which parts follow

subxacts? — HAS_SUBXACT

files to drop? — HAS_RELFILELOCATORS

2PC xid/gid? — HAS_TWOPHASE / GID

distrib xid? — HAS_DISTRIB (CBDB)

All transaction-completion records belong to RM_XACT_ID, with the kind in the info byte: XLOG_XACT_COMMIT (0x00), XLOG_XACT_PREPARE (0x10), XLOG_XACT_ABORT (0x20), XLOG_XACT_COMMIT_PREPARED (0x30), XLOG_XACT_ABORT_PREPARED (0x40) — plus Cloudberry’s DISTRIBUTED_COMMIT/_FORGET in the same family (§20.2.7+).

A commit record is built by XactLogCommitRecord (xact.c:6964): a fixed xl_xact_commit header (commit time + tablespace OID) followed by optional sub-records flagged in xl_xinfo.xinfo (subxacts, relfilelocators to drop, dropped stats, invalidation messages, twophase XID/GID, replication origin). The on-disk header is intentionally minimal:

src/include/access/xact.h:347

typedef struct xl_xact_commit
{
    TimestampTz xact_time;      /* time of commit */
    Oid tablespace_oid_to_delete_on_commit;
    /* xl_xact_xinfo follows default */
    /* xl_xact_subxacts follows if XINFO_HAS_SUBXACT */
    /* xl_xact_twophase follows if XINFO_HAS_TWOPHASE */
    /* twophase_gid follows if XINFO_HAS_GID ... */
} xl_xact_commit;

XactLogAbortRecord (xact.c:7178) is the symmetric builder for xl_xact_abort. The PREPARE record is different — it is assembled in twophase.c’s EndPrepare (twophase.c:1186) from the in-memory 2PC chunks and flushed durably before the transaction is marked prepared:

src/backend/access/transam/twophase.c:1242

XLogBeginInsert();
for (record = records.head; record != NULL; record = record->next)
    XLogRegisterData(record->data, record->len);
gxact->prepare_end_lsn = XLogInsert(RM_XACT_ID, XLOG_XACT_PREPARE);
XLogFlush(gxact->prepare_end_lsn);
/* If we crash now, we have prepared: WAL replay will fix things */

The committing half of a prepared transaction reuses the shared builder: RecordTransactionCommitPrepared (twophase.c:2436) calls XactLogCommitRecord(...) with the twophase XID+GID (making it COMMIT_PREPARED) then flushes; RecordTransactionAbortPrepared mirrors it. On redo, xact_redo (xact.c:7620) masks the info byte and dispatches: COMMIT/COMMIT_PREPARED → xact_redo_commit (mark the tree committed in clog, apply commit-ts, process dropped files & invalidations); ABORT/ABORT_PREPARED → xact_redo_abort; the *_PREPARED variants additionally PrepareRedoRemove the gxact entry, and PREPARE re-creates one with PrepareRedoAdd:

src/backend/access/transam/xact.c:7636

else if (info == XLOG_XACT_COMMIT_PREPARED)
{
    xl_xact_commit *xlrec = (xl_xact_commit *) XLogRecGetData(record);
    xl_xact_parsed_commit parsed;
    ParseCommitRecord(XLogRecGetInfo(record), xlrec, &parsed);
    xact_redo_commit(&parsed, parsed.twophase_xid, record->EndRecPtr, ...);
    LWLockAcquire(TwoPhaseStateLock, LW_EXCLUSIVE);
    PrepareRedoRemove(parsed.twophase_xid, false);
    LWLockRelease(TwoPhaseStateLock);
}

Completion records describe only catalog/clog/file side effects — not page contents — so xact_redo asserts no backup blocks; the heap/index page changes were logged separately by their own resource managers earlier in the stream. The distributed variants reuse this very same xl_xact_commit payload and xact_redo dispatch, which is why these single-node records are the structural baseline for §20.2.7.

20.2.6 Checkpoints: online vs shutdown, triggering, and restartpoints

Recovery replays forward from the last checkpoint’s REDO point

← older WAL — already in data files · deletable

REDO point — checkPoint.redo

WAL after REDO → — replayed at recovery

A checkpoint is the durability anchor of WAL. It records a REDO point (an LSN) and guarantees every change logically before it is already flushed to the data files — so crash recovery never replays WAL older than the last checkpoint’s REDO point: it reads the checkpoint from pg_control, seeks to checkpoint.redo, and replays forward. The driver is CreateCheckPoint() (xlog.c:6940). The on-disk body is the CheckPoint struct, whose redo is the start point and whose other fields capture cluster-wide counters — including the CBDB-specific nextGxid:

src/include/catalog/pg_control.h:40

typedef struct CheckPoint
{
    XLogRecPtr  redo;           /* REDO start point */
    TimeLineID  ThisTimeLineID;
    ...
    FullTransactionId nextXid;
    DistributedTransactionId nextGxid;   /* next free gxid (CBDB) */
    Oid         nextOid;
    TransactionId oldestXid;
    /* IN XLOG RECORD, MORE DATA FOLLOWS FOR DTM CHECKPOINT */
} CheckPoint;

CreateCheckPoint acquires all WAL insertion locks, reads the current insert position and sets checkPoint.redo = curInsert, advances the shared RedoRecPtr, then releases the locks so backends keep inserting WAL while the checkpoint flushes. It gathers the counters, calls CheckPointGuts() to flush all dirty buffers/SLRUs/CLOG, collects the CBDB DTX payload via getDtxCheckPointInfo() (§20.4.4), and writes the record — the info byte being the only on-disk difference between online and shutdown:

src/backend/access/transam/xlog.c:7310

XLogBeginInsert();
XLogRegisterData((char *) (&checkPoint), sizeof(checkPoint));
/* Cloudberry checkpoints have extra info */
XLogRegisterData((char *) dtxCheckPointInfo, dtxCheckPointInfoSize);
recptr = XLogInsert(RM_XLOG_ID,
                    shutdown ? XLOG_CHECKPOINT_SHUTDOWN : XLOG_CHECKPOINT_ONLINE);
XLogFlush(recptr);

A shutdown checkpoint sets ControlFile->state = DB_SHUTDOWNING→DB_SHUTDOWNED, forbids further WAL writes, and runs with no concurrent activity; an online checkpoint runs alongside live traffic, logs a running-xacts standby snapshot, and folds cached id-allocation slack (nextGxid += GxidCount, nextOid += oidCount) so the recovered cluster won’t reissue ids. Either way it finally writes pg_control under ControlFileLock (UpdateControlFile()) — that atomic write is what makes the checkpoint official — then recycles old WAL (§20.2.4).

The guarantee: once pg_control points at this checkpoint, recovery may start at checkPoint.redo assured all earlier changes are in the data files — which is exactly why older WAL can be deleted.

Triggering. The dedicated checkpointer (CheckpointerMain, checkpointer.c) performs a checkpoint when: a request was posted in shared memory; checkpoint_timeout elapsed (cause TIME); or WAL volume crossed the max_wal_size-derived threshold — detected in the write path by XLogCheckpointNeeded(), which calls RequestCheckpoint(CHECKPOINT_CAUSE_XLOG):

src/backend/access/transam/xlog.c:2369

if (IsUnderPostmaster && XLogCheckpointNeeded(openLogSegNo))
{
    (void) GetRedoRecPtr();
    if (XLogCheckpointNeeded(openLogSegNo))
        RequestCheckpoint(CHECKPOINT_CAUSE_XLOG);
}

• CHECKPOINT_IS_SHUTDOWN — shutdown checkpoint
• CHECKPOINT_END_OF_RECOVERY — shutdown-like, issued while exiting recovery
• CHECKPOINT_IMMEDIATE — skip the spread-out throttling
• CHECKPOINT_FORCE — run even with no WAL activity
• CHECKPOINT_WAIT — caller blocks for completion
• CHECKPOINT_CAUSE_XLOG / CHECKPOINT_CAUSE_TIME — diagnostic cause bits

Restartpoints. A standby in recovery cannot write its own WAL, so when it replays a checkpoint record it instead creates a restartpoint via CreateRestartPoint() (xlog.c:7677): it reuses the replayed checkpoint’s REDO point, runs the same CheckPointGuts() flush, and updates pg_control so a crash during recovery can restart replay from there. It is skipped if no new checkpoint has been replayed since the last restartpoint. Crucially, restartpoint WAL recycling retreats the horizon to the later of replayed and received LSNs so it never deletes segments the walreceiver still needs — tying checkpoints into streaming replication (§20.3).

20.2.7 The distributed-commit story, precisely

The runtime protocol that decides when a distributed transaction commits — the QD’s DtxState and QE’s DtxContext state machines — is the subject of §12.2. This section covers the durable half: how that commit is encoded in WAL and replayed.

There are exactly nine xact opcodes (src/include/access/xact.h:181). Note that XLOG_XACT_DISTRIBUTED_COMMIT (0x70) does exist as its own opcode:

src/include/access/xact.h:181 — xact WAL opcodes

#define XLOG_XACT_COMMIT             0x00
#define XLOG_XACT_PREPARE            0x10
#define XLOG_XACT_ABORT              0x20
#define XLOG_XACT_COMMIT_PREPARED    0x30
#define XLOG_XACT_ABORT_PREPARED     0x40
#define XLOG_XACT_ASSIGNMENT         0x50
#define XLOG_XACT_INVALIDATIONS      0x60
#define XLOG_XACT_DISTRIBUTED_COMMIT 0x70
#define XLOG_XACT_DISTRIBUTED_FORGET 0x80

All commit variants are produced by the same builder, XactLogCommitRecord() (src/backend/access/transam/xact.c:6963), which picks the opcode up front: a prepared distributed transaction selects XLOG_XACT_DISTRIBUTED_COMMIT, otherwise a plain or prepared local commit. The global xid is not a separate record body — it reuses the ordinary xl_xact_commit layout and is carried as an optional sub-record gated by an xinfo flag:

src/backend/access/transam/xact.c:7084 (and :7154 for serialization)

/* include distributed xid if there's one */
if (distrib_xid != InvalidDistributedTransactionId)
{
    xl_xinfo.xinfo |= XACT_XINFO_HAS_DISTRIB;   /* (1U << 9), xact.h:210  */
    xl_distrib.distrib_xid = distrib_xid;       /* xl_xact_distrib, xact.h:269 */
}
...
if (xl_xinfo.xinfo & XACT_XINFO_HAS_DISTRIB)
    XLogRegisterData((char *) (&xl_distrib), sizeof(xl_xact_distrib));

So the gxid rides in any commit record that has one; the dedicated 0x70 opcode exists because the coordinator’s prepared distributed commit needs the extra redo bookkeeping in §20.2.4, whereas an ordinary segment commit merely recording its distrib xid does not.

Correction (confirmed in code): an earlier draft claimed there is no XLOG_XACT_DISTRIBUTED_COMMIT record type. That is wrong — opcode 0x70 exists (xact.h:188) with its own redo branch xact_redo_distributed_commit. What is true: the distributed commit reuses the xl_xact_commit body and carries the gxid via XACT_XINFO_HAS_DISTRIB + xl_xact_distrib; the opcode tag is what triggers redoDistributedCommitRecord().

20.2.8 The distributed-forget record

Phase 2 of 2PC leaves a second trace: once every segment has acknowledged its COMMIT PREPARED, the coordinator writes a standalone XLOG_XACT_DISTRIBUTED_FORGET (0x80) record carrying only the gxid (from doInsertForgetCommitted(), src/backend/cdb/cdbtm.c:533):

src/backend/access/transam/xact.c:1877

XLogRecPtr
RecordDistributedForgetCommitted(DistributedTransactionId gxid)
{
    xl_xact_distributed_forget xlrec;
    XLogRecPtr  recptr;

    xlrec.gxid = gxid;
    XLogBeginInsert();
    XLogRegisterData((char *) &xlrec, sizeof(xl_xact_distributed_forget));
    recptr = XLogInsert(RM_XACT_ID, XLOG_XACT_DISTRIBUTED_FORGET);
    if (synchronous_commit >= SYNCHRONOUS_COMMIT_REMOTE_APPLY)
        XLogFlush(recptr);
    return recptr;
}

20.2.9 Redo and the committed-gxid array

xact_redo() (xact.c:7619) dispatches on the opcode. XLOG_XACT_DISTRIBUTED_COMMIT replays the local commit (if any) then calls redoDistributedCommitRecord(gxid); XLOG_XACT_DISTRIBUTED_FORGET calls redoDistributedForgetCommitRecord(gxid). These maintain an in-memory set of committed-but-not-yet-forgotten global xids — shmCommittedGxidArray / *shmNumCommittedGxacts — which is exactly the set of in-doubt distributed transactions recovery must drive through 2PC phase 2 (see §20.4.4).

src/backend/cdb/cdbdtxrecovery.c:570 (forget redo, excerpt)

void
redoDistributedForgetCommitRecord(DistributedTransactionId gxid)
{
    for (int i = 0; i < *shmNumCommittedGxacts; i++)
        if (gxid == shmCommittedGxidArray[i])
        {
            (*shmNumCommittedGxacts)--;                       /* swap-remove */
            if (i != *shmNumCommittedGxacts)
                shmCommittedGxidArray[i] = shmCommittedGxidArray[*shmNumCommittedGxacts];
            /* on a hot-standby QD, advance latestCompletedGxid for snapshots */
            return;
        }
}

The two Cloudberry rmgrs from §20.2.2 surface here: the distributed log SLRU emits DISTRIBUTEDLOG_ZEROPAGE/_TRUNCATE (replayed by DistributedLog_redo), and Append-Optimized storage emits XLOG_APPENDONLY_INSERT/_TRUNCATE — whose appendonly_redo() skips replay in crash-recovery-only mode (if (IsCrashRecoveryOnly()) return;, cdbappendonlyxlog.c:210) because AO fsync happens on file close, so these records exist mainly to ship AO data to mirrors.

Bridge to §20.3. WAL makes every committed change recoverable on the same node — but a node’s disk can die outright, and replay needs the log to still exist somewhere. So the next pillar streams this WAL to a second machine the instant it is flushed: streaming replication.