The Dispatcher: Gangs, Gang Reuse and Async Dispatch

The dispatcher is the coordinator’s hardest job and the first thing to saturate on a busy cluster.

§13.3 summarised dispatch in one sentence: the QD serialises the plan and ships it to gangs. That sentence hides three engineering problems. Where do the executor processes come from, given that forking a backend costs milliseconds and a statement may need nine of them? How does one QD talk to nine processes at once without its own setup latency becoming the sum of nine round trips? And when one of the nine dies, how does the failure become a single error message at the client instead of a hang?

The answers are a gang (a group of QEs, one per segment, allocated as a unit), a per-session cache of idle QEs that survives statement boundaries, and an asynchronous libpq dispatcher built on PQsendQuery plus poll(). All three live in src/backend/cdb/dispatcher/ — cdbgang.c (1246 lines), cdbgang_async.c (427), cdbdisp_query.c (1864), cdbdisp_async.c (1226), cdbconn.c (873) — with the idle-QE pool in cdbutil.c. There is even a design note in the tree: src/backend/cdb/dispatcher/README.md.

One client connection, eleven backends. Two statements were run first (a redistributed self-join, then a join against a coordinator-only catalog); gp_backend_info() then reports every QE this session is holding.

select count(*) from m192_t a join m192_t b on a.b = b.b;
select count(*) from m192_t t, gp_segment_configuration c where t.a = c.dbid;
select * from gp_backend_info() order by id;

   count
-----------
 514285716
(1 row)

 count
-------
     4
(1 row)

 id | type | content |    host    | port |  pid
----+------+---------+------------+------+-------
 -1 | Q    |      -1 | cbdb-node1 | 7100 | 22404
  0 | w    |       0 | cbdb-node1 | 7102 | 22408
  1 | w    |       1 | cbdb-node1 | 7103 | 22409
  2 | w    |       2 | cbdb-node1 | 7104 | 22410
  3 | r    |       0 | cbdb-node1 | 7102 | 22414
  4 | r    |       1 | cbdb-node1 | 7103 | 22415
  5 | r    |       2 | cbdb-node1 | 7104 | 22416
  6 | r    |       0 | cbdb-node1 | 7102 | 22420
  7 | r    |       1 | cbdb-node1 | 7103 | 22421
  8 | r    |       2 | cbdb-node1 | 7104 | 22422
  9 | R    |      -1 | cbdb-node1 | 7100 | 23012
(11 rows)

Two setups, two networks. Gang setup and interconnect setup are different things and it is worth nailing the boundary now. Gang setup is QD → QE over libpq, to each segment’s ordinary postmaster port (7102/7103/7104 here); it is what this section is about. Interconnect setup is QE ↔ QE over separate, ephemeral listeners; §19.5 owns it. The two meet at exactly one point: when the QD finishes a libpq connection it immediately asks the new QE for its motion listener port (cdbconn.c:297) and refuses the gang if there isn’t one. That port is then shipped to every peer inside the slice table.

19.2.1 One writer, many readers

A gang is deliberately thin — a type, a size, and an array of connection descriptors (cdbgang.h:32). All the interesting policy is in the type, five values that have not changed in fifteen years:

src/include/nodes/plannodes.h:231

typedef enum GangType
{
	GANGTYPE_UNALLOCATED,       /* a root slice executed by the qDisp */
	GANGTYPE_ENTRYDB_READER,    /* a 1-gang with read access to the entry db */
	GANGTYPE_SINGLETON_READER,	/* a 1-gang to read the segment dbs */
	GANGTYPE_PRIMARY_READER,    /* a 1-gang or N-gang to read the segment dbs */
	GANGTYPE_PRIMARY_WRITER		/* the N-gang that can update the segment dbs */
} GangType;

GANGTYPE_UNALLOCATED — Not a gang at all: the slice runs in the QD itself. Slice 0 of every plan is this. InventorySliceTree still fills in a process list for it, from getCdbProcessesForQD (cdbgang.c:685), because the QD is a motion endpoint even though nobody dispatched to it.

GANGTYPE_ENTRYDB_READER — A one-process gang on the coordinator, used when a plan must scan a coordinator-only relation. That is row id=9, type=R, content=-1, port=7100 in the capture above: the coordinator connected to its own postmaster and forked a backend for itself.

GANGTYPE_SINGLETON_READER — A one-process gang on one segment — what you get for a numsegments=1 slice, e.g. after direct dispatch (§16.4) or a Gather into a single segment.

GANGTYPE_PRIMARY_READER — The workhorse: N processes, one per segment in the slice’s segment list. Read-only with respect to the distributed transaction.

GANGTYPE_PRIMARY_WRITER — Also N processes, but this gang owns the transaction on each segment. A session has at most one, and reader gangs cannot exist without it.

That last asymmetry is the load-bearing one, and it exists because a transaction has to live somewhere. Each segment is a full PostgreSQL instance with its own XID space, its own lock table and its own snapshot machinery; nine peer processes on three segments cannot each independently open a transaction and still be one atomic unit. So on every segment exactly one QE — the writer — begins the local transaction, holds the locks on behalf of the whole session (see the lockHolderProcPtr discussion at execUtils.c:1933), and publishes its local snapshot into shared memory. Readers attach to it. The branch is taken once per backend, during InitPostgres:

src/backend/utils/init/postinit.c:1453

	else if (Gp_role == GP_ROLE_EXECUTE)
	{
		if (Gp_is_writer)
		{
			addSharedSnapshot("Writer qExec", gp_session_id);
		}
		else
		{
			/*
			 * NOTE: This assumes that the Slot has already been
			 *       allocated by the writer.  ...
			 */
			lookupSharedSnapshot("Reader qExec", "Writer qExec", gp_session_id);
		}
	}

Read that comment as a scheduling constraint on the dispatcher, because that is what it is. The writer must be created first, or every reader on that segment fails its SharedSnapshotLookup (sharedsnapshot.c:472). The QD honours it explicitly: AssignGangs calls AssignWriterGangFirst (execUtils.c:1966) which descends the slice tree looking for the GANGTYPE_PRIMARY_WRITER slice before the ordinary top-down walk in InventorySliceTree (execUtils.c:1997) allocates anything else. The comment at execUtils.c:1942 gives the case that forced this: a writable CTE, where the updating slice sits below a reading slice, so a naive top-down walk would hand the writer QEs to the wrong slice. The same constraint explains the rest of this section’s behaviour — a session’s writer gang cannot be recycled away mid-transaction (cdbgang.c:761 spells it out), and killing a writer resets the whole session rather than just failing a statement.

Writer-ness is a property of the QE, not of the gang. A plain SELECT asks for SEGMENTTYPE_ANY and its slices are typed GANGTYPE_PRIMARY_READER — the debug log for the self-join above shows gangtype=3 for all three slices — yet gp_backend_info() still reports three w backends. The reason is cdbutil.c:863: isWriter = contentId == -1 ? false : (cdbinfo->numIdleQEs == 0 && cdbinfo->numActiveQEs == 0). The first QE ever created on a segment in this session becomes the writer, whatever asked for it. A DML statement is the case that forces the issue: an INSERT logs segment type = 1 (SEGMENTTYPE_EXPLICT_WRITER) and gangtype=4, and with direct dispatch gangsize=1.

19.2.2 Bringing a gang into being

The allocation path is four calls deep and each layer has exactly one job. AssignGangs (execUtils.c:1909) walks the slice tree; AllocateGang (cdbgang.c:107) maps the slice’s GangType to a SegmentType request and records the gang on the dispatcher state so it can be recycled later; buildGangDefinition (cdbgang.c:263) loops over the segment list calling cdbcomponent_allocateIdleQE(contentId, segmentType); and that function either pops a descriptor off the segment’s freelist or manufactures a fresh one. Note the division of labour in buildGangDefinition: it only reserves descriptors. Nothing is connected yet. That is what makes the next step batchable.

A QE is not a special kind of process. It is an ordinary backend, forked by the segment’s own postmaster in response to an ordinary libpq connection — cdbconn.c:283 calls PQconnectStartParams. What makes it a QE is two extra connection parameters that CBDB added to libpq itself (fe-connect.c:411 and :419): gpqeid and options/diff_options. gpqeid is a semicolon-delimited string built without any palloc or elog, because historically it ran on a thread:

src/backend/cdb/dispatcher/cdbgang.c:498

	len = snprintf(buf, bufsz, "%d;" TIMESTAMP_FORMAT ";%s;%d;%d;%d;%d",
				   gp_session_id, PgStartTime,
				   (is_writer ? "true" : "false"), identifier, hostSegs, icHtabSize,
				   qeidx);

postmaster.c:2574 picks gpqeid out of the startup packet and cdbgang_parse_gpqeid_params (cdbgang.c:533) applies it with PGC_S_OVERRIDE before authentication — which is how gp_is_writer is already set by the time postinit.c:1455 needs it. The options string is a different animal: makeOptions (cdbgang.c:442) emits -c name=value for every GUC flagged GUC_GPDB_NEED_SYNC, at its reset value, plus the QD’s own hostname and port; any GUC the session has since changed goes into a second diff_options list which the QE applies at PGC_S_SESSION (postinit.c:1718) so the reset values stay intact. On this cluster that string is 193 -c pairs, 6380 bytes, sent once per connection. Setting gp_log_gang = debug prints the whole thing, along with the interconnect port the QD just learned:

gp_log_gang=debug on a fresh connection. The 6380-byte options string is elided; every fragment shown is verbatim and in its real (alphabetical) order.

PGOPTIONS='-c gp_log_gang=debug -c client_min_messages=log' psql -p 7100 -d postgres
select count(*) from m192_s;

LOG:  AllocateGang begin.
LOG:  createGang size = 3, segment type = 3
LOG:  buildGangDefinition:Starting 3 qExec processes for gang
LOG:  buildGangDefinition done
LOG:  Connected to seg1 10.100.0.3:7103 motionListenerPorts=38140/0 with options
       -c gp_qd_hostname=10.100.0.3 -c gp_qd_port=7100
       -c allow_dml_directory_table=false -c array_nulls=true
       [... 187 more -c name=value pairs ...]
       -c gp_interconnect_type=udpifc -c gp_max_packet_size=8192
       -c statement_mem=128000 -c work_mem=32768
LOG:  createGang: 3 processes requested; 3 successful connections 0 in recovery
LOG:  AllocateGang end.
LOG:  getCdbProcessList slice1 gangtype=3 gangsize=3
LOG:  Gang assignment: slice1 seg0 10.100.0.3:58041 pid=14392
LOG:  Gang assignment: slice1 seg1 10.100.0.3:38140 pid=14393
LOG:  Gang assignment: slice1 seg2 10.100.0.3:50190 pid=14394

Two details in that log repay attention. motionListenerPorts=38140/0 packs two ports into one word (cdbconn.c:305 masks the low and high halves) — the UDP listener and the TCP one, only one of which is populated depending on gp_interconnect_type. And the Gang assignment lines print 10.100.0.3:58041, not :7102 — by that point the QD has stopped caring about postmaster ports and is recording the interconnect endpoints it will hand to the peers. Connections are opened in a batch, exactly as buildGangDefinition set up: cdbgang_async.c fires cdbconn_doConnectStart for all N segments (:177), then drives them to completion in a single poll() loop (:315) with a deadline computed from gp_segment_connect_timeout (180 s here; getPollTimeout, :413). The per-QE elapsed time is recorded, and gp_print_create_gang_time prints it:

gp_print_create_gang_time=on: the same three-slice self-join, run twice. Timings are ratios on one —enable-cassert build, not benchmarks.

set gp_print_create_gang_time = on;
select count(*) from m192_s a join m192_s b on a.b=b.b;
select count(*) from m192_s a join m192_s b on a.b=b.b;

INFO:  The shortest establish conn time: 6.04 ms, segindex: 0,
       The longest  establish conn time: 6.57 ms, segindex: 2
INFO:  (Slice2) The shortest establish conn time: 5.21 ms, segindex: 0,
                The longest  establish conn time: 5.81 ms, segindex: 1
INFO:  (Slice3) The shortest establish conn time: 5.01 ms, segindex: 0,
                The longest  establish conn time: 5.77 ms, segindex: 2
INFO:  (Slice1) is reused
 count
-------
 10002
(1 row)

INFO:  (Slice2) is reused
INFO:  (Slice3) is reused
INFO:  (Slice1) is reused
 count
-------
 10002
(1 row)

Three segments, and the slowest connection in a gang finishes 0.5–0.8 ms after the fastest — the fork-and-authenticate work overlapped. Serially it would have cost the sum. The gang-creation loop also has a retry path most people never see: a PGRES_POLLING_FAILED whose error text matches POSTMASTER_IN_STARTUP_MSG or POSTMASTER_IN_RECOVERY_MSG (segment_failure_due_to_recovery, cdbgang.c:168 — it really does grep the message string, and the comment apologises for it) is treated as retryable, and after gp_gang_creation_retry_timer the whole create_gang_retry label runs again. Everything else, including the deadline, becomes ERROR: failed to acquire resources on one or more segments with the reason in errdetail — timeout expired (cdbgang_async.c:302), a libpq message, or FTS detected one or more segments are down (:378). I did not induce a segment-down condition on this cluster.

19.2.3 Gangs outlive statements

This is the part that is most often got wrong. A gang is not created per statement. cdbdisp_destroyDispatcherState recycles a session’s gangs rather than dropping them, and cdbcomponent_recycleIdleQE pushes each descriptor — live libpq connection and all — back onto a per-segment freelist. The next statement pops them straight off. The proof is that the pids do not move:

Nine QEs, two statements, same pids. Then gp_vmem_idle_resource_timeout is lowered to 100 ms and the client goes idle for two seconds.

select count(*) from m192_s a join m192_s b on a.b=b.b;
select count(*) as qes, string_agg(pid::text,' ' order by id) as pids
  from gp_backend_info() where content>=0;
select count(*) from m192_s a join m192_s b on a.b=b.b;   -- same statement again
select count(*) as qes, string_agg(pid::text,' ' order by id) as pids
  from gp_backend_info() where content>=0;
set gp_vmem_idle_resource_timeout = 100;   -- ms; default is 600000
\! sleep 2

 qes |                         pids
-----+-------------------------------------------------------
   9 | 22031 22032 22033 22037 22038 22039 22043 22044 22045
(1 row)

 qes |                         pids
-----+-------------------------------------------------------
   9 | 22031 22032 22033 22037 22038 22039 22043 22044 22045
(1 row)

SET
 qes |       pids
-----+-------------------
   3 | 22031 22032 22033
(1 row)

Nine, then the same nine, then three — and the three survivors are precisely the writers, pids 22031 22032 22033. That is DisconnectAndDestroyUnusedQEs calling cdbcomponent_cleanupIdleQEs(false), whose false means do not touch the writers; §19.2.1 explains why it can’t. Whether a recycled QE goes back on the freelist or dies is one decision function, cleanupQE plus the length check that follows it:

src/backend/cdb/cdbutil.c:941

	if (forceDestroy || !cleanupQE(segdbDesc))
		goto destroy_segdb;

	/* If freelist length exceed gp_cached_gang_threshold, destroy it */
	maxLen = segdbDesc->segindex == -1 ?
					MAX_CACHED_1_GANGS : gp_cached_gang_threshold;
	if (!isWriter && list_length(cdbinfo->freelist) >= maxLen)
		goto destroy_segdb;

What ends a QE’s life	Where	Note
Explicit forceDestroy from the caller	RecycleGang(gp, true), cdbgang.c:945	Set for extended-query/named-portal gangs, and for a partly built gang whose PG_CATCH fired
Connection already bad, or FTS says the segment is down	cleanupQE, cdbutil.c:895 / :899	A dead connection is never cached
Reader’s peak memory above gp_vmem_protect_segworker_cache_limit (500 MB)	cdbutil.c:903	Uses libpq’s mop_high_watermark; a memory-hungry reader is dropped, not reused
QE still busy after a 20-attempt result discard	cdbconn_discardResults, cdbutil.c:908	Logs cleaning up segN while it is still busy
Freelist already at gp_cached_segworkers_threshold (5)	cdbutil.c:947	Only readers; writers are exempt and always cached
Session idle longer than gp_vmem_idle_resource_timeout (600000 ms)	DisconnectAndDestroyUnusedQEs, cdbgang.c:758	Readers only
Writer gang lost	markCurrentGxactWriterGangLost, cdbutil.c:992	Forces resetSessionForPrimaryGangLoss and a new gp_session_id

The freelist cap is easy to see at work. Drop it to 1 and the reader gangs can no longer all be cached, so their pids churn from statement to statement while the writers stay put:

gp_cached_segworkers_threshold=1: only one idle reader per segment may be cached, so the second statement has to fork fresh readers.

set gp_cached_segworkers_threshold = 1;   -- real GUC name; default 5
select count(*) from m192_s a join m192_s b on a.b=b.b;
select type, string_agg(pid::text,' ' order by id) as pids
  from gp_backend_info() where content>=0 group by type order by type;
select count(*) from m192_s a join m192_s b on a.b=b.b;
select type, string_agg(pid::text,' ' order by id) as pids
  from gp_backend_info() where content>=0 group by type order by type;

 type |       pids
------+-------------------
 r    | 23454 23455 23456
 w    | 23440 23441 23442
(2 rows)

 type |       pids
------+-------------------
 r    | 23464 23465 23466
 w    | 23440 23441 23442
(2 rows)

A cached QE is a process with stale GUC state, so reuse imposes an obligation: every SET must be replayed to gangs that already exist. CdbDispatchSetCommand (cdbdisp_query.c:303) does exactly that, dispatching the text of the SET to all live gangs except extended-query ones — and marking those non-recyclable afterwards (cdbdisp_markNamedPortalGangsDestroyed, :283), because they missed the update:

The same three QEs before and after a SET; the value changed underneath them.

select seg, setting from gp_dist_random('m192_wm') order by seg;  -- view over pg_settings
select string_agg(pid::text,' ' order by id) as qe_pids
  from gp_backend_info() where content>=0;
set work_mem = '64MB';
select seg, setting from gp_dist_random('m192_wm') order by seg;
select string_agg(pid::text,' ' order by id) as qe_pids
  from gp_backend_info() where content>=0;

 seg | setting
-----+---------
   0 | 32768
   1 | 32768
   2 | 32768
(3 rows)

      qe_pids
-------------------
 23475 23476 23477
(1 row)

SET
 seg | setting
-----+---------
   0 | 65536
   1 | 65536
   2 | 65536
(3 rows)

      qe_pids
-------------------
 23475 23476 23477
(1 row)

Put a number on what reuse buys. Establishing one QE cost 5–7 ms above, and it carried 6380 bytes of GUC options; the three-slice self-join needs nine of them. Without the cache, every statement in a session would pay that — and on a real cluster with 48 segments, a nine-slice plan means 432 forks and authentications per statement. This is the deeper half of §13.2’s point that a prepared statement reuses more than a plan: on CBDB, a warm session is warm in processes, and the first statement after connecting is measurably the expensive one.

19.2.4 Async dispatch: one QD, N sockets

With gangs in hand, the QD has nine open libpq connections and one serialised plan. It does not walk them in order waiting for each to finish. cdbdisp_dispatchToGang_async (cdbdisp_async.c:303) loops over the gang and calls dispatchCommand on each member, which is a thin wrapper over PQsendGpQuery_shared(conn, query_text, query_text_len, true) (:681) — the true being non-blocking, so a congested socket does not stall the loop. Only after every slice has been kicked off does the QD start reading. Two loops make that work: cdbdisp_waitDispatchFinish_async (:218) polls for POLLOUT and flushes the leftovers, then checkDispatchResult (:450) polls for POLLIN:

src/backend/cdb/dispatcher/cdbdisp_async.c:551

			/*
			 * Add socket to fd_set if still connected.
			 */
			sock = PQsocket(conn);
			Assert(sock >= 0);
			fds[nfds].fd = sock;
			fds[nfds].events = POLLIN;
			nfds++;
		}
		/* Break out when no QEs still running or required QEs acked. */
		if (nfds <= 0 || ...)
			break;
		n = poll(fds, nfds, timeout);

Whichever descriptors come back readable are handed to handlePollSuccess → processResults → PQconsumeInput / PQisBusy (:1006, :1019). Nothing enforces an order. Turning on gp_log_gang = debug shows the whole shape of a dispatch in one trace:

One statement, three gangs, nine QEs. The interleaved ‘PQsocket says’ and ’-> ok SELECT 0’ lines have been dropped to keep the trace readable; the ordering of the lines shown is exactly as captured.

PGOPTIONS='-c gp_log_gang=debug -c client_min_messages=log' psql -p 7100 -d postgres
select count(*) from m192_s a join m192_s b on a.b=b.b;

LOG:  createGang: 3 processes requested; 3 successful connections 0 in recovery
LOG:  getCdbProcessList slice1 gangtype=3 gangsize=3
LOG:  createGang: 3 processes requested; 3 successful connections 0 in recovery
LOG:  getCdbProcessList slice2 gangtype=3 gangsize=3
LOG:  createGang: 3 processes requested; 3 successful connections 0 in recovery
LOG:  getCdbProcessList slice3 gangtype=3 gangsize=3
LOG:  Query plan size to dispatch: 2KB
LOG:  Command dispatched to QE (seg0 slice2 10.100.0.3:7102 pid=23814)
LOG:  Command dispatched to QE (seg1 slice2 10.100.0.3:7103 pid=23815)
LOG:  Command dispatched to QE (seg2 slice2 10.100.0.3:7104 pid=23816)
LOG:  Command dispatched to QE (seg0 slice3 10.100.0.3:7102 pid=23822)
LOG:  Command dispatched to QE (seg1 slice3 10.100.0.3:7103 pid=23823)
LOG:  Command dispatched to QE (seg2 slice3 10.100.0.3:7104 pid=23824)
LOG:  Command dispatched to QE (seg0 slice1 10.100.0.3:7102 pid=23808)
LOG:  Command dispatched to QE (seg1 slice1 10.100.0.3:7103 pid=23809)
LOG:  Command dispatched to QE (seg2 slice1 10.100.0.3:7104 pid=23810)
LOG:  processResults says we are finished with 4 of 9 (seg0 slice3 ... pid=23822)
LOG:  processResults says we are finished with 5 of 9 (seg1 slice3 ... pid=23823)
LOG:  processResults says we are finished with 6 of 9 (seg2 slice3 ... pid=23824)
LOG:  processResults says we are finished with 2 of 9 (seg1 slice2 ... pid=23815)
LOG:  processResults says we are finished with 3 of 9 (seg2 slice2 ... pid=23816)
LOG:  processResults says we are finished with 1 of 9 (seg0 slice2 ... pid=23814)
LOG:  processResults says we are finished with 9 of 9 (seg2 slice1 ... pid=23810)
LOG:  processResults says we are finished with 7 of 9 (seg0 slice1 ... pid=23808)
LOG:  processResults says we are finished with 8 of 9 (seg1 slice1 ... pid=23809)

Four facts fall out of that trace. All three gangs are allocated before anything is dispatched — AssignGangs runs at cdbdisp_query.c:1110, ahead of serialisation. The plan is serialised exactly once: one Query plan size to dispatch line for nine sends, and PQsendGpQuery_shared does not even copy the buffer — it points each PGconn->outBuffer at the shared text (cdbpq.c:45) and restores the original afterwards. Slices are dispatched leaves-first: slice2, slice3, then the root slice1, because fillSliceVector sorts by number of dependent children (compare_slice_order, cdbdisp_query.c:719) so producers are running before their consumer starts pulling. And completions arrive out of order — 4,5,6,2,3,1,9,7,8 — which is what asynchrony looks like from the inside. A synchronous dispatcher would have serialised nine PQexec round trips, adding nine setup latencies to a statement whose actual work is embarrassingly parallel; the legacy thread-pool knob (gp_connections_per_thread) that once controlled this no longer exists in the tree. One last detail worth holding on to: every QE’s libpq result here is ok SELECT 0, including the root slice’s senders. Tuples never travel on these sockets. libpq carries control and completion; the data goes over the interconnect (§19.5).

19.2.5 What actually crosses the wire

“Dispatching the plan” is a single message, type 'M', hand-marshalled by buildGpQueryString (cdbdisp_query.c:868). Its size computation is the most compact inventory of a dispatch payload in the tree:

src/backend/cdb/dispatcher/cdbdisp_query.c:921

	total_query_len = 1 /* 'M' */ +
		sizeof(len) /* message length */ +
		sizeof(gp_command_count) +
		sizeof(sessionUserId) + sizeof(outerUserId) + sizeof(currentUserId) +
		sizeof(n32) * 2 /* currentStatementStartTimestamp */ +
		sizeof(is_hs_dispatch) +
		sizeof(command_len) + sizeof(plantree_len) +
		sizeof(sddesc_len) + sizeof(dtxContextInfo_len) +
		dtxContextInfo_len + command_len + plantree_len + sddesc_len +
		sizeof(numsegments) + sizeof(resgroupInfo.len) + resgroupInfo.len +
		sizeof(tempNamespaceId) + sizeof(tempToastNamespaceId);

Three variable-length blobs carry the weight. plantree is the serialised PlannedStmt, produced by serializeNode (cdbdisp_query.c:656) and compressed — the function returns both the wire length and the uncompressed length, and it is the uncompressed one that gp_max_plan_size polices. sddesc is the serialised QueryDispatchDesc (execdesc.h:202), the per-dispatch sidecar the QD cannot put in the plan: the SliceTable itself (§19.3), paramInfo with both PARAM_EXTERN and already-evaluated PARAM_EXEC values, oidAssignments so a CREATE picks the same OIDs on every segment, cursorPositions, parallelCursorName for §19.6, secContext, and the matview fields. dtxContextInfo is the distributed transaction context and snapshot from qdSerializeDtxContextInfo (§12.2, §10.5) — the reason all nine QEs see one consistent database state. The rest is small but load-bearing: three user OIDs so SECURITY DEFINER and SET ROLE behave, the statement start timestamp so now() agrees everywhere, the temp-namespace OIDs, and a resource-group blob (Chapter 21). The command field is the SQL text, deliberately truncated to QUERY_STRING_TRUNCATE_SIZE when a plan is present (:913) — it is only there for pg_stat_activity on the segment.

Query	Dispatched slices	Uncompressed plan size
select count(*) from m192_s	1	0 KB
2-table self-join on a non-distribution key	3	2 KB
3-table self-join	4	3 KB
4-table self-join	5	4 KB
6-table self-join	7	6 KB

Roughly a kilobyte per slice on this schema, and it is the same bytes to every QE — the payload scales with the plan, not with the segment count. Each QE then peels the message apart in PostgresMain at postgres.c:5942 (case 'M': /* MPP dispatched stmt from QD */), sets its snapshot from the DTX context first, and calls exec_mpp_query (postgres.c:6124, defined at :1174), which installs the slice table, works out which slice it is (§18.1’s getGpExecIdentity), and runs the ordinary executor lifecycle. §13.3 owns that protocol; the contribution here is the fan-out view. One buildGpQueryString, one 2 KB buffer, nine PQsendGpQuery_shared calls, and ten independent ExecutorStart/ExecutorRun traversals of the same plan tree — each one entering it at a different height.

19.2.6 When a QE dies

Every segment-side error in this book has carried a parenthesised suffix — (seg0 slice1 10.100.0.3:7102 pid=9934). It is manufactured on the QD, not the segment, by cdbconn_setQEIdentifier, and it is rebuilt whenever a QE is assigned to a different slice:

src/backend/cdb/dispatcher/cdbconn.c:429

	if (segdbDesc->segindex >= 0)
	{
		appendStringInfo(&string, "seg%d", segdbDesc->segindex);
		if (sliceIndex > 0)
			appendStringInfo(&string, " slice%d", sliceIndex);
	}
	else
		appendStringInfo(&string, SEGMENT_IS_ACTIVE_PRIMARY(cdbinfo) ?
						 "entry db" : "mirror entry db");
	appendStringInfo(&string, " %s:%d", cdbinfo->config->hostip, cdbinfo->config->port);
	if (segdbDesc->backendPid != 0)
		appendStringInfo(&string, " pid=%d", segdbDesc->backendPid);

The result is stashed as segdbDesc->whoami and stapled onto every incoming PGresult as a private diagnostics field, PG_DIAG_GP_PROCESS_TAG (cdbdispatchresult.c:345). When cdbdisp_get_PQerror rebuilds a QE’s error as a local ErrorData, it pulls the tag back out and composes errmsg("%s (%s)", fld, whoami) (:598) while preserving the QE’s SQLSTATE, file, line and function. That is why a segment error looks like a local error with a return address. To see it, I terminated a reader QE on seg0 from a utility-mode connection while a long join was running, with statement_timeout set as a guard:

Killing a reader QE mid-statement. Session A ran the join twice with statement_timeout=90s; between them a second connection did PGOPTIONS=’-c gp_role=utility’ psql -p 7102 -c ‘select pg_terminate_backend(10791)’.

-- session A
SET statement_timeout = '90s';
select min(pid) from gp_backend_info() where type='r' and content=0;   --> 10791
select count(*) from m192_t a join m192_t b on a.b = b.b;
-- (meanwhile, on seg 7102 in utility mode: pg_terminate_backend(10791))
select 'SESSION A STILL ALIVE' as post;
select count(*) from m192_s;

   count
-----------
 514285716
(1 row)

ERROR:  terminating connection due to administrator command  (seg0 slice2 10.100.0.3:7102 pid=10791)
         post
-----------------------
 SESSION A STILL ALIVE
(1 row)

 count
-------
   300
(1 row)

The QE’s FATAL arrived at the QD as an ERROR — deliberately, per the comment at cdbdispatchresult.c:579: a QE dying must not kill the QD. The statement, however, is unconditionally dead. There is no way to re-run one slice: that QE held part of a hash table and half a redistribution stream, and its peers’ Motion receivers were already waiting on a route that will never produce another chunk. So the dispatcher escalates — checkDispatchResult promotes waitMode to DISPATCH_WAIT_CANCEL as soon as any QE reports an error and cancelOnError is set (cdbdisp_async.c:502), signalQEs sends libpq cancels to the survivors (:912), and cdbdisp_getDispatchResults rethrows the first collected error. Cleanup then runs through RecycleGang, and here the writer/reader asymmetry pays off one final time: because the victim was a reader, only its descriptor was destroyed, the session’s writer gang and transaction survived, and the next statement ran normally on rebuilt readers. Had the writer died, markCurrentGxactWriterGangLost (cdbutil.c:992) would have forced resetSessionForPrimaryGangLoss — a new gp_session_id, dropped temp tables, and the log line “The previous session was reset because its gang was disconnected” (cdbgang.c:814). What none of this covers is admission: how many gangs a cluster will let you hold at once, and what happens when the answer is “not yet”. That is Chapter 21’s subject. Between here and there, §19.3 takes the payload apart — the slice table that tells each of these nine processes which slice it is.