The Cloudberry Interconnect: UDPIFC, TCP and Proxy

The interconnect is a flow-controlled protocol built on UDP. When people blame the network, this is usually what they mean.

Everything so far in this chapter has been inside the backend. nodeMotion.c decides that a tuple must go to route 2; cdbmotion.c serialises it into TupleChunks and hands them off (§19.4). The handing-off is a single call — and on the other side of that call there is no code at all in src/backend. There is a table of function pointers, filled in at postmaster startup by a shared library that lives in contrib/.

That is the fact worth establishing first, because it reframes the whole subsystem: the Cloudberry interconnect is a pluggable, self-registering transport module, and the set of legal values of gp_interconnect_type is not a compile-time enum but a list of whoever registered. Three implementations register on this cluster; a fourth exists in the tree and does not. This section owns the seam, the setup/teardown lifecycle, all three-plus transports, the UDP flow-control machinery — and, crucially, the exact place where a Motion sender blocks, which is the hazard that §18.1’s squelch and §18.4’s prefetch exist to avoid.

19.5.1 A vtable, not a subsystem

MotionIPCLayer is declared in src/include/cdb/ml_ipc.h:36–:292: two identity fields (ic_type, type_name) and 22 function pointers. The process-wide selection lives in two globals at the top of the Motion layer — MotionIPCLayer *CurrentMotionIPCLayer (src/backend/cdb/motion/cdbmotion.c:33) and the registry static MotionIPCLayer *IPCLayerImpls[MAX_NUMBER_TYPES] (:36). Implementations put themselves in the registry; nothing in the backend knows their names.

src/backend/cdb/motion/cdbmotion.c:1313

/*
 * Called by interconnect.so to register a new IPC layer implement.
 */
void
RegisterIPCLayerImpl(MotionIPCLayer *impl)
{
	...
	IPCLayerImpls[CurrentIPCLayerImplNum++] = impl;
}

The caller is contrib/interconnect/ic_modules.c, which defines three static MotionIPCLayer initialisers — tcp_ipc_layer (:26), proxy_ipc_layer (:65), udpifc_ipc_layer (:105) — and registers all three from _PG_init (:145–:155). A detail that saves a lot of confusion later: proxy_ipc_layer is not a separate transport. Every method slot in it is a …TCP function; ic-proxy is a set of #ifdef ENABLE_IC_PROXY hooks inside ic_tcp.c plus a separate proxy process. Only udpifc_ipc_layer has its own send/receive path.

So how does interconnect.so get loaded, given that the module must be present before the first query? Not through shared_preload_libraries as a DBA would write it. Cloudberry compiles a built-in preload list into the server:

src/include/utils/process_shared_preload_libraries.h:1

#ifdef ENABLE_PRELOAD_IC_MODULE
  "interconnect",
#endif
#ifdef ENABLE_IC_UDP2
  		"udp2",
#endif
#ifdef USE_PAX_STORAGE
		"pax",
#endif

That header is #included as an array initialiser at src/backend/utils/init/miscinit.c:1962, and expand_shared_preload_libraries_string() (:2005) prepends it to the user’s setting before loading. postmaster.c:1275 calls process_shared_preload_libraries() — which runs _PG_init and therefore the three registrations — and postmaster.c:1280 then calls InitializeCurrentMotionIPCLayer() (cdbmotion.c:1347) to resolve gp_interconnect_type against whatever registered. The consequence is visible from SQL: the GUC does not mention the interconnect, but the library is mapped into every backend.

The interconnect is loaded, but not because anyone asked for it. /proc/<pid>/maps is the honest answer.

show shared_preload_libraries;

-- what is actually mapped into this backend?
select pg_backend_pid();
\! grep -oE "/[^ ]*postgresql/[a-z_0-9]*\.so" /proc/PID/maps | sort -u

 shared_preload_libraries 
--------------------------
 pg_stat_statements
(1 row)

/usr/local/cloudberry-db/lib/postgresql/interconnect.so
/usr/local/cloudberry-db/lib/postgresql/pax.so
/usr/local/cloudberry-db/lib/postgresql/pg_stat_statements.so

Now the payoff. gp_interconnect_type is a string GUC (guc_gp.c:5115, context PGC_BACKEND) with a check hook, CheckGpInterconnectTypeStr (cdbmotion.c:1292), whose entire body is a loop over IPCLayerImpls[]. There is no static list of names anywhere. contrib/udp2/ic_udp2.c (1027 lines) exists in this tree and registers a udp2_ipc_layer, but ENABLE_IC_UDP2 is #undef in src/include/pg_config.h:56, so udp2 is never preloaded, never registers — and is therefore not a legal value of the GUC on this cluster:

The enum is dynamic: udp2 is a real implementation in the tree, and an illegal GUC value here purely because it did not register.

-- the three that registered
PGOPTIONS='-c gp_interconnect_type=tcp'    psql -p 7100 -c 'show gp_interconnect_type'
PGOPTIONS='-c gp_interconnect_type=proxy'  psql -p 7100 -c 'show gp_interconnect_type'

-- the one that exists in contrib/ but was not built
PGOPTIONS='-c gp_interconnect_type=udp2'   psql -p 7100 -c 'show gp_interconnect_type'

 gp_interconnect_type 
----------------------
 tcp

 gp_interconnect_type 
----------------------
 proxy

psql: error: connection to server on socket "/tmp/.s.PGSQL.7100" failed:
FATAL:  invalid value for parameter "gp_interconnect_type": "udp2"

Because gp_interconnect_type is PGC_BACKEND, not PGC_POSTMASTER, switching transports needs no restart — PGOPTIONS picks one per connection. Every transport comparison in this section is one psql invocation against the same live cluster. The postmaster-side InitializeCurrentMotionIPCLayer() has a second, nicer error path: if the startup value is unknown it reports Invalid gp_interconnect_type: %s, valid values are: %s (cdbmotion.c:1398), building the list of valid values by walking the registry — the clearest possible statement that the enum is assembled at runtime.

19.5.2 What a transport must provide

The 22 pointers are not all interesting; roughly a dozen carry the contract. The boundary with §19.4 is exact: the Motion layer’s last act is CurrentMotionIPCLayer->SendTupleChunkToAMS, and that symbol is itself provided by the module — contrib/interconnect/ic_common.c:41, shared by all three implementations, which fans a chunk list out to doBroadcast or a single route and then calls the per-transport SendChunk.

Method	What the transport must therefore do
GetMaxTupleChunkSize	Tell the Motion layer how big a chunk may be. Called once per statement at cdbmotion.c:169 to set Gp_max_tuple_chunk_size. The transport dictates the chunk size — see below.
GetListenPort	Publish a port the peers can be told about, so the QD can put it in the slice table (§19.3).
InitMotionLayerIPC / CleanUpMotionLayerIPC	Per-process one-time setup: create the listening socket, start any threads. Called from cdbutil.c:1162 and :3556 at backend init.
SetupInterconnect / TeardownInterconnect	Per-statement: build the per-slice connection state from the slice table, then tear it all down whatever the outcome.
SendTupleChunkToAMS	Shared entry point in ic_common.c. Resolves motion id and route to a MotionConn, handles the broadcast fan-out, and delegates.
SendChunk	The actual per-transport send. This is where the sender can block.
SendEOS / SendStopMessage	End-of-stream downstream; stop upstream. SendStopMessage is the wire form of §18.1’s ExecSquelchNode.
RecvTupleChunkFromAny	Block until any active peer has data, with fairness across routes. The receive side of an unordered Gather or Redistribute.
RecvTupleChunkFrom	Block for one specific route. What a sorted (merge) Gather needs, one route at a time (§19.4).
RecvTupleChunk / DirectPutRxBuffer	Pull the next chunk from one connection, and hand the buffer back when the Motion layer is finished with it.
GetTransportDirectBuffer / PutTransportDirectBuffer	Let the serialiser write straight into the transport’s outgoing packet buffer, avoiding one memcpy. Same implementation for all three here.
DeregisterReadInterest	Drop a route from the poll set once it has sent EOS or been stopped.
IcProxyServiceMain	The main loop of the proxy process. NULL unless the server was built with —enable-ic-proxy; non-NULL in all three vtables when it was.
GetMotionConnTupleRemapper / GetMotionSentRecordTypmod	Per-connection storage for the record-typmod remapping state of §19.4 — the transport owns the lifetime, the Motion layer owns the meaning.

GetMaxTupleChunkSize is the one that deserves a second look, because it is the transport reaching upward and changing the Motion layer’s behaviour. Both implementations compute the same thing — the packet budget minus their own header — and their headers differ by 84 bytes:

contrib/interconnect/udp/ic_udpifc.c:8179 and tcp/ic_tcp.c:3363

int GetMaxTupleChunkSizeUDP(void)
{ return Gp_max_packet_size - sizeof(struct icpkthdr) - TUPLE_CHUNK_HEADER_SIZE; }

int GetMaxTupleChunkSizeTCP(void)
{ return Gp_max_packet_size - PACKET_HEADER_SIZE - TUPLE_CHUNK_HEADER_SIZE; }

sizeof(icpkthdr) is 88 bytes on this platform; PACKET_HEADER_SIZE is 4 (ml_ipc.h:34) and TUPLE_CHUNK_HEADER_SIZE is 4 (tupchunk.h:49). So at the default gp_max_packet_size=8192 the maximum chunk is 8100 over udpifc and 8184 over tcp — a 1% difference that no query will notice. Shrink the packet and it becomes measurable. Three rows of 30 098 serialised bytes each, gathered to the QD, with gp_log_interconnect=verbose reporting the chunk count:

The transport chooses the chunk size, and you can count it. 1956-byte chunks over UDP versus 2040 over TCP for the same three tuples.

create table m195_wide (id int, big text) distributed by (id);
insert into m195_wide select i,
  (select string_agg(md5(random()::text||i||j),'') from generate_series(1,940) j)
  from generate_series(1,3) i;          -- 30 080 incompressible chars each

-- same query, four transport/packet-size combinations
set gp_log_interconnect=verbose; set client_min_messages=log;
select id, big from m195_wide;

##### type=udpifc gp_max_packet_size=8192 #####
LOG:  Interconnect seg-1 slice0 received from slice1: 3 tuples, 90354 total bytes, 90294 tuple bytes, 15 chunks.
##### type=udpifc gp_max_packet_size=2048 #####
LOG:  Interconnect seg-1 slice0 received from slice1: 3 tuples, 90498 total bytes, 90294 tuple bytes, 51 chunks.
##### type=tcp    gp_max_packet_size=8192 #####
LOG:  Interconnect seg-1 slice0 received from slice1: 3 tuples, 90354 total bytes, 90294 tuple bytes, 15 chunks.
##### type=tcp    gp_max_packet_size=2048 #####
LOG:  Interconnect seg-1 slice0 received from slice1: 3 tuples, 90486 total bytes, 90294 tuple bytes, 48 chunks.

The arithmetic closes exactly. At 2048 bytes udpifc allows 2048−88−4 = 1956 bytes per chunk, so each 30 098-byte tuple needs ⌈30098/1956⌉ = 16 chunks: 3×16 = 48, plus one end-of-stream chunk per sender = 51. TCP allows 2048−4−4 = 2040, so ⌈30098/2040⌉ = 15: 45 + 3 = 48. And total bytes − tuple bytes is 4 × chunks in every row, which is TUPLE_CHUNK_HEADER_SIZE measured rather than read. At 8192 both transports land on 4 chunks per tuple and the difference vanishes — which is why nobody notices.

19.5.3 Setup and teardown are per statement

SetupInterconnect is called from three places, all in the executor: execMain.c:536 and execMain.c:751 for the statement itself, and nodeSubplan.c:1197 — an InitPlan gets its own interconnect setup on top of its own gang (§18.6). Teardown is symmetric but paranoid: execUtils.c:2100 for the normal path and execUtils.c:2280 with hasErrors = true for the error path, because sockets and the receive thread must be reclaimed whatever happened. For udpifc the entry points are SetupInterconnectUDP (ic_udpifc.c:3833) and TeardownUDPIFCInterconnect (:4251); for tcp, SetupInterconnectTCP (ic_tcp.c:1915). Setting gp_log_interconnect=debug narrates the whole thing, and is the fastest way to see a slice table turn into sockets:

Setup, narrated. Each QE learns its peers’ host/port from the slice table and dials them; the receiving side just counts arrivals. Abridged — one of the four slices shown.

set gp_log_interconnect=debug;
set client_min_messages=debug1;
select count(*) from (select d.label, count(*) from m195_fact f
                      join m195_dim d on f.k=d.k group by d.label) s;

DEBUG:  SetupUDPIFCInterconnect: queue depth, queue_depth=4, snd_queue_depth=2, mem_size=10, slices=4, packet_size=8192
DEBUG:  Setup recving connections: my slice 0, childId 1
DEBUG:  SetupUDPInterconnect will activate 3 incoming, 3 expect incoming, 0 outgoing, 0 expect outgoing routes for ic_instancce_id 1. Listening on ports=0/33254 sockfd=6.
...
DEBUG:  Interconnect seg0 slice2 setting up sending motion node  (seg0 slice2 pid=6959)
DEBUG:  Interconnect connecting to seg0 slice1 10.100.0.3:53109 pid=6953 sockfd=-1  (seg0 slice2 pid=6959)
DEBUG:  setupOutgoingUDPConnection: node 2 route 0 srccontent 0 dstcontent 0: 10.100.0.3:53109  (seg0 slice2 pid=6959)
DEBUG:  Interconnect connecting to seg1 slice1 10.100.0.3:43894 pid=6954 sockfd=-1  (seg0 slice2 pid=6959)
DEBUG:  Interconnect connecting to seg2 slice1 10.100.0.3:32981 pid=6955 sockfd=-1  (seg0 slice2 pid=6959)
DEBUG:  SetupUDPInterconnect will activate 3 incoming, 3 expect incoming, 3 outgoing, 3 expect outgoing routes for ic_instancce_id 1. Listening on ports=0/51184 sockfd=6.

Note the contrast with §19.2, which established that gangs persist across statements in a session. The interconnect does not. Each statement gets a fresh ic_instance_id, allocated by the QD and carried in the slice table, and every packet header carries it so that a straggler from the previous statement can be recognised and discarded. Three identical queries in one session, with gp_interconnect_log_stats=on, show the same three QE pids and three different instances:

Same processes, three interconnects. The gang is a session resource; the interconnect is a statement resource.

set gp_interconnect_log_stats=on;
set client_min_messages=log;
select count(*) from m195_fact;   -- x3, same session

ic_instance_id 1 ic_id_last_teardown 1 ... (seg0 slice1 10.100.0.3:7102 pid=24530)
ic_instance_id 1 ic_id_last_teardown 1 ... (seg1 slice1 10.100.0.3:7103 pid=24531)
ic_instance_id 1 ic_id_last_teardown 1 ... (seg2 slice1 10.100.0.3:7104 pid=24532)
ic_instance_id 2 ic_id_last_teardown 2 ... (seg0 slice1 10.100.0.3:7102 pid=24530)
ic_instance_id 2 ic_id_last_teardown 2 ... (seg1 slice1 10.100.0.3:7103 pid=24531)
ic_instance_id 2 ic_id_last_teardown 2 ... (seg2 slice1 10.100.0.3:7104 pid=24532)
ic_instance_id 3 ic_id_last_teardown 3 ... (seg0 slice1 10.100.0.3:7102 pid=24530)
ic_instance_id 3 ic_id_last_teardown 3 ... (seg1 slice1 10.100.0.3:7103 pid=24531)
ic_instance_id 3 ic_id_last_teardown 3 ... (seg2 slice1 10.100.0.3:7104 pid=24532)

Two documentation traps worth flagging. (1) gp_interconnect_setup_timeout is 7200 seconds here, and it is honoured only by ic_tcp.c (:1572, :1888) and ic_proxy_backend.c (:310); grep ic_udpifc.c and you will not find it — udpifc bounds setup through its ordinary retransmit/deadlock checks instead. (2) The SetupUDPIFCInterconnect auto-tuning block (:3760) recomputes the queue depths as Gp_interconnect_mem_size / (Gp_max_packet_size * sliceNum), but gp_interconnect_mem_size is documented and set in megabytes (10) while Gp_max_packet_size is in bytes — so the quotient is 0 and the depths stay at their defaults, exactly as the DEBUG1 line above reports. On this cluster the auto-tuner is a no-op.

19.5.4 UDPIFC: the packet

contrib/interconnect/udp/ic_udpifc.c is 8233 lines — more than twice the TCP implementation and the proxy put together — and essentially all of the excess is reliability: sequence numbers, acknowledgements, a retransmit time-wheel, a congestion window, out-of-order caching, and a background receive thread. It is a small transport protocol built on unreliable datagrams, and the place to start is its header, icpkthdr (contrib/interconnect/udp/ic_udpifc.h:32–:96).

byte

+0

+4

+8

+12

0

motNodeId

srcPid

srcListenerPort

dstPid

16

dstListenerPort

sessionId

icId

recvSliceIndex

32

sendSliceIndex

srcContentId

dstContentId

crc

48

⟦flags⟧

len

⟦seq⟧

⟦extraSeq⟧

64

send_time — uint64

recv_time — uint64

80

retry_times

7 bytes tail padding — alignment

↑———————— sizeof(icpkthdr) = 88 ————————↑

⟦88⟧ header + ⟦4⟧ TupleChunk header + payload ≤ gp_max_packet_size ⟦8192⟧ ⇒ Gp_max_tuple_chunk_size = ⟦8100⟧ — one datagram, one send, no IP fragmentation

lavender = routing identity · amber = protocol state · green = integrity and timing

The first eleven fields are pure identity, and they are why udpifc needs no connection: a datagram arriving on a QE’s single listening socket carries enough information (motNodeId, source and destination pid and port, sessionId, icId, both slice indices, both content ids) to be matched to exactly one logical connection in a hash table. crc was added because UDP’s own 16-bit checksum is too weak for the volumes involved. flags is a bitmask (ic_udpifc.c:211–:219): RECEIVER_TO_SENDER, ACK, STOP, EOS, NAK, DISORDER, DUPLICATE, CAPACITY, FULL — so a single 88-byte header, with no payload, is also the ack, the negative ack, the out-of-order report, the flow-control credit and the stop message of §18.1. The seq/extraSeq pair is overloaded by flag combination, and the header comment enumerates six meanings; the two that matter for flow control are, in an ACK|CAPACITY, seq = highest contiguously cached sequence and extraSeq = highest consumed sequence. The receiver is therefore reporting two different watermarks in one datagram: what it has, and what the executor above it has actually eaten.

19.5.5 UDPIFC flow control, and where a Motion sender blocks

Two GUCs set the sizes. gp_interconnect_queue_depth (4) is the receiver’s per-connection packet ring: conn->pkt_q is allocated with exactly that many slots (ic_udpifc.c:7770–:7783), and a packet is filed at (pkt->seq - 1) % pkt_q_capacity (:7211) — which is what gp_interconnect_cache_future_packets=on buys, since an out-of-order arrival goes straight into its slot instead of being dropped. gp_interconnect_snd_queue_depth (2) is the sender’s per-connection allowance of packet buffers, and the sender’s buffers are pooled: snd_buffer_pool.maxCount += Gp_interconnect_snd_queue_depth once per outgoing connection at setup (:3149). A sender to R receivers therefore owns 2R buffers in total, not 2 per peer.

The credit is conn->capacity, initialised to Gp_interconnect_queue_depth (:3146) — the sender’s belief about free slots in that receiver’s ring — decremented on each send and restored by an ACK|CAPACITY. On top of that sits a genuine TCP-style congestion window: snd_control_info holds cwnd, minCwnd and ssthresh (:428–:443), cwnd starts at one per connection with minCwnd pinned to that starting value (:3149, :3700), grows on un-retried acks and is halved on loss. sendBuffers is where all of it meets:

contrib/interconnect/udp/ic_udpifc.c:5584

while (conn->capacity > 0 && icBufferListLength(&conn->sndQueue) > 0)
{
	/* loss-based FC: respect the shared congestion window */
	if ((Gp_interconnect_fc_method == INTERCONNECT_FC_METHOD_LOSS || ...) &&
		(icBufferListLength(&conn->unackQueue) > 0 &&
		 unack_queue_ring.numSharedOutStanding >= (snd_control_info.cwnd - snd_control_info.minCwnd)))
		break;
	...
	conn->capacity--;
	icBufferListAppend(&conn->unackQueue, buf);

Read the guard carefully, because it encodes a deliberate fairness rule stated in the comment above the function: every connection is always allowed one outstanding buffer for free (numSharedOutStanding is only incremented when a connection’s unack queue already has more than one entry, :5613). Beyond that first packet, all connections compete for cwnd − minCwnd shared slots. So a sender with R receivers has R free slots plus a shared window that grows to R, capped at the 2R buffers that exist — and a sender with one receiver, such as a Gather Motion QE, is limited to two packets in flight no matter what.

And here is the block itself. SendChunkUDPIFC (ic_udpifc.c:6502) aggregates chunks into the current packet buffer and, when the buffer is full, must obtain a new one. If the pool is empty — every buffer sitting in some connection’s unack queue — there is nowhere to put the next chunk:

contrib/interconnect/udp/ic_udpifc.c:6562

while (doCheckExpiration || (conn->curBuff = getSndBuffer(&conn->mConn)) == NULL)
{
	int timeout = (doCheckExpiration ? 0 : computeTimeout(&conn->mConn, retry));

	if (pollAcks(transportStates, pEntry->txfd, timeout))
		if (handleAcks(transportStates, &pEntry->entry, true))
			gotStops = true;
	checkExceptions(transportStates, &pEntry->entry, &conn->mConn, retry++, timeout);
	...
}

That loop is the mechanism the rest of the book has been referring to. A Motion sender that cannot get a buffer sits in pollAcks, and it cannot get a buffer until a receiver acknowledges — which a receiver only does once its executor consumes from the ring. A slow or stalled consumer therefore stops a producer several processes away, through nothing more than 4 ring slots and 2R packet buffers. §18.1’s ExecSquelchNode exists so a receiver that will read no more (a LIMIT, a hash join whose build side found nothing) sends SendStopMessage (:6756) instead of simply going quiet; §18.4’s prefetch_inner exists so a join never puts itself in the position of blocking one of its own inputs. Both are avoidance strategies for this loop. Note also what the loop keeps doing while it waits: checkExceptions (:6405) folds in expiration checks, a deadlock check every fourth retry, and a PostmasterIsAlive() probe every 64th — the sender is the process that notices the cluster has broken.

The window, measured. gp_interconnect_log_stats dumps the whole controller at teardown; abridged to one sender per slice.

set gp_interconnect_log_stats=on;
set client_min_messages=log;
select count(*) from m195_fact f1 join m195_fact f2 on f1.id = f2.k;
-- Redistribute Motion 3:3 (slice2, 3 receivers) under Gather Motion 3:1 (slice1, 1 receiver)

LOG:  Interconnect State: isSender 1 isReceiver 0 snd_queue_depth 2 recv_queue_depth 4 Gp_max_packet_size 8192
      UNACK_QUEUE_RING_SLOTS_NUM 2000 TIMER_SPAN 5000 DEFAULT_RTT 20000 ...
      snd_pkt_count 447 retransmits 0 crc_errors 0 recv_pkt_count 0 recv_ack_num 668
      recv_queue_size_avg NaN capacity_avg 2.421171 freebuf_avg 1.218289
      mismatch_pkt_num 0 disordered_pkt_num 0 duplicated_pkt_num 0
      rtt/dev [338/200, 534.666667/261.000000, 705/311]  cwnd 6.000000   (seg0 slice2 pid=8005)

LOG:  Interconnect State: isSender 1 isReceiver 1 ... snd_pkt_count 1 recv_pkt_count 451 recv_ack_num 1
      capacity_avg NaN freebuf_avg 2.000000 rtt/dev [17518/4343, ...]  cwnd 2.000000   (seg0 slice1 pid=7999)

Every number in that dump is predicted by the source. The slice2 senders each have R = 3 outgoing connections, so snd_buffer_pool.maxCount = 2 × 3 = 6, ssthresh is set to that at setup, and cwnd duly grows to and stops at 6.000000. The slice1 senders have R = 1, maxCount = 2, and cwnd stops at 2.000000. capacity_avg 2.42 is the average remaining receiver credit at the moment a packet was sent — out of 4 — and freebuf_avg 1.22 is the average number of free buffers in a pool of 6. Both are low: this sender spent most of the query one or two packets away from the wait loop, on a loopback interface with zero loss. rtt is in microseconds (338 µs minimum, 705 µs maximum on slice2; ~17.5 ms on the slice1→QD link, which is the QD’s own consumption latency showing up as RTT), against a DEFAULT_RTT of 20 000 µs — the estimator (:5033–:5045, an exponential filter with shift coefficients 1/8 for RTT and 1/4 for deviation, clamped to MIN_RTT 100 µs and MAX_RTT 200 ms) has plenty of room.

Retransmission is driven by a time wheel rather than per-packet timers: UnackQueueRing (:610–:650) has UNACK_QUEUE_RING_SLOTS_NUM = 2000 slots each covering gp_interconnect_timer_period = 5 ms, so a buffer is filed in the slot corresponding to its computed expiration and checkExpiration (:5981) walks the wheel from the last checked position, resending everything it passes — which caps how much can be resent in one burst. The expiration period is bounded by gp_interconnect_min_rto = 20 ms below and 1 s above. Two escalating failure reports come out of checkNetworkTimeout (:5921): after gp_interconnect_min_retries_before_timeout = 100 retries a once-per-statement WARNING (“interconnect may encountered a network error”), and only after 100 retries and gp_interconnect_transmit_timeout = 3600 seconds an ERROR. gp_interconnect_debug_retry_interval = 10 makes every tenth retry log at gp_log_interconnect=debug. The whole receive side, meanwhile, runs in a dedicated pthread — rxThreadFunc (:7301), created once per backend at :1908 — which polls the single listening socket, files packets into rings, sends acks, and famously must not call elog (write_log instead), because elog is not thread-safe.

gp_interconnect_fc_method	What changes
loss (default)	Shared buffer pool; AIMD congestion window; retransmit from the time wheel; timeout is TIMER_CHECKING_PERIOD (20 ms) once anything is unacked. cwnd += 1 per un-retried ack below ssthresh.
capacity	Per-connection buffer accounting only (getSndBuffer at :3043 refuses when unackQueue + sndQueue reaches snd_queue_depth), no shared window, and its own expiration path checkExpirationCapacityFC (:6370) with an exponential per-retry timeout. The older, simpler scheme.
loss_advance	Loss-based, but polls for acks with a zero timeout when the window is open (:6275, :6473), and answers a full receive ring with an explicit UDPIC_FLAGS_FULL message (:7199). cwnd += 2 in slow start. Aims at lower latency.
loss_timer	Loss-based plus an explicit per-connection RTO list (addtoRTOList at :5011) and a path that wakes the blocked main thread when a packet is dropped at a full ring (:7178). Aims at faster recovery under skew.

The four names are not in any header the reader would find first — they are only in guc_gp.c:562–:566 — but the server will list them: an illegal value produces HINT: Available values: loss, capacity, loss_advance, loss_timer. The three loss variants share the window machinery of the figure above and differ in how aggressively they poll and how they react to a full receiver; capacity is a different algorithm and is the one to reach for when diagnosing whether the congestion window is the problem.

19.5.6 ic-tcp: correctness by delegation, connections by multiplication

contrib/interconnect/tcp/ic_tcp.c (3378 lines) is what you would write if the kernel were allowed to do the hard part: one stream socket per sender–receiver pair, a 4-byte length prefix per packet, and a RegisterMessage sent on each new connection (:788) so the accepting side can match the socket to a route — the same identity problem icpkthdr solves per datagram, solved once per connection. There is no sequence number, no ack, no congestion window; the sender blocks in write() and the kernel handles the rest. gp_interconnect_tcp_listener_backlog = 256 sizes the listen() queue, and SetupInterconnectTCP warns outright when a slice’s expected fan-in exceeds it (:1420, hinting at both the GUC and net.core.somaxconn).

The cost is the connection count, and it is directly observable. Hold a Gather Motion 3:1 open with a cursor and look at the QD’s sockets under each transport:

Same query, same cluster, two transports. The Gather Motion’s three senders cost the QD three TCP connections — or none.

BEGIN;
DECLARE m195_cur CURSOR FOR SELECT id FROM m195_fact ORDER BY id;
FETCH 1 FROM m195_cur;      -- interconnect now set up and idle
\! ss -tanp / ss -uanp for the QD pid, excluding the libpq dispatch sockets

########## gp_interconnect_type = udpifc ##########
QD pid=11006 (session 209)
-- QD TCP sockets NOT to a segment libpq port = interconnect:
        (none)
-- QD TCP libpq dispatch sockets (to 7102/3/4): 6
-- QD UDP sockets:
    10.100.0.3:41788 0.0.0.0:*
    10.100.0.3:43355 0.0.0.0:*        <-- exactly 2, for the whole process

########## gp_interconnect_type = tcp ##########
QD pid=11046 (session 211)
-- QD TCP sockets NOT to a segment libpq port = interconnect:
     10.100.0.3:48749 -> 10.100.0.3:37558
     10.100.0.3:48749 -> 10.100.0.3:37556
     10.100.0.3:48749 -> 10.100.0.3:37554     <-- 3, one per sending QE
-- QD TCP libpq dispatch sockets (to 7102/3/4): 6
-- QD UDP sockets:  (none)

Three connections is nothing. Now do the arithmetic that matters. A Redistribute Motion between two slices of S and R processes needs S×R sockets — 9 here, but on 100 segments a single 100:100 redistribute is 10 000 connections, each with its own kernel send and receive buffers, and a plan with four such motions multiplies that again. udpifc’s socket count does not move: two UDP sockets per process (one listener, one sender), regardless of peers, regardless of plan. That, not throughput, is why UDPIFC is the default — and it is also why a large cluster eventually wants ic-proxy, which collapses the count again along a different axis.

Wall clock for a real 3:3 redistribute join, alternating transports on one host. Three runs each. This is an assert-enabled build on a loopback interface: read the ratio, never the numbers.

-- select count(*) from m195_fact f1 join m195_fact f2 on f1.id = f2.k;  -> 599400
-- alternating PGOPTIONS='-c gp_interconnect_type=<t>', \timing on

udpifc  Time: 517.377 ms
tcp     Time: 576.854 ms
udpifc  Time: 440.263 ms
tcp     Time: 523.495 ms
udpifc  Time: 429.150 ms
tcp     Time: 500.028 ms

udpifc is consistently ahead, by roughly 1.15×, with the gap stable across runs — but on a single host with three segments and a loopback interface this measures almost nothing about either protocol. The honest statement is the negative one: on a cluster this small the transport choice is not a performance decision, and anyone who wants to see the difference the interconnect makes needs the connection count, not the clock.

19.5.7 ic-proxy: fewer connections by adding a hop

ic-proxy attacks the S×R problem from the other side. One proxy process per host multiplexes every QE-to-QE stream onto a single connection per host pair, so the physical connection count scales with hosts rather than with QEs, while the logical connection count stays exactly what the plan needs. Its own README.ic-proxy.md is unusually candid about the design and states the motivation plainly: “it requires only one network connection between every two segments… and has better performance than ic-udp mode in high-latency network.”

Mechanically the proxy sits below ic-tcp, which is why proxy_ipc_layer reuses every TCP method: SetupInterconnectTCP takes an #ifdef ENABLE_IC_PROXY branch (ic_tcp.c:1414) that hands the connection set to ic_proxy_backend_run_loop, which binds domain-socket fds to libuv pipes and then lets the ordinary ic_tcp send/receive code drive them. The proxy process itself is a background worker (src/backend/cdb/motion/ic_proxy_bgworker.c, 42 lines: ICProxyMain simply calls CurrentMotionIPCLayer->IcProxyServiceMain()), and the vtable slot exists in all three layers when built with --enable-ic-proxy, NULL otherwise. Its flow control is a third design again — neither TCP’s backpressure nor udpifc’s packet loss will do, because many logical connections share one physical one and a dropped packet cannot be dropped after it has arrived — so the README describes explicit unackSendPkt/unackRecvPkt counters with pause and resume thresholds, and a rule that the sender may only pause on packet boundaries.

None of which can be exercised here, and it is worth being precise about why. This server was built with --enable-ic-proxy (ENABLE_IC_PROXY is defined in src/include/pg_config.h:53), so proxy is a legal GUC value and the connection opens. But there is no ic proxy process on this cluster, and gp_interconnect_proxy_addresses is empty. A motion-bearing query therefore does not fail — it waits, forever, for a proxy that will never answer:

proxy is accepted, then blocks in setup. A statement_timeout is the polite way to find out.

PGOPTIONS='-c gp_interconnect_type=proxy' psql -p 7100 -d postgres
  -c "show gp_interconnect_type;"
  -c "set statement_timeout='15s'; select count(*) from m195_fact;"

 gp_interconnect_type 
----------------------
 proxy
(1 row)

SET
ERROR:  canceling statement due to statement timeout

There is a second, more interesting reason the proxy is absent here, visible in the coordinator log at every postmaster start: background worker “ic proxy process”: background workers without shared memory access are not supported (bgworker.c:706). The entry in PMAuxProcList at postmaster.c:450 declares bgw_flags = 0, and PostgreSQL 16 removed support for shmem-less background workers — so on this PG16-based tree the proxy worker is rejected at registration, before gp_interconnect_proxy_addresses is ever consulted. Configuring the addresses would not have helped. This section could not exercise ic-proxy on the cluster, and everything above is from the source and the README — except for one measurement, which comes from somewhere else entirely.

19.5.8 The transport has a test harness of its own

Splitting the interconnect out of the backend bought something the README lists first among the benefits: “It is more convenient to test & debug & benchmark.” contrib/interconnect/test/ is a CMake project that compiles the real ic_common.c, ic_tcp.c, ic_udpifc.c and all twelve proxy files against a mocked backend — elog_mock.c (1024 lines) stubs out the error subsystem, ic_test_env.c (411) fakes an EState and a slice table, ic_interface_test.c (345) drives the vtable through cmockery, and ic_bench.c (704) forks a client and a server and pushes tuples through a chosen MotionIPCLayer for a fixed interval. A transport you can benchmark without a database is a rare thing to find in a database.

It does not build against this tree unmodified: four errors, all in elog_mock.c, all the same kind of drift — PG16’s utils/elog.h now declares write_csvlog, write_pipe_chunks and error_severity as extern where the mock declares them static, and errdetail now returns int where the mock returns void. Every interconnect and proxy source file compiles clean. Patching those four declarations in an out-of-tree copy was enough to get ic_bench running:

ic_bench, off-cluster, single client + single server, 10 s, default MTU 1500 and 200-byte buffers. Including the proxy transport this cluster cannot run.

# out-of-tree build of contrib/interconnect/test with the four elog_mock.c
# declarations reconciled with PG16's elog.h
./ic_bench -t 0 -i 10   # tcp
./ic_bench -t 1 -i 10   # udpifc
./ic_bench -t 2 -i 10   # proxy

##### -t 0  tcp #####            ##### -t 1  udpifc #####         ##### -t 2  proxy #####
| Total time(s) |  10.000 |      | Total time(s) |  10.178 |      | Total time(s) |  10.000 |
| Loop times    | 2695493 |      | Loop times    |   37216 |      | Loop times    |  927160 |
| LPS(l/ms)     | 269.548 |      | LPS(l/ms)     |   3.657 |      | LPS(l/ms)     |  92.718 |
| Recv mbs      |    3670 |      | Recv mbs      |      43 |      | Recv mbs      |    1262 |
| TPS(mb/s)     | 367.084 |      | TPS(mb/s)     |   4.268 |      | TPS(mb/s)     | 126.267 |
| Recv counts   |18868451 |      | Recv counts   |  223296 |      | Recv counts   | 6490120 |
| Items ops/ms  |1886.838 |      | Items ops/ms  |  21.939 |      | Items ops/ms  | 649.024 |

An 86× gap in favour of TCP looks alarming until you notice that the benchmark is exactly the configuration in which udpifc’s machinery cannot help, and §19.5.5 already predicted the number. One client and one server means one logical connection, so R = 1: snd_buffer_pool.maxCount = 2, cwnd reaches 2, and cwnd − minCwnd = 1 — the sender may have at most two 1408-byte datagrams in flight and must then wait for an ack, per packet, plus a CRC over each one. Loss-based flow control has degenerated to near stop-and-wait. The harness’s own README reports the same shape from different hardware (tcp 654, proxy 156, udpifc 5.0 MB/s) and appends the right caveat: “Lower TPS does not mean the protocol is slower, might means that the cpu time taken by the protocol is low… it satisfies the highest tps required by cbdb, at the same time it occupies a lower cpu than other types of interconnect.” udpifc is not optimised for one stream; it is optimised for a hundred thousand of them not exhausting the machine’s file descriptors. Add the --enable-cassert build and these are ratios between transports under one synthetic load, nothing more.

What the vtable bought, then, is on the table in front of us. A transport that can be selected per connection without a restart; a transport that exists in the tree, registers itself, and is simply absent when not compiled in (udp2); a transport whose background worker is broken by a PostgreSQL upgrade without breaking the two that work (proxy); and a transport that can be compiled, linked and benchmarked with no cluster at all. And running the other way through the same seam, the one upward-facing method in the whole interface: GetMaxTupleChunkSize, by which the wire tells the executor how big a piece of a tuple is allowed to be — 8100 bytes today, because an 88-byte header has to fit in front of it. The last piece of the interconnect is the one case where the wire is exposed to the client rather than to another executor, and that is §19.6.

Cleanup. All transport switching in this section was per-connection PGOPTIONS or session-level SET; no persistent GUC was changed and the cluster is as it was found.

drop table if exists m195_fact;
drop table if exists m195_dim;
drop table if exists m195_wide;
select relname from pg_class where relname like 'm195%';
show gp_interconnect_type;

 relname 
---------
(0 rows)

 gp_interconnect_type 
----------------------
 udpifc
(1 row)