PCIe — Transaction Layer & Data Flow

Introduction

PCI Express (PCIe) is a high‑performance, packet‑based, point‑to‑point interconnect widely used in CPUs, GPUs, accelerators, and storage devices.
At the heart of PCIe is the Transaction Layer, which defines how requests and completions are packetized, sequenced, routed, and flow‑controlled over the link.

This page focuses on the Transaction Layer and data flow, showing how PCIe builds, transports, and processes Transaction Layer Packets (TLPs) on top of the Data Link and Physical layers.

PCIe Layering Overview

PCIe is organized into three main protocol layers:

Transaction Layer
- request/completion semantics
- packetization of reads, writes, messages
- ordering rules and attributes
Data Link Layer
- sequence numbers
- ACK/NAK protocol
- LCRC (Link CRC) and retry
Physical Layer

8b/10b (Gen1/2), 128b/130b (Gen3+)
SERDES, equalization, CDR
lane bonding and training

The Transaction Layer operates entirely in the packet domain, independent of the physical signaling details.

Transaction Layer Packets (TLPs)

TLP Types

Common TLP types include:

Memory Read / Memory Write
I/O Read / I/O Write (legacy)
Configuration Read / Write
Messages (MSI, MSI‑X, interrupts, events)
Completions (for reads and some requests)

Each type has a specific header format and semantics.

TLP Structure

A TLP typically consists of:

Header (3 or 4 DW)
Optional Extended Header (for some features)
Payload (optional, for writes/messages)
ECRC (optional end‑to‑end CRC)

The header encodes addressing, attributes, length, and routing information.

Transaction Layer Responsibilities

The Transaction Layer is responsible for:

Packetization of requests and completions
Address translation and routing (ID‑based, not physical topology)
Ordering rules (posted, non‑posted, completions)
Flow control using credits
Segmentation and reassembly of large transfers
End‑to‑end integrity (optional ECRC)

It defines the logical behavior of PCIe as a load/store interconnect.

Data Flow: Request and Completion

Memory Write (Posted)

The requester builds a Memory Write TLP with address, attributes, and payload.
The TLP is sent downstream without requiring a completion.
The completer writes data to its local memory or registers.

Posted writes do not generate completions, reducing latency and overhead.

Memory Read (Non‑Posted)

The requester builds a Memory Read TLP with address and length.
The TLP is sent downstream to the target.
The completer returns one or more Completion TLPs with data.
The requester reassembles the data and completes the transaction.

Non‑posted requests require reliable completion handling and ordering.

Flow Control in PCIe

Credit‑Based Flow Control

PCIe uses strict credit‑based flow control at the Transaction Layer:

The receiver advertises credits for:
- header space
- data payload space
The transmitter may send TLPs only if it has sufficient credits.
Credits are returned as TLPs are consumed and buffers are freed.

This guarantees lossless operation without buffer overflow.

Virtual Channels (VCs)

PCIe supports multiple Virtual Channels:

independent credit pools
separate traffic classes
deadlock avoidance and QoS
isolation between traffic types

The scheduler and flow control logic operate per VC.

Ordering and Attributes

Ordering Rules

PCIe defines ordering rules for:

posted vs. non‑posted requests
completions relative to requests
relaxed ordering and no‑snoop attributes

These rules ensure correctness while allowing optimizations.

Attributes

TLP headers include:

traffic class (TC)
attributes (RO, NS)
ID‑based routing fields
length and byte enables

These fields drive routing, QoS, and caching behavior.

Interaction with Lower Layers

Data Link Layer

The Data Link Layer:

adds sequence numbers
appends LCRC
manages ACK/NAK and replay

It ensures reliable delivery of TLPs over the physical link.

Physical Layer

The Physical Layer:

serializes and transmits data
performs equalization and training
handles lane bonding and alignment

It is unaware of TLP semantics.

Performance Considerations

Segmentation: large transfers are split into multiple TLPs.
Reassembly: completions are reassembled at the requester.
Scheduling: TLPs are arbitrated across VCs and traffic classes.
Latency: dominated by link traversal, buffering, and completion return.

Efficient Transaction Layer design is critical for PCIe performance.

Summary

The PCIe Transaction Layer defines how requests and completions are packetized, sequenced, and flow‑controlled over a reliable, credit‑based link.
Understanding TLP structure, data flow, and flow control is essential for designing PCIe endpoints, root complexes, and bridges that achieve high performance and strict protocol compliance.

DexterLab