PCIe Data link layer

The PCIe Gen 5 DLL serves as the intermediary between the Transaction Layer (TL) and Physical Layer (PL), ensuring reliable data transfer across the link. Its primary tasks include:

1. Link Management State Machine (DLCMSM) Supervision

• Role: Manages flow control initialization, tracks link status, and coordinates activity changes (e.g., power states, retraining).
• Details: Conveys link status (e.g., active, down) to TL and PL. Triggers actions like link retraining on failure.

2. Data Exchange

• Transmit Path: Receives TLPs from TL, adds a 12-bit Sequence Number and 32-bit Link CRC (LCRC), then passes them to PL for encoding and transmission.
• Receive Path: Validates incoming TLPs (LCRC, Sequence Number), ensuring error-free delivery to TL.

3. Error Detection

• Mechanism: Uses LCRC to verify TLP integrity.
• Actions: Error-free TLPs pass to TL. Errors trigger NAK, link retraining, or retry mechanisms.

4. Flow Control

• Function: Exchanges buffer credit information (e.g., PH, PD) via DLLPs between transmitter and receiver.
• Purpose: Prevents buffer overflow by informing TL of available space, gating TLP generation.

Data Link Layer Packets (DLLPs)

DLLPs are 8-byte packets generated by the DLL for link management, distinct from routed TLPs.

Structure

• Size: 8 bytes.
• Fields: 2 bytes framing (start/end symbols, handled by PL). 2 bytes LCRC (16-bit CRC for DLLP integrity). 4 bytes payload (type-specific data, e.g., Sequence Number for ACK/NAK). 

Characteristics

• Non-Routed: Used for nearest-neighbor communication only (e.g., between adjacent ports).
• No Payload: Unlike TLPs, DLLPs carry control info, not data.
• Types: ACK, NAK, InitFC (1/2), UpdateFC, Power Management, Vendor-Specific.
• Error Handling: Discarded if corrupted (LCRC mismatch), with periodic updates mitigating loss.

1. ACK/NAK: ACK: Confirms successful TLP receipt (e.g., Sequence Number 5 acknowledges all prior TLPs). NAK: Indicates TLP error (e.g., LCRC mismatch), triggers retransmission.
2. Flow Control: InitFC1/InitFC2: Initialize credits for P, NP, Cpl categories post-reset. UpdateFC: Updates credits during runtime.
3. Power Management: Controls device power states.

Behavior

• Sent periodically, updating prior info (e.g., ACK for Sequence 5 implies all prior TLPs acknowledged).
• No acknowledgment protocol for DLLPs; recovery relies on timeouts.

TLP & DLLP Processing in Data Link Layer

The DLL processes two packet types:

TLP Processing

• Input: TLPs from TL (header, payload, optional ECRC).
• Modification: Adds 12-bit Sequence Number (header) and 32-bit LCRC (footer).
• Output: Framed TLPs to PL for transmission.
• Purpose: Ensures integrity and order across routed paths.

DLLP Processing

• Creation: Generated by DLL (e.g., ACK/NAK based on TLP receipt).
• Structure: 4-byte payload + 2-byte LCRC + 2-byte framing.
• Purpose: Local link control (non-routed).

Structure of TLP (Output of DLL)

• Format: Sequence Number: 12 bits (0-4095), tracks TLP order. TLP Body: From TL (header, payload, ECRC if enabled). LCRC: 32 bits, calculated over TLP (excluding framing symbols).

DLCMSM States

The Data Link Control and Management State Machine (DLCMSM) configures and monitors the link:

1. DL_Inactive: Condition: Link non-operational (e.g., disconnected or PL establishing link). Output: DL_Down (link paused).
2. DL_Feature (Optional): Condition: Link operational at 16+ GT/s (e.g., Gen 5). Action: Exchanges feature info (e.g., buffer size) between ports.
3. DL_Init: Sub-States: InitFC1: Sends InitFC1 DLLPs for VC0 credits (PH, PD, NPH, NPD, CplH, CplD). InitFC2: Confirms credit exchange completion. Output: DL_Down (InitFC1), DL_Up (InitFC2). TLPs Blocked: Until initialization completes.
4. DL_Active: Condition: Normal operation, TLP transmission enabled. Output: DL_Up (link active).

Flow Control in DLL

Mechanism

• Credit-Based: Transmitter checks receiver buffer space via DLLPs.
• Credit Types: PH, PD, NPH, NPD, CplH, CplD (per VC).
• Formula: (Credit Limit - Cumulative Credit Required) mod 2^[Field Size] ≤ 2^[Field Size]/2 Field Size: 8 (headers), 12 (data). Credit Limit (CL): Total credits advertised.
• Cumulative Credit Required: Consumed + pending credits.

Example

• Buffer: 1 KB (51 credits, 20 bytes each, CL = 33h).
• Case 1: Consumed = 20h, Pending = 01h: (33h - 21h) mod 256 ≤ 128 → True, TLP sent.
• Case 2: Consumed = 33h, Pending = 01h (buffer full): (33h - 34h) mod 256 ≤ 128 → False (255 > 128), TLP halted.
• Case 3: UpdateFC adds 2 credits (CL = 35h): (35h - 34h) mod 256 ≤ 128 → True, TLP resumes.

Scaled Flow Control

• Need: Higher speeds (32 GT/s) require larger buffers beyond 127 headers/2047 data credits.
• Solution: Scales credits (e.g., ×4 or ×16) using a scale factor in DLLPs.
• Example: 3-bit field (7 max) scaled by 4 = 28 (11100b).

Retry Mechanism in DLL

Purpose

• Ensures data integrity by retransmitting corrupted/lost TLPs.

Triggers

1. NAK: Receiver detects error (e.g., LCRC mismatch).
2. REPLAY_TIMER Expiry: No ACK/NAK within timeout.

Components

• Retry Buffer: Stores TLPs until acknowledged.
• REPLAY_TIMER: Max wait time for ACK/NAK, resets on ACK/NAK or TLP send.
• ACKNAK_LATENCY_TIMER: Delays ACK to batch TLPs, resets on ACK/NAK send.
• REPLAY_NUM: 2-bit counter (0-3), triggers retraining on rollover (11→00).
• NEXT_RCV_SEQ: Expected Sequence Number at receiver.
• ACKD_SEQ: Last acknowledged Sequence Number.
• NAK_SCHEDULER: Flag for pending NAK, blocks further ACK/NAK.
• ACKNAK_SEQ_NUM: Last correctly received TLP Sequence Number. 

Examples

Normal Operation (ACK Case)

1.  Device A: Sends TLPs (e.g., Sequence Numbers 6-10).

2.  Device B:

• Receives TLP 6 successfully, schedules ACK (ACKNAK_LATENCY_TIMER starts).
• Receives TLPs 7-8 before timer expires, updates expected Sequence Number.
• Timer expires → Sends ACK with Sequence Number 8.

3.  Device A:

• Receives ACK 8, purges TLPs ≤ 8 from retry buffer.
• Resets REPLAY_TIMER (forward progress made).
• Timer restarts if buffer has TLPs, else stays at 0.

Error Case (NAK Case)

1. Device A: Sends TLPs 4095, 0-3.
2. Device B: Receives TLP 4095, then detects CRC error in TLP 0. Sends NAK with Sequence Number 4095 immediately (no latency wait), resets ACKNAK_LATENCY_TIMER. Drops subsequent TLPs (1-3) as out-of-order.
3. Device A: Receives NAK 4095, purges ≤ 4095, resends 0-3. Resets REPLAY_TIMER, starts counting post-retransmission.

NOTE: REPLAY_NUM is a 2-bit counter, it will count 00,01,10,11 and then rolls back to 00 when it rolls back, a fatal error is generated and the link will go into the inactive state and link retraining should be initiated.

Timeout Case (RETRY_TIMER Expiry)

Receiver has Sent ACK for TLPs till sequence number 15 and has set Next_SEQ_NUM as 16. TLP with sequence number 16 is lost during transmission and the receiver receives TLP with sequence number 17.

Since Next_SEQ_NUM is not the same as the received TLP sequence number, the receiver schedules a NAK with seq number 15 and rejects the TLP with the sequence number 17.

This allows the transmitter to understand that TLPs till 15 were received correctly and TLP with sequence number 16 was not exchanged. Transmitter sends a TLP with sequence number 16 and data exchange continues without problems.