CMS Endcap Muon DDU: Output Data Format

Production Version

DDU Transmission Format

The DDU will send at least some minimal information for EVERY L1A received, even if there was no CSC track-stub data to read out. This minimal DDU data consists of 3 HEADER words (header 1, 2, 3) and 3 TRAILER words (trail-2, trail-1, trail). Any CSC data for the event (if present) is contained between header3 and trail-2. The CSC data is read out sequentially from every CSC source that has data for the event.

Note: the Lone Word from CSC "No Data" events (always Code word "8"; see the DMB-DDU Format page) are processed and discarded by the DDU zero suppression algorithm. The useful information contained in those words is summarized in the DDU event header. The DDU operates using a 64-bit word architecture.

The DDU output format is equivalent in both the Main (S-Link64) and EMU SPY (Gigabit Ethernet, or GbE) DAQ paths:

The EMU DDU output data format is compatible with the CMS Common Data Format (CDF). Specifications for the CMS CDF are found here.

The main CMS DAQ path uses the S-Link64 protocol and transmits 64-bit wide data, plus a "65th" bit (bit64) for use as the control character (aka "K-bit"); this bit is only true (i.e. "high") for the first header and last trailer of every event. There are additional "CMS Directives" in the CMS DAQ protocol that are specified in the CMS CDF and implemented in the DDU Format (these details below).

DDU SPY data is transmitted byte-wise (from each 64-bit DDU data word) via Gigabit Ethernet lowest-byte first; the LSB (bit0) is always on the right. The SPY path data format is the same as for the S-Link path, except there is no 65th bit.

Overview of the DDU event record

Critical event information is encoded in the format of the DDU headers and trailers, as shown below.
DDU Word Type	bit63<------------------>bit0 (characters are HEX placeholders)	comment
Header1	5TNN/NNNN/XXXI/IIVK	bit64 = S-Link "K-bit" set HIGH
Header2	8000/0001/8000/HHHH	The "8/1/8" pattern is somewhat unique, highest bit of HHHH set to zero
Header3	LLLL/oooo/ZZZZ/GGMY	Highest bit of LLLL, ZZZZ & GG are set to Zero
CSC Data	Loop through CSCs with data for the event	DMB Code word 8 is suppressed at DDU
Trailer-2	8000/FFFF/8000/8000	This "8/F/8/8" pattern is unique
Trailer-1	SSSS/SSSS/QQQQ/PPPP	SSSS is detailed DDU status/error at End of Event, highest bit of QQQQ is DMB Full flag, highest bit of PPPP set to Zero
Trailer	A?WW/WWWW/RRRR/UUMK	bit64 = S-Link "K-bit" set HIGH

Where:

T = Event Type (4 bits)
N = Event Number (24 bits)
X = BX number (12bits)
I = Source ID (10+2 bits) //post-LS1 it is real CMS Source ID; before it was decimal 760 (2F8h) for TF S-Link, otherwise "dRR" where RR is RUI #
V = Format Version (4 bits) //post-LS1 it is DDU Format Version 7 (for new DMB/ODMB format); pre-2014 it was Version 6
K = S-Link Status (lowest 4 bits are reserved in each S-Link64 K-word)
H = DMBs that reached Full FIFO (15 bits)
L = DDU Inputs Connected to "Live" CSCs (15 bits)
o = DDU Output Path Status (16 bits)
Z = DDU Inputs (i.e. CSCs) with Data for this Event (15 bits)
G = DDU Beginning of Event Status (7 bits)
M = DDU TTS/FMM Status (4 bits)
Y = Count of CSCs with Data for this Event (4 bits)
S = DDU Status/Error (32 bits)
Q = CSCs in Error State (15 bits)
P = CSCs in Warning State (15 bits)
W = DDU 64-bit Word Count (24 bits)
DDU_WC = (6 + 25*N_ts*nCFEB + 3*nDMB) + 64*nTMB + 99*nALCT
R = CRC Check Word (16 bits)
U = Event Status (8 bits) {repeat some elements of DDU Status here}
? = not used yet, but reserved for future use (4 bits each)
More details below

Header1 (S-Link64 DAQ Control word)

bit[63:60] = 0x5       //hard-coded in DDU firmware by CMS directive
bit[59:55] = 4-bit "event type"       //CMS directive, no details yet
bit[55:32] = 24-bit Level 1 event number       //from DDU L1A counter
bit[31:20] = 12-bit Bunch Crossing Number       //by CMS directive
*may come from DDU, DMB, ALCT or TMB
*appropriate matching req'd at DDU
bit[19:8] = 12-bit Source ID          //CMS directive, no details yet
bit[7:4] = 4-bit "format version"   //currently "6" for Production DDU, 5 for previous version, 3-4 for preproduction boards
bit[3:0] = 4-bits Reserved               //CMS directive, no details yet
*special "S-Link64 Protocol" status bits

Header2

bit[63:16] = a 48-bit constant, hard-coded in DDU firmware: 0x8000/0001/8000
*this is a "somewhat unique DDU word" near the start of the event packet; rare, but not guaranteed to be unique
bit[15] = 0, hard-coded in DDU firmware
bit[14:0] = shows which DMBs have reached a FIFO Full state (High True)

Header3

bit[63] = 0, hard-coded in DDU firmware
bit[62:48] = 15-bits indicating which DDU Fiber Inputs have a "Live" fiber (High True, 1 Fiber Input per CSC)
bit[47:32] = 16-bits indicating the DDU Output Path status, defined below:
bit47: DDU Output-Limited Buffer Overflow occurred (may be OK)
*an output path (GbE/SPY or DCC/S-Link) could not handle the rate, resulting in a buffer overflow
bit46: DAQ Wait was asserted by S-Link or DCC (status only)
bit45: Link Full (LFF) was asserted by S-Link (status only)
bit44: DDU S-Link Never Ready (status only)
bit43: GbE/SPY FIFO Overflow occurred (status only)
bit42: GbE/SPY Event was skipped to prevent overflow (status only)
bit41: GbE/SPY FIFO Always Empty (status only)
bit40: Gbe/SPY Fiber Connection Error occurred (status only)
bit39: DDU Buffer Overflow caused by DAQ Wait (RESET req'd)
*S-Link could not handle the rate, resulting in a buffer overflow
bit38: DAQ Wait is set by DCC/S-Link (status only)
bit37: Link Full (LFF) is set by DDU S-Link (status only)
bit36: Not Ready is set by DDU S-Link (status only)
bit35: GbE/SPY FIFO is Full (status only)
bit34: GbE/SPY Path was Not Enabled for this event (status only)
*no SPY data was sent due to prescale or throughput limitation
bit33: GbE/SPY FIFO is Not Empty (status only)
bit32: DCC Link is Not Ready (may be OK)
bit[31] = 0, hard-coded in DDU firmware
bit[30:16] = 15-bits indicating which CSCs have data for this event; one bit allocated per DDU fiber input
bit15 = 0, hard-coded in DDU firmware
bit[14:8] = 7-bit DDU Beginning of Event Status, defined below:
bit14: DDU single event warning (possible data problem, may be OK; RESET?)
*minor format error, fiber/RX error, or the DDU lost it's clock for some time; possible data loss
*consider RESET if this warning continues for consecutive events
bit13: DDU SyncError (bad event, RESET req'd)
*Multiple L1A errors or FIFO Full; possible data loss
bit12: DDU detected Fiber Error (hardware configuration change, RESET req'd)
*change of fiber connection status or No Live Fibers; a hardware problem probably exists
bit11: DDU detected Critical Error, irrecoverable (RESET req'd)
*OR of all possible "RESET required" cases
bit10: DDU detected Single Error (bad event)
*OR of all possible "bad" cases at Beginning of Event
bit9: DDU detected DMB L1A Match Error (bad event, RESET?)
*the DDU L1A event number match failed for 1 or more CSCs; possible one-time bit error
*if error continues for consecutive events then RESET req'd
bit8: DDU Timeout Error (bad event, RESET req'd)
*data from a CSC never arrived
*an unknowable amount of data has been irrevocably lost

bit[7:4] = 4-bit DDU TTS/FMM Status //current TTS status as reported to FMM, see CDF specification
bit[3:0] = number of CSCs which have data for this event

Trail-2

bit[63:0] = a 64-bit constant, hard-coded in DDU firmware: 0x8000/FFFF/8000/8000
*this is a "DDU unique word" near the end of the event packet

Trail-1

bit[63:32] = 32-bit DDU status/error code, defined below: (32-bit word, shifted by 32 bits)
bit63: CSC LCT/DAV Mismatch occurred (bad event)
bit62: DDU-CFEB L1 Number Mismatch occurred (bad event)
bit61: No Good DMB CRCs were detected in this Event (perfectly normal empty event or possible bad event?)
bit60: CFEB Count Error occurred (bad event)
bit59: DDU Bad First Data Word From CSC Error (bad event)
bit58: DDU L1A-FIFO Full Error (RESET req'd)
*the DDU L1A-event info FIFO went full; some triggers/events may be lost or garbled
bit57: DDU Data Stuck in FIFO Error (RESET req'd)
bit56: DDU NoLiveFibers Error (status only)
*no DDU fiber inputs are connected, something is wrong; will cause other errors...
bit55: DDU Special Word Inconsistency Warning (possible bad event?)
*a bit-vote failure occured on an input fiber channel
bit54: DDU Input FPGA Error (bad event)
bit53: DCC/S-Link Wait is set (status only)
bit52: DCC Link is Not Ready (may be OK)
bit51: DDU detected TMB Error (bad event)
*TMB trail word not found or TMB L1A, CRC or wordcount inconsistent
bit50: DDU detected ALCT Error (bad event)
*ALCT trail word not found or ALCT L1A, CRC or wordcount inconsistent
bit49: DDU detected TMB or ALCT Word Count Error (bad event, RESET?)
*TMB/ALCT wordcount inconsistent
*if error continues for consecutive events then RESET req'd
bit48: DDU detected TMB or ALCT L1A Number Error (bad event, RESET?)
*TMB/ALCT L1A Number mismatch with DDU
*if error continues for consecutive events then RESET req'd
bit47: DDU detected Critical Error, irrecoverable (RESET req'd)
*OR of all possible "RESET required" cases
bit46: DDU detected Single Error (bad event)
*OR of all possible "bad event" cases
bit45: DDU Single Warning (possible bad event?)
*OR of bit55, bit42
bit44: DDU FIFO Near Full Warning or DAQ Wait is set (status only)
*OR of all possible "Near Full" cases
bit43: DDU detected Data Alignment Error from 1 or more inputs (bad event)
*CSC data violated the 64-bit word boundary
bit42: DDU Clock-DLL Error (may be OK, RESET?)
*the DDU lost it's clock for an unknown period of time; some triggers/events/data may be lost
bit41: DDU detected CSC Error (bad event)
*Timeout, DMB CRC, CFEB Sync/Overflow, or missing CFEB data
bit40: DDU Lost In Event Error (bad event, but end was found)
*the DDU failed to find an expected control word within the event, NOT fatal
bit39: DDU Lost In Data Error (bad event, RESET req'd)
*usally Fatal; DDU checking algorithms are irrevocably lost in the data stream
*mis-sequenced data structure, possible that different events were run together
*found at least one of the following in the event data stream, all of which are very bad:
-Extra CSC_First_Word before CSC_Last_Word
-Extra DMB_Header2 before DMB_Last_Word
-Lone Word before DMB_Last_Word
-Extra TMB/ALCT_Trailer before DMB_Last_Word
-Extra DMB_Trailer1 before DMB_Last_Word
-DMB_Trailer2 before DMB_Trailer1
Note: CSC_Last_Word == DMB_Trailer2

bit38: DDU Timeout Error (bad event, RESET req'd)
*data from a fiber input either never started or never finished
*an unknowable amount of data has been irrevocably lost
bit37: DDU detected TMB or ALCT CRC Error (bad event, RESET?)
*CRC check failed on 1 or more TMB/ALCT; possible one-time bit error
*if error continues for consecutive events then RESET req'd
bit36: DDU Multiple Transmit Errors (bad event, RESET req'd)
*one bit-vote failure (or Rx Error) has occured on multiple occassions for the same CSC
bit35: DDU Sync Lost/Buffer Overflow Error (bad event, RESET req'd)
*an unknowable amount of data has been irrevocably lost
bit34: DDU detected Fiber Error (hardware configuration change, RESET req'd)
*change of connection status on 1 or more DDU fiber inputs; a hardware problem probably exists
bit33: DDU detected DMB or CFEB L1A Match Error (bad event, RESET?)
*the DDU L1A event number match failed for 1 or more CSC boards; possible one-time bit error
*if error continues for consecutive events then RESET req'd
bit32: DDU detected DMB or CFEB CRC Error (bad event, RESET?)
*CRC check failed for ADC data on 1 or more CFEBs; possible one-time bit error
*if error continues for consecutive events then RESET req'd
Note: error bits requiring RESET persist until the RESET occurs.
bit[31] = DMB Full Flag (status only)
bit[30:16] = shows which CSCs are in an Error state
bit[15] = 0, hard-coded in DDU firmware
bit[14:0] = shows which CSCs are in a Warning state

Trail (S-Link64 DAQ Control word)
bit[63:60] = 0xA       //hard-coded in DDU firmware by CMS directive
bit[59:56] = 4-bits unspecified       //CMS directive, not assigned yet so I used them...
*currently use bit58 as "Track Finder data detected" flag, and bit59 as "Track Finder data error" flag

*currently use bit56 as "Close L1As" flag that indicates events closer than 1000 nsec
bit[55:32] = 24-bit DDU Word Count for the Event       //number of 64-bit words sent from DDU
bit[31:16] = 16-bit Event CRC
bit[15:12] = 4-bits Reserved      //CMS directive, do not use
bit[11:8] = 4-bit Event Status       //from DDU
*currently equivalent to Trail-1 bits [47:44]
bit[7:4] = 4-bit DDU TTS/FMM Status       //current TTS status as reported to FMM, see CDF specification
bit[3:0] = 4-bits Reserved       //used for CMS CDF protocol

Link to previous "Version 6" DDU format (used 2007-2014).
Link to second DDU Prototype (small green DDU) Output Data Format.

Useful for beamtest data from October 2004.

Also good for big red DDU at CERN Slice test (old DDU with updated firmware).

Link to old DDU Prototype (big red DDU) Output Data Format.

Useful for beamtest data from May 2003 through June 2004.

Also good for big red DDU at UCLA (has original firmware).

Revision Summary:

23 October 2002: Original version (J. Gilmore, gilmore@mps.ohio-state.edu)

5 December 2002: Updated status decode in Header3 (bit57, 40, 39) and added comments in table (J. Gilmore)

1 May 2003: Modify trigger board info now that TMB and ALCT have separate data flows (J. Gilmore)

19 April 2004: Fix error in Header1 "fov" description (J. Gilmore)

4 May 2004: Fix typo in bit50 (J. Gilmore)

24 Sept. 2004: Modify for New DDU; code 9's are kept now, last Trailer data is rearranged to match CMS directive, DDU Status bit37 redefined (J. Gilmore)

11 July 2005: Modify for Production DDU; redefined Headers 2&3 and Trail-1 (J. Gilmore)

24 Aug 2005: Update for production history (J. Gilmore)

28 April 2006: Update definition for SourceID and Beginning of Event Status (J. Gilmore)

13 October 2006: Improved trailer status definitions (J. Gilmore)

16 November 2006: Added links to CMS C.D.F. specification (J. Gilmore)

30 November 2007: New ALCT/TMB Format version, FOV=6 now (J. Gilmore)

9 April 2014: New DMB/ODMB Format version, FOV=7 now (J. Gilmore)

10 April 2014: Source ID is now the real CMS DAQ Source ID, not the RUI # (J. Gilmore)