DDU Master Log ============== Current firmware versions (as of 8 Sept 2009): B0020A04 11/13/08 CF048A01 6/19/09 DF025A06 4/24/09 Send back to USC55: #19, #40, #17, #24? ================== To-Repair List ======================== DDU#7: water damage. at b904-office... cleaned again 27 May 2014, might be OK; it has bad 80.000 MHz oscillator! -took 80.000 MHz oscillator off of dead DDU#24, but it's still buggy (30 May 2014) DDU#18: fibers 4-7 have trouble, sometimes "see" data when they have none. Bad at b904 too. Keep At B904 for now. -keep at B904, OK for test stand (28 May 2014) DDU#24: fiber #3 still bad after Finisar swap? FPGA pins A6+A7 Reflowed at b904... now Much Worse! -have to R&R InFPGA0... fix at OSU (30 May 2014) --sent to OSU 25 June 2014 (Xiaofeng) DDU#27: DDUfpga "IN14" from gigabit FIFO looks bad, probably ball joint where FIFO pin F1 (bottom edge) goes to FPGA pin A25 (LR corner), send to TAMU? -Reflowed at b904... looks OK! Needs More Testing (30 May 2014) but did not get it. ***now in USC! as Source ID 855 (FED crate 4, slot 9)*** DDU#6: seems like DDU 27... lots of CRC errors are created on the local DAQ path, probably a bad bit between FIFO and FPGA. -Reflow at b904 SOON July 2014 problems with S-Links at USC: DDU#30 (does not send S-Link data for any L1A) --from FED crate 4 slot 6 DDU#23 (S-Link detects CRC errors from DDU), see http://cmsonline.cern.ch/cms-elog/819384, replaced by #27 --from FED crate 4 slot 9... #23 is probably still a good board! Looks like the S-Link was bad. SrcID 861: worked for one day during MWGR#1, but sent no data on S-Link the next day. local DAQ is OK though. ** In b904: #2-ME1/1 From old eLog: RUI #34 (slot 11 in crate 4) board ID=18 was replaced with DDU board ID=17. board ID=18 input 5 was "seeing data" when any fiber DMB fiber is plugged in (even though DMB is not sending any data) and also inputs 6 and 7 were always in error. See Misha's elog here: http://cmsonline.cern.ch/cms-elog/816055... is it Input 4 or 5? -keep at B904, OK for test stand ========================<<<<<< Production DDUs >>>>>>======================== Location Log ------------ location update on 3/31/2014: DDU LOCATIONS 3 at TAMU (#48 - 50) 1 at UF... but now at UCSB for ODMB testing 1 in SX5 for ME1/1 testing? 36 in USC (minimum req'd is 36+1 for normal operation) 8 in b904 (4 in FED crate, 1 in ME1/1, 3 in office area & 1 for ME4/2)? DDU #24 at OSU for repair TOTAL = 50; all are accounted for older location status, pre-2012: #1 at UF (firmware update 30 Sept. 2005 (board ID 99 on ISPROM) -- still there, 12/14/2011. But NO Front Panel cutout!!! #2,4,6 at CERN (update 30 Sept. 2005) --more recently in Nov? #5 sent to CERN (b904?) on 24 Jan 2006. #7-11 updated, tested and sent to CERN 6 April 2006; rcv'd 12 April 2006. #3 sent to CERN (b904?) on 16 June 2006. #12-50 at OSU (some updates & testing required) #12-39 sent to CERN with 2 DCCs & 9U filler panels; 28 Sept 2006. #4 modified for STEP, like a TF-DDU now, but no S-Link connectors; Feb 2007. #24 brought back to OSU for repair, programming problem? -No, old "Sometimes DDU stays Busy..." problem! Fixed 25 Oct 2007. #24,40-45: updated firmware, ready for shipment to CERN; 25 Oct 2007. -shipped 9 Nov 2007. 8 Sept 2009, currently at OSU: -48, 49, 50 are good to go > one of these has some testing wires soldered on it -23 had a problem at Cern, OK here? Resolved itself, details below > now at b904... Ben took it March 2014. see below -27 recently returned from Cern, likely bad ball joint on FPGA or FIFO > bad data on a GbE data bit, see below > now at b904... Ben took it March 2014 > bring it to TAMU for repair -07 stays busy on powerup/hotplug. tapping on faceplate or clicking one of the VME latches makes it program! Strange...bad joint? > now at Rice, since 10 Aug 2011. Why did they send this one? > came to TAMU & Elec shop fixed the bad ball joint in 2012 (soldered a wire under the FPGA to close the gap) > now at b904? ====================<<<<<< Production Testing Notes >>>>>>==================== *** Don't forget to watch Grn/Yel Fiber LEDs during tests 6 & 7 *** DDUtest bug? Running through from test5-to-test6, software doesn't check correctly and doesn't load InPROM, then fails test 6 due to errors from wrong firmware. --double-check if check fails! Fixes for DDUtest: -Version checks should be re-done if 1st try is a mismatch -Version check should look for Hex vvErr or Hex vvArr as needed -> B/C/D determine which FPGA it is for -> see test 11 -also reformat BitTest "fake.event" to print Hdr/Tr as 4 * 16-bits -check Board ID at start of each test? Recheck once if mismatch. -> Abort or Warn if mismatch repeats? =========================<<<<<< DDU Repair Log >>>>>>========================= #17 blew a cap at 40MHz osc, destroyed pad & via...Shane can fix? :: FIXED! -can't program VME FPGA, although DTACK does flash as normal... -Reset push-button ARST has bad joint....ok now. -use laptop to Readback/Verify PROM! No good... used laptop to PROGRAM & verify is OK....must be VME problem! --reflowed many VME pins (esp. U24) & ARST...BETTER! -FIXED...but still needs some touch-up work...DONE! -Several Finnisar's were not soldered well! Nov 2013: taken to b904 from Pt5 due to dead GbE Spy channel. --> Special firmware that ignores FOK false shows it works, so the FOK is broken here --> replaced the GbE Finisar... yes this FIXED it! (29 May 2014) #18 got through test5 OK; fibers 4-7 give RxError constantly with :: FIXED! fibers plugged in (DAVs flashing), fails test6 -maybe a problem with power to InRD1? -80MHz ClkDrv joint or BREF term? R133,108,131,110 (50Z?) All GOOD. ---reflowed all, no better. -power checks OK...test Clk80 & termination: compare to Good Board. -4 fibers have common: BREF, Clk Driver, power & filters -2.5VA has only ~90ohms to GND....S.B.~8kohms...WHY? Related? -> reflowed Finisar's 4-7, got 100 ohms--no better, EXCEPT when the board was hot-plugged did better; why? -> reflowed all 2.5VA passives, now 8kohm to GND...but Fibers4-7 still bad unless HOT-PLUGGED....Why? -Check BREF Clk for InRD1 on scope? Compare with other good BREF? InCtrl-1 FPGA LEDs never light up...8 BACKWARDS LEDs...done. -as board gets warm, usually No DONE for InFPGAs! -> OK sometimes; could be water issue from recent washing--RETEST! Finnisar #7 needed alignment...used 2 irons, somewhat better now. --> V25P ~SHDN pullup resistor shorted to V33P, OK. --> broke pad off of V15P EN pullup resistor....OK. ===>> Checked v15p power-on curve: power cycle looks Crappy, but Hot-Plug ===>> looks clean....need to delay V15P EN turn-on: ===>> Added 3.3uF cap on V15P EN (RC const=30msec): FIXED! June 2008: see repeat of "fibers 4-7 RxError" problem in FED4-slot11, Hot-Plug to fix. #19 fails test 5 after few events, many bad bits, FIFO3 (& 2?) :: FIXED! -often good for a while, then burst of errors...on 3, primarily low bits (12,8,3,2...) ->made it past 80sec in test 5 once...then 8700 errors by 100sec. -maybe bad R/WCLK termination? R81,84 (220Z?) All GOOD, ~114Z also possibly r69f01,r70f01,r67f01,r68f01 (220Z?) All GOOD, ~114Z -80MHz ClkDrv joint or BREF term? R132,109,130,111 (50Z?) All GOOD. -reflowed termination R's... -reflowed ClkDrv80 pins 5-10; _OE_ WEN/WCLK, REN/RCLK on FIFO 2 & 3. -> OE2 looked bad...redone: still bad. -Note that InFPGA1 L1 scaler gets behind, sometimes STOPS... ---> possible DLL Lock Error (Check LEDs!), or maybe bad C-codes? ---> Rflow more (all?) FIFO2/3 pins with blow gun, no better -==> BUT, error-free when HOT-PLUGGED....Why? -FIFO3-A corner pins (OE/VCC) have bad pads....seems OK though. -Check all Clks (InFPGA & FIFOs) on scope & Compare to good ones! -add 10pF cap on CkFB? no... --> V25P ~SHDN pullup resistor shorted to V33P, OK? ===>> Add 3.3uF cap on V15P EN (RC const=30msec): slow turn-on FIXED! 3 July 2008: eLog4477 or 6989, InFPGA1 fails JTAG communication, swap OUT of FED4-slot12. -tested JTAG 28 May 2014, it works fine... maybe it was a random glitch at USC. #20 failed test 4:: fixed! Retest with updated DDUtest? -InFPGAs frequently fail to program after HardReset; sometimes OK -Check Clk80 & termination. Reflowed Clk80 & InPROM joints. -Reflow is better....then bit errors, FIFO0 b3,4; add 10pF cap #21 failed test 5:: fixed! -bits 10 & 42 on FIFO3 -newest firmware will fail test 6...Single Warn, DLL Error? Rx Error? InCritErr? Never saw DLL error from DDUctrl... #22 fails test1:: fixed! -VMEctrl programs, but won't respond to VME commands -FIXED: Bad 80MHz oscillator, R&R. #23 fails test5:: fixed! -FIFO1 got 5 vote errors on bits 4,3 -reflow & retry...no good. -add 10pF cap on CkFB. -> BETTER, past test 5 now...FIXED! March 2009: InFPGAs won't set DONE after firmware upgrade --PROMs read UserID OK, but both FPGAs give FFFs --assume a bad solder joint, but problem goes away with no action! Tried reflow anyway on PROMs (saw no problems), still OK...watch for repeat. #24* had no Fuse sockets & bad DONE LED (yellow) on front panel :: FIXED! -note: Sometimes DDU stays Busy on power-up (hot-plug usually) and HardReset does not help recover, but power-cycle does...cause? --> check clk40/80 & Reset out of FPGAs...bad joint? ===>> Add 3.3uF cap on V15P EN (RC const=30msec): slow turn-on, no help. --> try shorting ~SHDN on V25P, no good, back to 10k ohm. Note that Hot Plugging the board works fine when warm! (but this Did Not Work in May 2014) 25 Oct 2007: repeat of "Sometimes DDU stays Busy..." problem! See that CK80 is DEAD when cold! checked in VMEctrl and on board -replaced 80MHz osc. (cs3280al), no help? reflow, try again when cool! -OK after last reflow: hot or cold, hot-plug or normal power-up. fixed! 25 May 2014: dead #3 Input channel. Also all green LEDs blink non-stop after power-up. No 80.000 MHz on scope. -replaced 80.000 MHz oscillator, FIXED green blinking LED problem (27 May 2014) -replaced Finisar#3, but still no good (28 May 2014) -reflowed InFPGA0 (29 May 2014) #25 Short 3.3V-GND at Finisar, bit errors on FIFO0, bits 1-4,8 :: FIXED! -add 10pF cap on CkFB....Fixed! #26 fails test7: InRD2 gives RxError a lot (Inputs 8-11) after a few :: FIXED! good events, probably during IDLE between events. -reflowed Finisar's & BREF at both ends, no better. -> Try again, & clean it up...no better! -==> BUT, error-free when HOT-PLUGGED....Why? -Yellow DAV LED 9 does not work...loose joint, done. -latest: seemed better now in Slot3; passed 3 times with power cycle, but then did PROGRAM button & power cycle and failed. ===>> Add 3.3uF cap on V15P EN (RC const=30msec): slow turn-on FIXED! #27 backwards 80MHz osc, 16ohms on 3.3V-GND :: FIXED! -bit errors on FIFO0 bits 11,8,5-1, add 10pF cap. -Yellow DAV LEDs 0,1,2,6,7,8 are GREEN, fixed. June 2009: DDUfpga "IN14" from gigabit FIFO looks bad, probably ball joint where FIFO pin F1 (bottom edge) goes to FPGA pin A25 (LR corner), try to reflow sometime. #37 bad joints on Temp/Voltage Serial ADC, fails test 2 :: FIXED! #39 VME FPGA won't program after Emergency Load :: FIXED! -bad joint on L-L corner of VME chips, reflow no help -load and verify PROM from Laptop is OK, but still no DONE ===>> Problem is V25P regulator joints; reflow is GOOD! #40 got through test5 OK; fibers 4-7 give immediate busy/error (DLL :: FIXED! Error/RxError with fibers plugged in, DAVs flashing), fails test6 -seems OK when Hot-Plugged, add 3.3uF cap on V15P, FIXED! March 2014: taken to b904 from Pt5 due to errors on fiber input #14 or 13? --> these corruption errors usually occurred a few minutes into a cosmic run -tested at b904, but problem was not observed, just known ODMB errors (28 May 2014) In USC: seems OK when Hot-Plugged, but otherwise fiber #8-9 flash yellow LEDs when not plugged in (no green LEDs lit; 30 May 2014) #41 bad joints on Temp/Voltage Serial ADC, fails test 2 :: FIXED! #42 bad 80MHz Osc. & bad GbE DAV LED! :: FIXED! #44 bad joints on Temp/Volt serial ADC :: FIXED! #47 bad joints on Temp/Volt serial ADC :: FIXED! #50 backwards VME Chip :: FIXED! Production issues: frequent bad solder joints on corner pins #17,20,21,39 backwards 80MHz oscillators; R&R #11,13,14,16 VME receive buffer U32 pins 26-28 shorted #14 FIFO2-A bits D1-D2 shorted #14 bad joints on Temp/Voltage Serial ADC #37,41,44,47 bad joints on 2.5V regulator #39 backwards 47uF filter cap at 1.5V regulator #15 Soft Reset button does not work; reflow solder #15 ----------- total of 14 boards had solder joint issues ----------- CkFB too fast (FIFO0/1 b1,2,3,4,8); add 10pF cap #11,15,20,21,23,31, to resolve .5ns FPGA variations 34,35,42,43,46,47, 48,49,50 bad FIFO0 (bit errors), R&R FIFO0 (A or B?) one of 5-10 Fiber/FIFO bit errors, entire Bank, many bits 18,19,26,40 24? -often fixed when Hot-Plugged: cause is Bad V15P turn-on, add 3.3uF cap to 10k-ohm Regulator Enable circuit bad 80MHz Oscillator one of 5-10? 22 missing Fuse Sockets #24,42 missing Discharge Resistors #34 wrong grn/yel Input LEDs (0,1,2,4,5,9,10,11,12,13) #13,14 bad yellow Input LEDs, #44(4), 46(1,6) double HSC's to DCC #11,12,14,15,27 missing all HSCs! 9 boards total Sometimes DDU Busy on power-up, HardRst no help #24 (hot-plug usually) 3.3V-to-Gnd Resistance 15+ ohms: 3,5,7,8,9,10,11,13,15,27,42-49 all others are 4.5-5.5 ohms. ========================<<<<<< Old DDU-4 status >>>>>>======================== short Green DDU --------------- Board has 6 channels installed (all working) -Normal Firmware, InCtrl:df012a01, DDUctrl:cf028a02, VMEctrl:b0005a01 -FiberMult4 option, InCtrl:df008a01 (in4ctrlv8r1_[0,1].svf?) -Random data (DDR test) option, Inctrl:df004e01 (testin) DDUctrl:cf012e02 (ddutest) In FED crate at OSU since Oct. 04... Scavenged for parts Dec. 2005: Es Ist kaput! ====================<<<<<< Older DDU (DDU-3) status >>>>>>==================== 2 Big Red DDUs -------------- Full Board has 14 working channels, big GbE FIFO, new 2.5V power for SLINK, tune SLINK-Full logic, new hand-wired SLINK UCLK via TP3 -tested all input fibers 2/16/05: #0-8,10-14 worked perfectly (GbE & SLINK readout). #9 passes no data, due to mortality in early tests...why? #4 used to have problems (MT route/timing) but it's fine now. Firmware: Dctrl:v22r1 (22d00001), Ectrl:v20r15512 (20a15512) --2/16/2005 was Dctrl:v19r15509, Ectrl:v20r15512 --2/8/2005 -latest firmware adds: Special I/O to drive SLINK UCLK (v22) ---> Dctrl v22r2-6 are for UCLK timing tests, don't use! DDU-CRC16 to SLINK for CMS-C.D.F. compatibility (v19) fixes minor VME-JTAG bug (v19) makes EthCtrl compatible with new DDU GbE drivers ---> removes FF bytes at Start, adds 2-byte Pkt Counter at End -old versions were Dctrl:v15r1 (15cc0001), Ectrl:v13r1_tb versions used since mid-2003 beamtests 2/17/05: Sent to CERN (Geurts via Levchenko) 2/15/05: Re-route SLINK UCLK & pdate Dctrl firmware to drive it via TP3 2/8/05: Update Dctrl firmware 2/7/05: Update Ectrl firmware 2/2/05: Fix SLINK 2.5V power source 1/28/05: Rcv'd from CERN (Geurts) Oct. 04: Left at CERN for Slice Test (F. Geurts) Partial Board has 4 working channels, smaller GbE FIFO, no 2.5V power for SLINK (?), original SLINK UCLK direct from Clk Driver Firmware: Dctrl:v15r1 (15cc0001), Ectrl:v13r1_tb (13a00001)? versions used since mid-2003 beamtests Oct. 04: Taken to UCLA for TMB tests (M. von der Mey) To Do: update to Ectrl:v20r15512, Dctrl:v21r1 (v22r1 not needed!) only update to Dctrl:v22r1 if SLINK UCLK wire is changed DDU Production Plans (DDU-5) -------------------- We will manufacture 50 PCBs and assemble them all, but we ordered parts for 60 DDUs to give us 10 boards worth of spares. Install DDUs in FED Crates as follows: FED 1: DDUs 31-39. 32,34,36,38,39,37,35,33,31 FED 2: DDUs 21-29. 22,24,26,28,29,27,25,23,21 Note, currently DDU 24 is at OSU, so 30 is in its place. FED 3: DDUs 11-19. 12,14,16,18,19,17,15,13,11 FED 4: DDUs 41-45,6-9. 42,44,6,8,9,7,45,43,41 Note, DDUs 40-45 and 24 will be shipped to Cern soon. Would be great to swap DDUs for 1 and 3! Spare DDUs: 5,10,20,30,40 TF-DDUs: 1,2,3 STEP DDU: 4 (re-programmed as BrdID 1) 5 DDUs at OSU: 46-50 TF DDUs get VME pins D5 and D8 wired to U24 pins 1 and 36 respectively - for Extender Card direction control. - these are VME reserved pins, no connections to anything. > safe to plug TF DDU into FED Crate.