I had a VD offline due to a firmware bug.
I'm using the latest firmware version.
This is what I can see in our logs:
07/02/16 3:31:57: EVT#30207-07/02/16 3:31:57: 113=Unexpected sense: PD 05(e0x20/s5) Path 5000c500134f0029, CDB: 28 00 0f f1 b4 46 00 00 08 00, Sense: 6/29/cd
07/02/16 3:31:57: Raw Sense for PD 5: f0 00 06 00 23 5f 29 0a 00 00 00 00 29 cd ce 00 00 00
07/02/16 3:31:57: EVT#30208-07/02/16 3:31:57: 113=Unexpected sense: PD 05(e0x20/s5) Path 5000c500134f0029, CDB: 28 00 0a 0a 45 0e 00 00 08 00, Sense: 2/04/01
07/02/16 3:31:57: Raw Sense for PD 5: 70 00 02 00 00 00 00 0a 00 00 00 00 04 01 00 00 00 00
07/02/16 3:31:57: EVT#30209-07/02/16 3:31:57: 113=Unexpected sense: PD 05(e0x20/s5) Path 5000c500134f0029, CDB: 28 00 0b a0 e5 86 00 00 18 00, Sense: 2/04/01
07/02/16 3:31:57: Raw Sense for PD 5: 70 00 02 00 00 00 00 0a 00 00 00 00 04 01 00 00 00 00
07/02/16 3:31:57: EVT#30210-07/02/16 3:31:57: 113=Unexpected sense: PD 05(e0x20/s5) Path 5000c500134f0029, CDB: 28 00 09 88 c5 fe 00 00 02 00, Sense: 2/04/01
07/02/16 3:31:57: Raw Sense for PD 5: 70 00 02 00 00 00 00 0a 00 00 00 00 04 01 00 00 00 00
07/02/16 3:31:57: EVT#30211-07/02/16 3:31:57: 113=Unexpected sense: PD 05(e0x20/s5) Path 5000c500134f0029, CDB: 28 00 0e 90 4e 00 00 00 56 00, Sense: 2/04/01
07/02/16 3:31:57: Raw Sense for PD 5: 70 00 02 00 00 00 00 0a 00 00 00 00 04 01 00 00 00 00
07/02/16 3:31:57: EVT#30212-07/02/16 3:31:57: 113=Unexpected sense: PD 05(e0x20/s5) Path 5000c500134f0029, CDB: 28 00 0f f1 b4 46 00 00 08 00, Sense: 2/04/01
07/02/16 3:31:57: Raw Sense for PD 5: 70 00 02 00 00 00 00 0a 00 00 00 00 04 01 00 00 00 00
07/02/16 3:31:57: EVT#30213-07/02/16 3:31:57: 113=Unexpected sense: PD 05(e0x20/s5) Path 5000c500134f0029, CDB: 00 00 00 00 00 00, Sense: 2/04/01
07/02/16 3:31:57: Raw Sense for PD 5: 70 00 02 00 00 00 00 0a 00 00 00 00 04 01 00 00 00 00
07/02/16 3:31:58: MPI_EVENT_SAS_DISCOVERY: PortBitmap ff - Discovery is in progress
07/02/16 3:31:58: MPI_EVENT_SAS_DISCOVERY: PortBitmap 0 - Discovery is complete
07/02/16 3:31:58: Disc-prog= 0....resetProg=0 aenCount=0 transit=0
07/02/16 3:31:59: MPI_EVENT_SAS_DISCOVERY: PortBitmap ff - Discovery is in progress
07/02/16 3:31:59: MPI_EVENT_SAS_PHY_LINK_STATUS - PhyNum 5 DevHandle 6 Link 09
07/02/16 3:31:59: MPI_EVENT_SAS_DEVICE_STATUS_CHANGE
07/02/16 3:31:59: MPT_EventDeviceStatusChange: Device Removed DevId 5 Tgt 5 Sas 5000c500:134f0029
07/02/16 3:31:59: curQdepth 0 WaitQCount 4 path 0 flags:f0500005
07/02/16 3:31:59: DM_DevicePathRemoved devId 5 Tid 5 Path 0
07/02/16 3:31:59: PD 5 Removed :DeviceCount=6
07/02/16 3:31:59: MPI_EVENT_SAS_DISCOVERY: PortBitmap 0 - Discovery is complete
07/02/16 3:31:59: Disc-prog= 0....resetProg=0 aenCount=0 transit=0
07/02/16 3:31:59: DM_PdScsiTypeSet: Pd 5 type 1f isSata 0
07/02/16 3:31:59: EVT#30214-07/02/16 3:31:59: 112=Removed: PD 05(e0x20/s5)
07/02/16 3:31:59: EVT#30215-07/02/16 3:31:59: 248=Removed: PD 05(e0x20/s5) Info: enclPd=20, scsiType=0, portMap=05, sasAddr=5000c500134f0029,0000000000000000
07/02/16 3:31:59: EVT#30216-07/02/16 3:31:59: 114=State change on PD 05(e0x20/s5) from ONLINE(18) to FAILED(11)
07/02/16 3:31:59: EVT#30217-07/02/16 3:31:59: 81=State change on VD 00/0 from OPTIMAL(3) to PARTIALLY DEGRADED(1)
07/02/16 3:31:59: modify_log_drv_state: oldState: 3 newState: 1 pinned_cache_present: 0 targetId: 0
07/02/16 3:31:59: EVT#30218-07/02/16 3:31:59: 250=VD 00/0 is now PARTIALLY DEGRADED
07/02/16 3:31:59: EVT#30219-07/02/16 3:31:59: 114=State change on PD 05(e0x20/s5) from FAILED(11) to UNCONFIGURED_BAD(1)
07/02/16 3:32:04: EVT#30220-07/02/16 3:32:04: 332=Enclosure PD 20(c None/p0) element (SES code 0x17) status changed
07/02/16 3:32:04: SES_BackplaneMapping: Un-Associated device on enclPd 20 StsCode = 6 elmtType 17 elmtIndex 5 slotPd =5 SasAddr =0
07/02/16 3:32:04: SES_MarkBadElement: enclPd 20 timeDiff e1a1b slot 5 badElmt 1 retryCnt 0 oldTime:0 currentTime:e1a1b
07/02/16 3:32:07: DM_REC: TUR Failed, devId =5
07/02/16 3:32:09: SES_BackplaneMapping: Un-Associated device on enclPd 20 StsCode = 6 elmtType 17 elmtIndex 5 slotPd =5 SasAddr =0
07/02/16 3:32:09: SES_MarkBadElement: enclPd 20 timeDiff 5 slot 5 badElmt 1 retryCnt 1 oldTime:e1a1b currentTime:e1a20
07/02/16 3:32:11: MPI_EVENT_SAS_DISCOVERY: PortBitmap ff - Discovery is in progress
07/02/16 3:32:11: MPI_EVENT_SAS_PHY_LINK_STATUS - PhyNum 5 DevHandle 6 Link 90
07/02/16 3:32:11: MPI_EVENT_SAS_DEVICE_STATUS_CHANGE
07/02/16 3:32:11: MPT_EventDeviceStatusChange: Device Inserted Tgt 5 Sas 5000c500:134f0029
07/02/16 3:32:11: MPI_EVENT_SAS_DISCOVERY: PortBitmap 0 - Discovery is complete
07/02/16 3:32:11: DM_DevMgrIsChipInit 0 State 400
07/02/16 3:32:11: DM: DM_DevSSUCallback TID 5 FAILED Cnt 0 Retry 0 Status 2
07/02/16 3:32:11: DM : DM_DevSSUCallback SENSE Len 12 Code 70 senseKey 6 asc 29 ascq cd
07/02/16 3:32:11: DM_DevNotifyRAID: Notify Done. Check for Removal
07/02/16 3:32:11: gDevInfo=a11ebd20, size=140
07/02/16 3:32:11: Total Device = 7
07/02/16 3:32:11: PD Flags State Type Size S N Vendor Product Rev P C ID SAS Addr Port Phy DevH
07/02/16 3:32:11: --- -------- ----- ---- -------- - - -------- ---------------- ---- - - -- ---------------- ---- --- ----
07/02/16 3:32:11: 0 f0400005 00020 00 22ecb25b 0 0 SEAGATE ST3300657SS 0008 0 0 00 5000c500094d8a29 00 00 0a
07/02/16 3:32:11: 1 f0400005 00020 00 11177327 0 0 SEAGATE ST3146356SS HS11 0 0 01 5000c5001351cf75 01 01 0b
07/02/16 3:32:11: 2 f0400005 00020 00 11177327 0 0 SEAGATE ST3146356SS HS11 0 0 02 5000c50013514b99 02 02 0e
07/02/16 3:32:11: 3 f0400005 00020 00 11177327 0 0 SEAGATE ST3146356SS HS11 0 0 03 5000c500135178b1 03 03 0f
07/02/16 3:32:11: 4 f0400005 00020 00 22eec12f 0 0 HITACHI HUS156030VLS600 A5D0 0 0 04 5000cca02a556ded 04 04 0c
07/02/16 3:32:11: 5 f0400005 00020 00 11177327 0 0 SEAGATE ST3146356SS HS11 0 0 05 5000c500134f0029 05 05 0d
07/02/16 3:32:11: 20 00400005 00020 0d 00000000 0 0 DP BACKPLANE 1.07 0 0 20 500240807e894400 09 08 09
07/02/16 3:32:11: 100 00400005 00020 03 00000000 0 0 LSI SMP/SGPIO/SEP 1909 0 0 ff 0 00 ff 00
07/02/16 3:32:11: PhyId 0 Sas 5000c500094d8a29 Type 1 IsSata 0, Smp 0:0
07/02/16 3:32:11: PhyId 1 Sas 5000c5001351cf75 Type 1 IsSata 0, Smp 0:0
07/02/16 3:32:11: PhyId 2 Sas 5000c50013514b99 Type 1 IsSata 0, Smp 0:0
07/02/16 3:32:11: PhyId 3 Sas 5000c500135178b1 Type 1 IsSata 0, Smp 0:0
07/02/16 3:32:11: PhyId 4 Sas 5000cca02a556ded Type 1 IsSata 0, Smp 0:0
07/02/16 3:32:11: PhyId 5 Sas 5000c500134f0029 Type 1 IsSata 0, Smp 0:0
07/02/16 3:32:11: PhyId 0 Sas 0 Type 0 IsSata 0, Smp 0:0
07/02/16 3:32:11: PhyId 0 Sas 0 Type 0 IsSata 0, Smp 0:0
07/02/16 3:32:11: Load Balance Statistics Path0PDs 0 Path1PDs 0
07/02/16 3:32:11: EVT#30221-07/02/16 3:32:11: 91=Inserted: PD 05(e0x20/s5)
07/02/16 3:32:11: EVT#30222-07/02/16 3:32:11: 247=Inserted: PD 05(e0x20/s5) Info: enclPd=20, scsiType=0, portMap=05, sasAddr=5000c500134f0029,0000000000000000
07/02/16 3:32:11: ArDiskTypeMisMatch : NO_MIXING_VIOLATION array=0 destPD=5
07/02/16 3:32:11: EVT#30223-07/02/16 3:32:11: 114=State change on PD 05(e0x20/s5) from UNCONFIGURED_BAD(1) to UNCONFIGURED_GOOD(0)
07/02/16 3:32:19: EVT#30224-07/02/16 3:32:19: 332=Enclosure PD 20(c None/p0) element (SES code 0x17) status changed
07/02/16 3:36:53: mfiIsr: idr=00000020
07/02/16 3:36:53: Driver detected possible FW hang, halting FW.
07/02/16 3:36:53: Pending Command Details:
07/02/16 3:36:53: cmdId= 68: cmd=11, cmdStat=0, num_sg_elements=1, status=1 [PCI_COMMAND]
07/02/16 3:36:53: mfa=cd856800, mf=a041a000, mfSge=a041a028, bytesTransferred=0, next ffff
07/02/16 3:36:53: startTime=0, lines=a17c0680, lineMap=0, activeRecoveryCount=0, lockPromotedByRec=0
07/02/16 3:36:53: ldbbmAlreadyTried=0, ldbbmIssueWriteAsWV=0
07/02/16 3:36:53: ld=0, ioFlags=0, start_block=3fc6d0c6, num_blocks=8, savedNumBlocks=8
07/02/16 3:36:53: group: 0068
07/02/16 3:36:53: start_row=1fe368, start_strip=7f8da1, num_strips=1
07/02/16 3:36:53: ref_in_start_stripe=46, ref_in_end_stripe=4d, num_lines=1, num_lines_to_be_processed=1
07/02/16 3:36:53: wait_q_next_ptr=0, num_lines_in_wait_q=0, immediate_already_tried=0, bbmHead=ffffffff, bbmTail=ffffffff
07/02/16 3:36:53: rmw: op=0, error=0, first_data_arm=0, last_data_arm=0
07/02/16 3:36:53: wt: countRowsNotUsingReadPeer=0
07/02/16 3:36:53: cmdId=199: cmd=11, cmdStat=0, num_sg_elements=3, status=1 [PCI_COMMAND]
07/02/16 3:36:53: mfa=cd821c00, mf=a0466400, mfSge=a0466428, bytesTransferred=0, next ffff
07/02/16 3:36:53: startTime=0, lines=a185a190, lineMap=0, activeRecoveryCount=0, lockPromotedByRec=0
07/02/16 3:36:53: ldbbmAlreadyTried=0, ldbbmIssueWriteAsWV=0
07/02/16 3:36:53: ld=0, ioFlags=0, start_block=2e839706, num_blocks=18, savedNumBlocks=18
07/02/16 3:36:53: group: 0199
07/02/16 3:36:53: start_row=1741cb, start_strip=5d072e, num_strips=1
07/02/16 3:36:53: ref_in_start_stripe=6, ref_in_end_stripe=1d, num_lines=1, num_lines_to_be_processed=1
07/02/16 3:36:53: wait_q_next_ptr=0, num_lines_in_wait_q=0, immediate_already_tried=0, bbmHead=ffffffff, bbmTail=ffffffff
07/02/16 3:36:53: rmw: op=0, error=0, first_data_arm=0, last_data_arm=0
07/02/16 3:36:53: wt: countRowsNotUsingReadPeer=0
07/02/16 3:36:53: cmdId=1ab: cmd=11, cmdStat=0, num_sg_elements=2, status=1 [PCI_COMMAND]
07/02/16 3:36:53: mfa=cd833800, mf=a046ac00, mfSge=a046ac28, bytesTransferred=0, next 2b7
07/02/16 3:36:53: startTime=0, lines=a18632b0, lineMap=0, activeRecoveryCount=0, lockPromotedByRec=0
07/02/16 3:36:53: ldbbmAlreadyTried=0, ldbbmIssueWriteAsWV=0
07/02/16 3:36:53: ld=0, ioFlags=0, start_block=2623167e, num_blocks=10, savedNumBlocks=10
07/02/16 3:36:53: group: 01ab
07/02/16 3:36:53: start_row=13118b, start_strip=4c462c, num_strips=2
07/02/16 3:36:53: ref_in_start_stripe=7e, ref_in_end_stripe=d, num_lines=1, num_lines_to_be_processed=2
07/02/16 3:36:53: wait_q_next_ptr=0, num_lines_in_wait_q=0, immediate_already_tried=0, bbmHead=ffffffff, bbmTail=ffffffff
07/02/16 3:36:53: rmw: op=0, error=0, first_data_arm=0, last_data_arm=0
07/02/16 3:36:53: wt: countRowsNotUsingReadPeer=0
07/02/16 3:36:53: cmdId=260: cmd=11, cmdStat=0, num_sg_elements=14, status=1 [PCI_COMMAND]
07/02/16 3:36:53: mfa=cd8e2c00, mf=a0498000, mfSge=a0498028, bytesTransferred=0, next ffff
07/02/16 3:36:53: startTime=0, lines=a18be600, lineMap=0, activeRecoveryCount=0, lockPromotedByRec=0
07/02/16 3:36:53: ldbbmAlreadyTried=0, ldbbmIssueWriteAsWV=0
07/02/16 3:36:53: ld=0, ioFlags=0, start_block=3a413936, num_blocks=a0, savedNumBlocks=a0
07/02/16 3:36:53: group: 0260
07/02/16 3:36:53: start_row=1d209c, start_strip=748272, num_strips=2
07/02/16 3:36:53: ref_in_start_stripe=36, ref_in_end_stripe=55, num_lines=1, num_lines_to_be_processed=2
07/02/16 3:36:53: wait_q_next_ptr=0, num_lines_in_wait_q=0, immediate_already_tried=0, bbmHead=ffffffff, bbmTail=ffffffff
07/02/16 3:36:53: rmw: op=0, error=0, first_data_arm=0, last_data_arm=0
07/02/16 3:36:53: wt: countRowsNotUsingReadPeer=0
07/02/16 3:36:53: Total Pending Commands = 4
[0]: fp=a00bee78, lr=a0c41918 - _MonTask+1a8
[1]: fp=a00bf0a0, lr=a0cc17e4 - mfiIdrIsr+124
[2]: fp=a00bf0b8, lr=e401e960 - dispatchIsrs+c4
[3]: fp=a00bf0e8, lr=e401e9f0 - external_IRQ+34
[4]: fp=a00bf100, lr=e401e074 - wrapper__External_IRQ+74
[5]: fp=a00bf150, lr=a0c68358 - TaskStartNext+12c
[6]: fp=a00bf190, lr=a0c519c0 - set_state+54
[7]: fp=a00bf278, lr=a0c536d0 - raid_task+864
[8]: fp=a00bffa0, lr=a0cbd384 - _main+aa4
[9]: fp=a00bfff8, lr=fe001d58 - __start+ce0
MonTask: line 3622 in file ../../raid/1078dma.c
UIC_ER=10000ac:5500063, UIC_MSR=0:40, MSR=21000, sp=a00bee78
MegaMon>
T0: LSI Logic ROC firmware
T0: Copyright (C) LSI Logic, 2004
T0: Firmware version 1.22.52-1909 built on Sep 21 2012 at 15:29:16
T0: pciInit: O_PCI_SERVICE = 00000005
T0: Initializing memory pool size=00300B24 bytes
T0: Press '!' within 3 seconds to enter debugger before INIT
T3: LogInit: Flushing events from previous boot
T3: EVT#30225-07/02/16 3:36:53: 15=Fatal firmware error: Driver detected possible FW hang, halting FW.
T3: EVT#30226-07/02/16 3:36:53: 15=Fatal firmware error: Line 3622 in ../../raid/1078dma.c