Openindiana oi_151a7. zpool with 15x 1TB raidz3 + 2x 8 disk 2TB raidz2. Many disk operations such as zpool status, creating a new iscsi volume (using sbdadm) etc hangs. Using iostat (-xne) I can determine that two block devices have errors, one is with high probability a cdrom, floppy or such, and the other is a 2TB disk (of one of the raidz2 vdevs the zpool consist of).iostat -xne0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0 1344  22 1366 c10t32d00.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0  52 385 437 c9t5000C50057CEF65Fd0The problem with c9t5000C50057CEF65Fd0 is that it's not listed in 'format'. 'format' further lists all 16 2TB disks, while iostat lists 17. One of the 2TB disks was replaced 2-3 months ago so Im thinking it's possible that iostat still think it's present although its not. Let me point out that I know I have 16 2TB disks physically in the host, and 16 (as 2x8  disk raidz2) in vdevs.This is a node for backup, so while a reboot is definitely possible it would be very convenient to be able to investigate it further.Iostat and format in it's entirety:                        extended device statistics       ---- errors --- r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b s/w h/w trn tot device0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0  69   0  69 c5t0d00.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c4t0d00.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c4t1d05.0    0.0   80.1    0.0  0.0  0.0    0.0    3.4   0   2   0   0   0   0 c4t2d07.0    4.0  112.1   96.1  0.0  0.0    0.0    2.7   0   3   0   0   0   0 c4t3d013.0    5.0  208.1  184.1  0.0  0.1    0.0    3.0   0   5   0   0   0   0 c4t4d00.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c7t50000393D8CAD353d0143.6  121.1  531.8  681.2  0.0  2.2    0.0    8.3   0  63   0   0   0   0 c7t50000393D8CAD313d0144.1  132.6  516.6  682.2  0.0  2.4    0.0    8.6   0  69   0   0   0   0 c7t50000393D8CAD243d0140.1  121.6  614.6  681.9  0.0  3.0    0.0   11.5   0  74   0   0   0   0 c7t50000393D8CAD217d0142.6  136.6  568.6  678.4  0.0  2.4    0.0    8.5   0  67   0   0   0   0  c7t50000393D8CAD2A7d0143.1  149.1  539.1  680.9  0.0  2.3    0.0    7.9   0  69   0   0   0   0 c7t50000393D8CAD32Bd0150.1  118.1  571.1  683.2  0.0  2.7    0.0   10.0   0  71   0   0   0   0 c7t50000393D8CAD29Bd0143.1  111.6  502.6  679.4  0.0  2.2    0.0    8.7   0  62   0   0   0   0 c7t50000393D8CAD1FBd0146.1  156.1  539.1  679.2  0.0  2.4    0.0    7.8   0  69   0   0   0   0 c 7t50000393D8CAD24Bd0146.1  136.6  533.8  680.7  0.0  2.4    0.0    8.5   0  67   0   0   0   0 c7t50000393D8CAD25Bd0143.1  129.6  529.1  679.9  0.0  2.1    0.0    7.8   0  60   0   0   0   0 c7t50000393D8CABD2Fd0149.1  123.6  511.6  682.9  0.0  2.5    0.0    9.3   0  73   0   0   0   0 c7t50000393D8CAD35Fd0142.6  140.6  540.6  680.4  0.0  2.4    0.0    8.3   0  69   0   0   0   0 c7t50000393D8CAD3BFd0143.6  112.6  555.4  683.4  0.0  2.8    0.0   10.8   0  70   0   0   0   0 c7t50000393D8CAD34Fd0140.1  143.6  514.1  680.9  0.0  2.5    0.0    8.9   0  71   0   0   0   0 c7t50000393D8CAD33Fd0138.6  137.6  624.9  682.4  0.0  2.3    0.0    8.4   0  70   0   0   0   0 c7t50000393D8CAD36Fd0123.1  227.1  494.3 1304.1  0.0  1.3    0.0    3.8   0  35   0   0   0   0 c9t5000C50056114317d0 54.0  122.6  336.2  426.5  0.0  0.3    0.0    1.5   0   8   0   0   0   0 c9t5000C500561161F3d051.0  104.1  319.7  427.3  0.0  0.5    0.0    3.2   0  12   0   0   0   0 c9t5000C5005611763Fd046.5  101.1  145.3  428.5  0.0  0.3    0.0    2.1   0   9   0   0   0   0 c9t5000C50055FFF9EBd038.5  128.6  198.6  427.0  0.0  0.3    0.0    1.6   0   9   0   0   0   0 c9t5000C50055FC6FD3d042.5   98.1  168.4  430.0  0.0  0.3    0.0    2.0   0  10   0   0   0   0 c9t5000C50055FFC403d053.5  143.6  336.5  423.3  0.0  0.3    0.0    1.5   0   9   0   0   0   0 c9t5000C5005611431Fd052.0  143.6  330.2  419.5  0.0  0.3    0.0    1.4   0   8   0   0   0   0 c9t5000C50056000BCBd036.0  148.6  270.4  426.5  0.0  0.2    0.0    1.2   0   7   0   0   0   0 c9t5000C50055FC7BF7d00.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c4t5d00.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0 1344  22 1366 c10t32d00.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   1  10  11 c10t33d0108.6  257.2  642.4 1297.8  0.0  1.0    0.0    2.7   0  29   0   0   0   0 c9t5000C50057CE4F2Bd0116.1  238.7  620.1 1295.3  0.0  1.1    0.0    3.2   0  34   0   0   0   0 c9t5000C50057CF51D7d00.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0  52 385 437 c9t5000C50057CEF65Fd0126.1  261.2  600.4 1294.3  0.0  1.0    0.0    2.5   0  32   0   0   0   0 c9t5000C50057CE4ED7d0119.6  238.2  653.7 1289.8  0.0  1.0    0.0    2.9   0  32   0   0   0   0 c9t5000C50057CE50A7d0112.1  220.6  484.6 1300.8  0.0  1.0    0.0    2.9   0  30   0   0   0   0 c9t5000C50057CEF10Bd0109.1  257.2  572.6 1294.6  0.0  1.1    0.0    3.0   0  31   0   0   0   0 c9t5000C50057CE37DFd0121.1  242.2  600.6 1293.8  0.0  1.1    0.0    3.0   0  33   0   0   0   0 c9t5000C50057CE4A6Bd0   0. c4t0d0 <HP-LOGICALVOLUME-2.74 cyl 17841 alt 2 hd 255 sec 63>      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@0,0   1. c4t1d0 <HP-LOGICALVOLUME-2.74 cyl 17841 alt 2 hd 255 sec 63>      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@1,0   2. c4t2d0 <HP-LOGICALVOLUME-2.74 cyl 17841 alt 2 hd 255 sec 63>      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@2,0   3. c4t3d0 <HP-LOGICAL VOLUME-2.74-68.33GB>      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@3,0   4. c4t4d0 <HP-LOGICAL VOLUME-2.74-68.33GB>      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@4,0   5. c4t5d0 <HP-LOGICAL VOLUME-2.74-68.33GB>      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@5,0   6. c7t50000393D8CABD2Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cabd2f,0   7. c7t50000393D8CAD1FBd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad1fb,0   8. c7t50000393D8CAD2A7d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad2a7,0   9. c7t50000393D8CAD3BFd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad3bf,0  10. c7t50000393D8CAD24Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad24b,0  11. c7t50000393D8CAD25Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad25b,0  12. c7t50000393D8CAD29Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad29b,0  13. c7t50000393D8CAD32Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad32b,0  14. c7t50000393D8CAD33Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad33f,0  15. c7t50000393D8CAD34Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad34f,0  16. c7t50000393D8CAD35Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad35f,0  17. c7t50000393D8CAD36Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad36f,0  18. c7t50000393D8CAD217d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad217,0  19. c7t50000393D8CAD243d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad243,0  20. c7t50000393D8CAD313d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad313,0  21. c7t50000393D8CAD353d0 <TOSHIBA-MK1001TRKB-0105 cyl 60798 alt 2 hd 255 sec 126>      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad353,0  22. c9t5000C50055FC6FD3d0 <SEAGATE-ST32000645SS-0004-1.82TB>      /scsi_vhci/disk@g5000c50055fc6fd3  23. c9t5000C50055FC7BF7d0 <SEAGATE-ST32000645SS-0004-1.82TB>      /scsi_vhci/disk@g5000c50055fc7bf7  24. c9t5000C50055FFC403d0 <SEAGATE-ST32000645SS-0004-1.82TB>      /scsi_vhci/disk@g5000c50055ffc403  25. c9t5000C50055FFF9EBd0 <SEAGATE-ST32000645SS-0004-1.82TB>      /scsi_vhci/disk@g5000c50055fff9eb  26. c9t5000C50057CE4A6Bd0 <SEAGATE-ST2000NM0023-0003-1.82TB>      /scsi_vhci/disk@g5000c50057ce4a6b  27. c9t5000C50057CE4ED7d0 <SEAGATE-ST2000NM0023-0003-1.82TB>      /scsi_vhci/disk@g5000c50057ce4ed7  28. c9t5000C50057CE4F2Bd0 <SEAGATE-ST2000NM0023-0003-1.82TB>      /scsi_vhci/disk@g5000c50057ce4f2b  29. c9t5000C50057CE37DFd0 <SEAGATE-ST2000NM0023-0003-1.82TB>      /scsi_vhci/disk@g5000c50057ce37df  30. c9t5000C50057CE50A7d0 <SEAGATE-ST2000NM0023-0003-1.82TB>      /scsi_vhci/disk@g5000c50057ce50a7  31. c9t5000C50057CEF10Bd0 <SEAGATE-ST2000NM0023-0003-1.82TB>      /scsi_vhci/disk@g5000c50057cef10b  32. c9t5000C50057CF51D7d0 <SEAGATE-ST2000NM0023-0003-1.82TB>      /scsi_vhci/disk@g5000c50057cf51d7  33. c9t5000C50056000BCBd0 <SEAGATE-ST32000645SS-0004-1.82TB>      /scsi_vhci/disk@g5000c50056000bcb  34. c9t5000C500561161F3d0 <SEAGATE-ST32000645SS-0004-1.82TB>      /scsi_vhci/disk@g5000c500561161f3  35. c9t5000C5005611763Fd0 <SEAGATE-ST32000645SS-0004-1.82TB>      /scsi_vhci/disk@g5000c5005611763f  36. c9t5000C5005611431Fd0 <SEAGATE-ST32000645SS-0004-1.82TB>      /scsi_vhci/disk@g5000c5005611431f  37. c9t5000C50056114317d0 <SEAGATE-ST32000645SS-0004-1.82TB>      /scsi_vhci/disk@g5000c50056114317  38. c10t32d0 <drive type unknown>      /iscsi/disk@0000iqn.1984-05.com.dell%3Apowervault.md3200i.6782bcb000311a0a000000004df9adc70001,0  39. c10t33d0 <drive type unknown>      /iscsi/disk@0000iqn.1984-05.com.dell%3Apowervault.md3200i.6782bcb000311a0a000000004df9adc70001,31
zpool status hangs forever
zfs;openindiana;sasl