Project

General

Profile

Bug #18510

iSCSI hangs after deploying VM from template (same datastore)

Added by Jan Eagleman over 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Important
Assignee:
Alexander Motin
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

This is the second time I experience this issue, once I want to clone a virtual machine from a template in vCenter (template on same datastore as target) the entire iscsi share begins to hang, including all management interfaces (esxi and venter not loading).

FreeNAS is also a VM on the same machine, reserved all memory and using an IBM M1015 in IT mode.

This are some of the logs in esxi (vmkernel.log), right after vmware hangs and all vms loose connection:

https://gist.github.com/Eagleman7/102bf7eae5cd6f187394f0f1ebc2fa33

And this is my pool: 4x 3TB wd red and a intel 750 400GB as slog and 300GB L2ARC, 64GB memory.

pool: currahee                                                                                                                    
state: ONLINE
scan: scrub repaired 0 in 3h59m with 0 errors on Tue Oct 11 03:59:05 2016
config:
NAME                                            STATE     READ WRITE CKSUM                                                  
currahee ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/2a71ea82-edc6-11e5-8db4-0050569a3b4a ONLINE 0 0 0
gptid/2b366928-edc6-11e5-8db4-0050569a3b4a ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/2bf79e44-edc6-11e5-8db4-0050569a3b4a ONLINE 0 0 0
gptid/2cb76576-edc6-11e5-8db4-0050569a3b4a ONLINE 0 0 0
logs
gptid/a453ec7b-84d4-11e6-8934-0050569a3b4a ONLINE 0 0 0
cache
gptid/a5c67d57-84d5-11e6-8934-0050569a3b4a ONLINE 0 0 0

errors: No known data errors

[root@freenas ~]# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
currahee 5.44T 292G 5.15T - 20% 5% 1.00x ONLINE /mnt

Then when I want to stop iscsi on freenas I get this error:

/var/log/messages
Oct 26 18:14:53 freenas notifier: ctld already running? (pid=2516).
Oct 26 18:14:55 freenas notifier: Stopping ctld.
Oct 26 18:14:55 freenas WARNING: 192.168.11.10 (iqn.1998-01.com.vmware:esxi-761aaf17): waiting for CTL to terminate 336 tasks
Oct 26 18:15:42 freenas notifier: Starting ctld.
Oct 26 18:15:42 freenas notifier: ctld: bind(2) failed for 192.168.11.11:3260: Address already in use
Oct 26 18:15:42 freenas ctld75163: bind(2) failed for 192.168.11.11:3260: Address already in use
Oct 26 18:15:42 freenas notifier: ctld: failed to apply configuration; exiting
Oct 26 18:15:42 freenas ctld75163: failed to apply configuration; exiting
Oct 26 18:15:42 freenas root: /etc/rc.d/ctld: WARNING: failed to start ctld
Oct 26 18:15:42 freenas notifier: /etc/rc.d/ctld: WARNING: failed to start ctld


Related issues

Related to FreeNAS - Bug #19892: connection error; dropping connectionClosed: Cannot reproduce2016-12-29

History

#1 Updated by Jan Eagleman over 4 years ago

Work around is to restart FreeNAS, so iSCSI is able to start again. However I would like to know why the iSCSI share stops repsonding.

#2 Updated by Josh Paetzel over 4 years ago

  • Status changed from Unscreened to 15
  • Assignee set to Josh Paetzel
  • Priority changed from No priority to Important

From the FreeNAS side could you get a tcpdump when this happens?

tcpdump -ni <ethernet port> host <vmware host> and port 3260

Also ctladm dumpooa would be usefui after iscsi has hung.

It would also be interesting to see /var/log/messages on the FreeNAS side.

#3 Updated by Jan Eagleman over 4 years ago

I have already included the /var/log/messages log from freenas:

/var/log/messages
Oct 26 18:14:53 freenas notifier: ctld already running? (pid=2516).
Oct 26 18:14:55 freenas notifier: Stopping ctld.
Oct 26 18:14:55 freenas WARNING: 192.168.11.10 (iqn.1998-01.com.vmware:esxi-761aaf17): waiting for CTL to terminate 336 tasks
Oct 26 18:15:42 freenas notifier: Starting ctld.
Oct 26 18:15:42 freenas notifier: ctld: bind(2) failed for 192.168.11.11:3260: Address already in use
Oct 26 18:15:42 freenas ctld75163: bind(2) failed for 192.168.11.11:3260: Address already in use
Oct 26 18:15:42 freenas notifier: ctld: failed to apply configuration; exiting
Oct 26 18:15:42 freenas ctld75163: failed to apply configuration; exiting
Oct 26 18:15:42 freenas root: /etc/rc.d/ctld: WARNING: failed to start ctld
Oct 26 18:15:42 freenas notifier: /etc/rc.d/ctld: WARNING: failed to start ctld

I will try to make it crash again tonight, and run the rest of the commands.

#4 Updated by Vaibhav Chauhan over 4 years ago

BRB: can you also provide the following info as requested by Josh.

tcpdump -ni <ethernet port> host <vmware host> and port 3260

Also ctladm dumpooa would be usefui after iscsi has hung.

#5 Updated by Jan Eagleman over 4 years ago

I tried to clone another VM from a template and it again began to hang, here is the info you requested:

[root@freenas ~]# ctladm dumpooa                                                
Dumping OOA queues                                                              
LUN 0 tag 0x13492950 RTR: EXTENDED COPY. CDB:  (230981 ms)                      
LUN 0 tag 0x13492951 RTR: EXTENDED COPY. CDB:  (230981 ms)                      
LUN 0 tag 0x13492952 RTR: EXTENDED COPY. CDB:  (230981 ms)                      
OOA queues dump done        

After several minutes have passed I ran it again:

[root@freenas ~]# ctladm dumpooa                                                
Dumping OOA queues                                                              
LUN 0 tag 0x13492950 RTR: EXTENDED COPY. CDB:  (682800 ms)                      
LUN 0 tag 0x13492951 RTR: EXTENDED COPY. CDB:  (682800 ms)                      
LUN 0 tag 0x13492952 RTR: EXTENDED COPY. CDB:  (682800 ms)                      
OOA queues dump done   

tcpdump from dedicated iscsi interface, only esxi has a mount on this network:

[root@freenas] ~# tcpdump -ni vmx1 port 3260
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx1, link-type EN10MB (Ethernet), capture size 65535 bytes
14:08:59.908881 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1272344097:1272344145, ack 3531430997, win 25344, options [nop,nop,TS val 2587649842 ecr 397330135], length 48
14:08:59.908947 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1:49, ack 48, win 514, options [nop,nop,TS val 397330336 ecr 2587649842], length 48
14:09:00.014316 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 49, win 25344, options [nop,nop,TS val 2587649947 ecr 397330336], length 0
14:09:01.908420 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 48:96, ack 49, win 25344, options [nop,nop,TS val 2587651841 ecr 397330336], length 48
14:09:01.908530 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 49:97, ack 96, win 514, options [nop,nop,TS val 397330536 ecr 2587651841], length 48
14:09:02.014289 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 97, win 25344, options [nop,nop,TS val 2587651947 ecr 397330536], length 0
14:09:03.941136 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 96:144, ack 97, win 25344, options [nop,nop,TS val 2587653874 ecr 397330536], length 48
14:09:03.941235 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 97:145, ack 144, win 514, options [nop,nop,TS val 397330739 ecr 2587653874], length 48
14:09:04.047332 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 145, win 25344, options [nop,nop,TS val 2587653980 ecr 397330739], length 0
14:09:04.817486 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 145:193, ack 144, win 514, options [nop,nop,TS val 397330827 ecr 2587653980], length 48
14:09:04.923316 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 193, win 25344, options [nop,nop,TS val 2587654856 ecr 397330827], length 0
14:09:05.972171 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 144:192, ack 193, win 25344, options [nop,nop,TS val 2587655905 ecr 397330827], length 48
14:09:05.972354 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 193:241, ack 192, win 514, options [nop,nop,TS val 397330943 ecr 2587655905], length 48
14:09:06.074109 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 241, win 25344, options [nop,nop,TS val 2587656007 ecr 397330943], length 0
14:09:07.981188 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 192:240, ack 241, win 25344, options [nop,nop,TS val 2587657914 ecr 397330943], length 48
14:09:07.981364 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 241:289, ack 240, win 514, options [nop,nop,TS val 397331143 ecr 2587657914], length 48
14:09:08.086322 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 289, win 25344, options [nop,nop,TS val 2587658019 ecr 397331143], length 0
14:09:10.017184 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 240:288, ack 289, win 25344, options [nop,nop,TS val 2587659950 ecr 397331143], length 48
14:09:10.017267 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 289:337, ack 288, win 514, options [nop,nop,TS val 397331347 ecr 2587659950], length 48
14:09:10.123346 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 337, win 25344, options [nop,nop,TS val 2587660056 ecr 397331347], length 0
14:09:12.040118 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 288:336, ack 337, win 25344, options [nop,nop,TS val 2587661973 ecr 397331347], length 48
14:09:12.040254 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 337:385, ack 336, win 514, options [nop,nop,TS val 397331549 ecr 2587661973], length 48
14:09:12.142048 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 385, win 25344, options [nop,nop,TS val 2587662075 ecr 397331549], length 0
14:09:14.047695 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 336:384, ack 385, win 25344, options [nop,nop,TS val 2587663980 ecr 397331549], length 48
14:09:14.047826 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 385:433, ack 384, win 514, options [nop,nop,TS val 397331750 ecr 2587663980], length 48
14:09:14.153351 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 433, win 25344, options [nop,nop,TS val 2587664086 ecr 397331750], length 0
14:09:14.818441 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 433:481, ack 384, win 514, options [nop,nop,TS val 397331827 ecr 2587664086], length 48
14:09:14.920064 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 481, win 25344, options [nop,nop,TS val 2587664853 ecr 397331827], length 0
14:09:16.092106 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 384:432, ack 481, win 25344, options [nop,nop,TS val 2587666025 ecr 397331827], length 48
14:09:16.092219 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 481:529, ack 432, win 514, options [nop,nop,TS val 397331955 ecr 2587666025], length 48
14:09:16.198315 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 529, win 25344, options [nop,nop,TS val 2587666131 ecr 397331955], length 0
14:09:18.100820 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 432:480, ack 529, win 25344, options [nop,nop,TS val 2587668034 ecr 397331955], length 48
14:09:18.100911 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 529:577, ack 480, win 514, options [nop,nop,TS val 397332155 ecr 2587668034], length 48
14:09:18.206279 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 577, win 25344, options [nop,nop,TS val 2587668139 ecr 397332155], length 0
14:09:20.100095 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 480:528, ack 577, win 25344, options [nop,nop,TS val 2587670033 ecr 397332155], length 48
14:09:20.100246 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 577:625, ack 528, win 514, options [nop,nop,TS val 397332355 ecr 2587670033], length 48
14:09:20.206274 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 625, win 25344, options [nop,nop,TS val 2587670139 ecr 397332355], length 0
14:09:22.135400 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 528:576, ack 625, win 25344, options [nop,nop,TS val 2587672068 ecr 397332355], length 48
14:09:22.135524 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 625:673, ack 576, win 514, options [nop,nop,TS val 397332559 ecr 2587672068], length 48
14:09:22.241351 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 673, win 25344, options [nop,nop,TS val 2587672174 ecr 397332559], length 0
14:09:24.140842 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 576:624, ack 673, win 25344, options [nop,nop,TS val 2587674074 ecr 397332559], length 48
14:09:24.140966 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 673:721, ack 624, win 514, options [nop,nop,TS val 397332759 ecr 2587674074], length 48
14:09:24.246350 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 721, win 25344, options [nop,nop,TS val 2587674179 ecr 397332759], length 0
14:09:24.820302 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 721:769, ack 624, win 514, options [nop,nop,TS val 397332827 ecr 2587674179], length 48
14:09:24.926291 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 769, win 25344, options [nop,nop,TS val 2587674859 ecr 397332827], length 0
14:09:26.152198 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 624:672, ack 769, win 25344, options [nop,nop,TS val 2587676085 ecr 397332827], length 48
14:09:26.152300 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 769:817, ack 672, win 514, options [nop,nop,TS val 397332961 ecr 2587676085], length 48
14:09:26.258284 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 817, win 25344, options [nop,nop,TS val 2587676191 ecr 397332961], length 0
14:09:28.175096 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 672:720, ack 817, win 25344, options [nop,nop,TS val 2587678108 ecr 397332961], length 48
14:09:28.175192 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 817:865, ack 720, win 514, options [nop,nop,TS val 397333163 ecr 2587678108], length 48
14:09:28.281276 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 865, win 25344, options [nop,nop,TS val 2587678214 ecr 397333163], length 0
14:09:30.185180 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 720:768, ack 865, win 25344, options [nop,nop,TS val 2587680118 ecr 397333163], length 48
14:09:30.185250 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 865:913, ack 768, win 514, options [nop,nop,TS val 397333364 ecr 2587680118], length 48
14:09:30.285940 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 913, win 25344, options [nop,nop,TS val 2587680219 ecr 397333364], length 0
14:09:32.237185 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 768:816, ack 913, win 25344, options [nop,nop,TS val 2587682170 ecr 397333364], length 48
14:09:32.237287 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 913:961, ack 816, win 514, options [nop,nop,TS val 397333569 ecr 2587682170], length 48
14:09:32.340696 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 961, win 25344, options [nop,nop,TS val 2587682273 ecr 397333569], length 0
14:09:34.267195 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 816:864, ack 961, win 25344, options [nop,nop,TS val 2587684200 ecr 397333569], length 48
14:09:34.267345 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 961:1009, ack 864, win 514, options [nop,nop,TS val 397333772 ecr 2587684200], length 48
14:09:34.373309 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1009, win 25344, options [nop,nop,TS val 2587684306 ecr 397333772], length 0
14:09:34.821478 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1009:1057, ack 864, win 514, options [nop,nop,TS val 397333827 ecr 2587684306], length 48
14:09:34.922047 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1057, win 25344, options [nop,nop,TS val 2587684855 ecr 397333827], length 0
14:09:36.268429 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 864:912, ack 1057, win 25344, options [nop,nop,TS val 2587686201 ecr 397333827], length 48
14:09:36.268557 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1057:1105, ack 912, win 514, options [nop,nop,TS val 397333972 ecr 2587686201], length 48
14:09:36.374318 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1105, win 25344, options [nop,nop,TS val 2587686307 ecr 397333972], length 0
14:09:38.300580 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 912:960, ack 1105, win 25344, options [nop,nop,TS val 2587688233 ecr 397333972], length 48
14:09:38.300667 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1105:1153, ack 960, win 514, options [nop,nop,TS val 397334175 ecr 2587688233], length 48
14:09:38.406326 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1153, win 25344, options [nop,nop,TS val 2587688339 ecr 397334175], length 0
14:09:40.316198 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 960:1008, ack 1153, win 25344, options [nop,nop,TS val 2587690249 ecr 397334175], length 48
14:09:40.316296 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1153:1201, ack 1008, win 514, options [nop,nop,TS val 397334377 ecr 2587690249], length 48
14:09:40.422362 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1201, win 25344, options [nop,nop,TS val 2587690355 ecr 397334377], length 0
14:09:42.320084 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1008:1056, ack 1201, win 25344, options [nop,nop,TS val 2587692253 ecr 397334377], length 48
14:09:42.320243 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1201:1249, ack 1056, win 514, options [nop,nop,TS val 397334577 ecr 2587692253], length 48
14:09:42.426315 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1249, win 25344, options [nop,nop,TS val 2587692359 ecr 397334577], length 0
14:09:44.392111 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1056:1104, ack 1249, win 25344, options [nop,nop,TS val 2587694325 ecr 397334577], length 48
14:09:44.392250 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1249:1297, ack 1104, win 514, options [nop,nop,TS val 397334785 ecr 2587694325], length 48
14:09:44.494031 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1297, win 25344, options [nop,nop,TS val 2587694427 ecr 397334785], length 0
14:09:44.822471 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1297:1345, ack 1104, win 514, options [nop,nop,TS val 397334828 ecr 2587694427], length 48
14:09:44.928280 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1345, win 25344, options [nop,nop,TS val 2587694861 ecr 397334828], length 0
14:09:46.438499 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1104:1152, ack 1345, win 25344, options [nop,nop,TS val 2587696371 ecr 397334828], length 48
14:09:46.438596 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1345:1393, ack 1152, win 514, options [nop,nop,TS val 397334989 ecr 2587696371], length 48
14:09:46.544290 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1393, win 25344, options [nop,nop,TS val 2587696477 ecr 397334989], length 0
14:09:48.468547 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1152:1200, ack 1393, win 25344, options [nop,nop,TS val 2587698401 ecr 397334989], length 48
14:09:48.468682 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1393:1441, ack 1200, win 514, options [nop,nop,TS val 397335192 ecr 2587698401], length 48
14:09:48.568508 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1441, win 25344, options [nop,nop,TS val 2587698501 ecr 397335192], length 0
14:09:50.515570 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1200:1248, ack 1441, win 25344, options [nop,nop,TS val 2587700448 ecr 397335192], length 48
14:09:50.515650 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1441:1489, ack 1248, win 514, options [nop,nop,TS val 397335397 ecr 2587700448], length 48
14:09:50.617108 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1489, win 25344, options [nop,nop,TS val 2587700550 ecr 397335397], length 0
14:09:52.548113 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1248:1296, ack 1489, win 25344, options [nop,nop,TS val 2587702481 ecr 397335397], length 48
14:09:52.548198 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1489:1537, ack 1296, win 514, options [nop,nop,TS val 397335600 ecr 2587702481], length 48
14:09:52.654372 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1537, win 25344, options [nop,nop,TS val 2587702587 ecr 397335600], length 0
14:09:54.580555 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1296:1344, ack 1537, win 25344, options [nop,nop,TS val 2587704513 ecr 397335600], length 48
14:09:54.580649 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1537:1585, ack 1344, win 514, options [nop,nop,TS val 397335803 ecr 2587704513], length 48
14:09:54.682822 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1585, win 25344, options [nop,nop,TS val 2587704616 ecr 397335803], length 0
14:09:54.824444 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1585:1633, ack 1344, win 514, options [nop,nop,TS val 397335828 ecr 2587704616], length 48
14:09:54.930303 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1633, win 25344, options [nop,nop,TS val 2587704863 ecr 397335828], length 0
14:09:56.610091 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1344:1392, ack 1633, win 25344, options [nop,nop,TS val 2587706543 ecr 397335828], length 48
14:09:56.610195 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1633:1681, ack 1392, win 514, options [nop,nop,TS val 397336006 ecr 2587706543], length 48
14:09:56.716304 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1681, win 25344, options [nop,nop,TS val 2587706649 ecr 397336006], length 0
14:09:58.620410 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1392:1440, ack 1681, win 25344, options [nop,nop,TS val 2587708553 ecr 397336006], length 48
14:09:58.620544 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1681:1729, ack 1440, win 514, options [nop,nop,TS val 397336207 ecr 2587708553], length 48
14:09:58.726313 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1729, win 25344, options [nop,nop,TS val 2587708659 ecr 397336207], length 0
14:10:00.644083 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1440:1488, ack 1729, win 25344, options [nop,nop,TS val 2587710577 ecr 397336207], length 48
14:10:00.644182 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1729:1777, ack 1488, win 514, options [nop,nop,TS val 397336410 ecr 2587710577], length 48
14:10:00.744032 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1777, win 25344, options [nop,nop,TS val 2587710677 ecr 397336410], length 0
14:10:02.662195 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1488:1536, ack 1777, win 25344, options [nop,nop,TS val 2587712595 ecr 397336410], length 48
14:10:02.662349 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1777:1825, ack 1536, win 514, options [nop,nop,TS val 397336611 ecr 2587712595], length 48
14:10:02.768347 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1825, win 25344, options [nop,nop,TS val 2587712701 ecr 397336611], length 0
14:10:04.693188 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1536:1584, ack 1825, win 25344, options [nop,nop,TS val 2587714626 ecr 397336611], length 48
14:10:04.693265 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1825:1873, ack 1584, win 514, options [nop,nop,TS val 397336815 ecr 2587714626], length 48
14:10:04.799317 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1873, win 25344, options [nop,nop,TS val 2587714732 ecr 397336815], length 0
14:10:04.826449 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1873:1921, ack 1584, win 514, options [nop,nop,TS val 397336828 ecr 2587714732], length 48
14:10:04.928064 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1921, win 25344, options [nop,nop,TS val 2587714861 ecr 397336828], length 0
14:10:06.701069 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1584:1632, ack 1921, win 25344, options [nop,nop,TS val 2587716634 ecr 397336828], length 48
14:10:06.701137 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1921:1969, ack 1632, win 514, options [nop,nop,TS val 397337015 ecr 2587716634], length 48
14:10:06.801026 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 1969, win 25344, options [nop,nop,TS val 2587716734 ecr 397337015], length 0
14:10:08.701067 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1632:1680, ack 1969, win 25344, options [nop,nop,TS val 2587718634 ecr 397337015], length 48
14:10:08.701144 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 1969:2017, ack 1680, win 514, options [nop,nop,TS val 397337215 ecr 2587718634], length 48
14:10:08.801022 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2017, win 25344, options [nop,nop,TS val 2587718734 ecr 397337215], length 0
14:10:10.704677 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1680:1728, ack 2017, win 25344, options [nop,nop,TS val 2587720637 ecr 397337215], length 48
14:10:10.704760 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2017:2065, ack 1728, win 514, options [nop,nop,TS val 397337416 ecr 2587720637], length 48
14:10:10.810337 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2065, win 25344, options [nop,nop,TS val 2587720743 ecr 397337416], length 0
14:10:12.745081 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1728:1776, ack 2065, win 25344, options [nop,nop,TS val 2587722678 ecr 397337416], length 48
14:10:12.745210 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2065:2113, ack 1776, win 514, options [nop,nop,TS val 397337620 ecr 2587722678], length 48
14:10:12.851307 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2113, win 25344, options [nop,nop,TS val 2587722784 ecr 397337620], length 0
14:10:14.773373 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1776:1824, ack 2113, win 25344, options [nop,nop,TS val 2587724706 ecr 397337620], length 48
14:10:14.773507 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2113:2161, ack 1824, win 514, options [nop,nop,TS val 397337823 ecr 2587724706], length 48
14:10:14.828020 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2161:2209, ack 1824, win 514, options [nop,nop,TS val 397337828 ecr 2587724706], length 48
14:10:14.828057 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2209, win 25343, options [nop,nop,TS val 2587724761 ecr 397337823], length 0
14:10:16.801234 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1824:1872, ack 2209, win 25344, options [nop,nop,TS val 2587726734 ecr 397337823], length 48
14:10:16.801347 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2209:2257, ack 1872, win 514, options [nop,nop,TS val 397338025 ecr 2587726734], length 48
14:10:16.905246 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2257, win 25344, options [nop,nop,TS val 2587726838 ecr 397338025], length 0
14:10:18.805153 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1872:1920, ack 2257, win 25344, options [nop,nop,TS val 2587728738 ecr 397338025], length 48
14:10:18.805265 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2257:2305, ack 1920, win 514, options [nop,nop,TS val 397338226 ecr 2587728738], length 48
14:10:18.911306 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2305, win 25344, options [nop,nop,TS val 2587728844 ecr 397338226], length 0
14:10:20.871086 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1920:1968, ack 2305, win 25344, options [nop,nop,TS val 2587730804 ecr 397338226], length 48
14:10:20.871228 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2305:2353, ack 1968, win 514, options [nop,nop,TS val 397338432 ecr 2587730804], length 48
14:10:20.977324 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2353, win 25344, options [nop,nop,TS val 2587730910 ecr 397338432], length 0
14:10:22.896072 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 1968:2016, ack 2353, win 25344, options [nop,nop,TS val 2587732829 ecr 397338432], length 48
14:10:22.896165 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2353:2401, ack 2016, win 514, options [nop,nop,TS val 397338635 ecr 2587732829], length 48
14:10:22.998103 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2401, win 25344, options [nop,nop,TS val 2587732931 ecr 397338635], length 0
14:10:24.829868 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2401:2449, ack 2016, win 514, options [nop,nop,TS val 397338828 ecr 2587732931], length 48
14:10:24.935341 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2449, win 25344, options [nop,nop,TS val 2587734868 ecr 397338828], length 0
14:10:25.910798 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 2016:2064, ack 2449, win 25344, options [nop,nop,TS val 2587735843 ecr 397338828], length 48
14:10:25.910924 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2449:2497, ack 2064, win 514, options [nop,nop,TS val 397338936 ecr 2587735843], length 48
14:10:26.016353 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2497, win 25344, options [nop,nop,TS val 2587735949 ecr 397338936], length 0
14:10:27.952085 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 2064:2112, ack 2497, win 25344, options [nop,nop,TS val 2587737885 ecr 397338936], length 48
14:10:27.953101 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2497:2545, ack 2112, win 514, options [nop,nop,TS val 397339140 ecr 2587737885], length 48
14:10:28.059299 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2545, win 25344, options [nop,nop,TS val 2587737992 ecr 397339140], length 0
14:10:29.997370 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 2112:2160, ack 2545, win 25344, options [nop,nop,TS val 2587739930 ecr 397339140], length 48
14:10:29.997502 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2545:2593, ack 2160, win 514, options [nop,nop,TS val 397339345 ecr 2587739930], length 48
14:10:30.103316 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2593, win 25344, options [nop,nop,TS val 2587740036 ecr 397339345], length 0
14:10:32.017591 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 2160:2208, ack 2593, win 25344, options [nop,nop,TS val 2587741950 ecr 397339345], length 48
14:10:32.017747 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2593:2641, ack 2208, win 514, options [nop,nop,TS val 397339547 ecr 2587741950], length 48
14:10:32.123291 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2641, win 25344, options [nop,nop,TS val 2587742056 ecr 397339547], length 0
14:10:34.040181 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 2208:2256, ack 2641, win 25344, options [nop,nop,TS val 2587743973 ecr 397339547], length 48
14:10:34.040282 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2641:2689, ack 2256, win 514, options [nop,nop,TS val 397339749 ecr 2587743973], length 48
14:10:34.140084 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2689, win 25344, options [nop,nop,TS val 2587744073 ecr 397339749], length 0
14:10:34.830299 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2689:2737, ack 2256, win 514, options [nop,nop,TS val 397339828 ecr 2587744073], length 48
14:10:34.936345 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2737, win 25344, options [nop,nop,TS val 2587744869 ecr 397339828], length 0
14:10:36.052174 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 2256:2304, ack 2737, win 25344, options [nop,nop,TS val 2587745985 ecr 397339828], length 48
14:10:36.052348 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2737:2785, ack 2304, win 514, options [nop,nop,TS val 397339951 ecr 2587745985], length 48
14:10:36.158335 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2785, win 25344, options [nop,nop,TS val 2587746091 ecr 397339951], length 0
14:10:38.066396 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 2304:2352, ack 2785, win 25344, options [nop,nop,TS val 2587747999 ecr 397339951], length 48
14:10:38.066504 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2785:2833, ack 2352, win 514, options [nop,nop,TS val 397340152 ecr 2587747999], length 48
14:10:38.172274 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2833, win 25344, options [nop,nop,TS val 2587748105 ecr 397340152], length 0
14:10:40.099149 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [P.], seq 2352:2400, ack 2833, win 25344, options [nop,nop,TS val 2587750032 ecr 397340152], length 48
14:10:40.099235 IP 192.168.11.10.22846 > 192.168.11.11.3260: Flags [P.], seq 2833:2881, ack 2400, win 514, options [nop,nop,TS val 397340355 ecr 2587750032], length 48
14:10:40.199774 IP 192.168.11.11.3260 > 192.168.11.10.22846: Flags [.], ack 2881, win 25344, options [nop,nop,TS val 2587750133 ecr 397340355], length 0
^C
169 packets captured
176 packets received by filter
0 packets dropped by kernel

#6 Updated by Jan Eagleman over 4 years ago

Any update on this?

#7 Updated by Vaibhav Chauhan over 4 years ago

  • Target version set to 9.10.2

@Jan Eagleman can you provide us the debug as well, as that would be really helpful.

#8 Updated by Jan Eagleman over 4 years ago

  • Target version deleted (9.10.2)

Vaibhav Chauhan wrote:

@Jan Eagleman can you provide us the debug as well, as that would be really helpful.

What debug?

#9 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Target version set to 9.10.2-U1

#10 Updated by Jan Eagleman over 4 years ago

  • Seen in changed from 9.10.1-U2 to 9.10.1-U4

In Freenas 9.10.1-U4 (latest) and VMWare vSphere 6.5 (latest) I am still having this issue, before cloning a template into a VM I made sure to shutdown all the other vms that reside on the freenas datastore, even after doing that and cloning a template in a VM it began to hang:

Dumping OOA queues
LUN 0 tag 0x1b72ed0 RTR: EXTENDED COPY. CDB:  (205977 ms)
LUN 0 tag 0x1b72ed1 RTR: EXTENDED COPY. CDB:  (205977 ms)
LUN 0 tag 0x1b72ed2 RTR: EXTENDED COPY. CDB:  (205977 ms)
LUN 0 tag 0x0001 RTR: READ(10). CDB: 28 00 25 80 98 00 00 08 00 00  (205977 ms)
LUN 0 tag 0x0002 RTR: READ(10). CDB: 28 00 25 80 a0 00 00 08 00 00  (205977 ms)
LUN 0 tag 0x0003 BLOCKED: READ(10). CDB: 28 00 25 80 a8 00 00 08 00 00  (205977 ms)
LUN 0 tag 0x0004 RTR: READ(10). CDB: 28 00 25 80 90 00 00 08 00 00  (205977 ms)
LUN 0 tag 0x0005 BLOCKED: READ(10). CDB: 28 00 25 80 b0 00 00 08 00 00  (205977 ms)
LUN 0 tag 0x0006 BLOCKED: READ(10). CDB: 28 00 25 80 b8 00 00 08 00 00  (205977 ms)
LUN 0 tag 0x0007 BLOCKED: READ(10). CDB: 28 00 25 80 c0 00 00 08 00 00  (205977 ms)
LUN 0 tag 0x0008 BLOCKED: READ(10). CDB: 28 00 25 80 c8 00 00 08 00 00  (205977 ms)
LUN 0 tag 0x0009 BLOCKED: READ(10). CDB: 28 00 25 80 d0 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x000a BLOCKED: READ(10). CDB: 28 00 25 80 d8 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x000b BLOCKED: READ(10). CDB: 28 00 25 80 e0 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x000c BLOCKED: READ(10). CDB: 28 00 25 80 e8 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x1b72ed3 RTR: EXTENDED COPY. CDB:  (205976 ms)
LUN 0 tag 0x1b72ed4 RTR: EXTENDED COPY. CDB:  (205976 ms)
LUN 0 tag 0x1b72ed5 RTR: EXTENDED COPY. CDB:  (205976 ms)
LUN 0 tag 0x1b72ed6 RTR: EXTENDED COPY. CDB:  (205976 ms)
LUN 0 tag 0x1b72ed7 RTR: EXTENDED COPY. CDB:  (205976 ms)
LUN 0 tag 0x000d BLOCKED: READ(10). CDB: 28 00 25 80 f0 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x000e BLOCKED: READ(10). CDB: 28 00 25 80 f8 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x000f BLOCKED: READ(10). CDB: 28 00 25 81 00 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x0010 BLOCKED: READ(10). CDB: 28 00 25 81 08 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x0011 BLOCKED: READ(10). CDB: 28 00 25 81 10 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x0012 BLOCKED: READ(10). CDB: 28 00 25 81 18 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x0013 BLOCKED: READ(10). CDB: 28 00 25 81 20 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x0014 BLOCKED: READ(10). CDB: 28 00 25 81 28 00 00 08 00 00  (205976 ms)
LUN 0 tag 0x0015 BLOCKED: READ(10). CDB: 28 00 25 81 30 00 00 08 00 00  (205975 ms)
LUN 0 tag 0x0016 BLOCKED: READ(10). CDB: 28 00 25 81 38 00 00 08 00 00  (205975 ms)
LUN 0 tag 0x0017 BLOCKED: READ(10). CDB: 28 00 25 81 40 00 00 08 00 00  (205975 ms)
LUN 0 tag 0x0018 BLOCKED: READ(10). CDB: 28 00 25 81 48 00 00 08 00 00  (205975 ms)
OOA queues dump done
[root@freenas] ~# tcpdump -ni vmx1 port 3260
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx1, link-type EN10MB (Ethernet), capture size 65535 bytes
12:40:16.585906 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 1057146012:1057146060, ack 627033020, win 23296, options [nop,nop,TS val 674946012 ecr 57612792], length 48
12:40:16.586094 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 1:49, ack 48, win 514, options [nop,nop,TS val 57612995 ecr 674946012], length 48
12:40:16.690225 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 49, win 23296, options [nop,nop,TS val 674946116 ecr 57612995], length 0
12:40:17.172363 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 49:97, ack 48, win 514, options [nop,nop,TS val 57613054 ecr 674946116], length 48
12:40:17.271789 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 97, win 23296, options [nop,nop,TS val 674946698 ecr 57613054], length 0
12:40:18.682932 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 48:96, ack 97, win 23296, options [nop,nop,TS val 674948109 ecr 57613054], length 48
12:40:18.683129 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 97:145, ack 96, win 514, options [nop,nop,TS val 57613204 ecr 674948109], length 48
12:40:18.784833 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 145, win 23296, options [nop,nop,TS val 674948211 ecr 57613204], length 0
12:40:20.723943 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 96:144, ack 145, win 23296, options [nop,nop,TS val 674950150 ecr 57613204], length 48
12:40:20.724130 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 145:193, ack 144, win 514, options [nop,nop,TS val 57613409 ecr 674950150], length 48
12:40:20.825855 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 193, win 23296, options [nop,nop,TS val 674950252 ecr 57613409], length 0
12:40:22.758109 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 144:192, ack 193, win 23296, options [nop,nop,TS val 674952184 ecr 57613409], length 48
12:40:22.758336 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 193:241, ack 192, win 514, options [nop,nop,TS val 57613612 ecr 674952184], length 48
12:40:22.864063 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 241, win 23296, options [nop,nop,TS val 674952290 ecr 57613612], length 0
12:40:24.781083 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 192:240, ack 241, win 23296, options [nop,nop,TS val 674954207 ecr 57613612], length 48
12:40:24.781317 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 241:289, ack 240, win 514, options [nop,nop,TS val 57613815 ecr 674954207], length 48
12:40:24.887108 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 289, win 23296, options [nop,nop,TS val 674954313 ecr 57613815], length 0
12:40:26.846964 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 240:288, ack 289, win 23296, options [nop,nop,TS val 674956273 ecr 57613815], length 48
12:40:26.847199 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 289:337, ack 288, win 514, options [nop,nop,TS val 57614021 ecr 674956273], length 48
12:40:26.953122 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 337, win 23296, options [nop,nop,TS val 674956379 ecr 57614021], length 0
12:40:27.170468 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 337:385, ack 288, win 514, options [nop,nop,TS val 57614054 ecr 674956379], length 48
12:40:27.271345 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 385, win 23296, options [nop,nop,TS val 674956697 ecr 57614054], length 0
12:40:28.862176 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 288:336, ack 385, win 23296, options [nop,nop,TS val 674958288 ecr 57614054], length 48
12:40:28.862438 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 385:433, ack 336, win 514, options [nop,nop,TS val 57614223 ecr 674958288], length 48
12:40:28.968190 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 433, win 23296, options [nop,nop,TS val 674958394 ecr 57614223], length 0
12:40:30.883966 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 336:384, ack 433, win 23296, options [nop,nop,TS val 674960310 ecr 57614223], length 48
12:40:30.884176 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 433:481, ack 384, win 514, options [nop,nop,TS val 57614425 ecr 674960310], length 48
12:40:30.985837 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 481, win 23296, options [nop,nop,TS val 674960412 ecr 57614425], length 0
12:40:32.903409 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 384:432, ack 481, win 23296, options [nop,nop,TS val 674962329 ecr 57614425], length 48
12:40:32.903593 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 481:529, ack 432, win 514, options [nop,nop,TS val 57614627 ecr 674962329], length 48
12:40:33.009098 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 529, win 23296, options [nop,nop,TS val 674962435 ecr 57614627], length 0
12:40:34.920212 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 432:480, ack 529, win 23296, options [nop,nop,TS val 674964346 ecr 57614627], length 48
12:40:34.920471 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 529:577, ack 480, win 514, options [nop,nop,TS val 57614829 ecr 674964346], length 48
12:40:35.026059 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 577, win 23296, options [nop,nop,TS val 674964452 ecr 57614829], length 0
12:40:36.928956 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 480:528, ack 577, win 23296, options [nop,nop,TS val 674966355 ecr 57614829], length 48
12:40:36.929185 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 577:625, ack 528, win 514, options [nop,nop,TS val 57615030 ecr 674966355], length 48
12:40:37.035117 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 625, win 23296, options [nop,nop,TS val 674966461 ecr 57615030], length 0
12:40:37.168844 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 625:673, ack 528, win 514, options [nop,nop,TS val 57615054 ecr 674966461], length 48
12:40:37.275059 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 673, win 23296, options [nop,nop,TS val 674966701 ecr 57615054], length 0
12:40:38.954908 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 528:576, ack 673, win 23296, options [nop,nop,TS val 674968381 ecr 57615054], length 48
12:40:38.955089 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 673:721, ack 576, win 514, options [nop,nop,TS val 57615232 ecr 674968381], length 48
12:40:39.056812 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 721, win 23296, options [nop,nop,TS val 674968483 ecr 57615232], length 0
12:40:40.995942 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [P.], seq 576:624, ack 721, win 23296, options [nop,nop,TS val 674970422 ecr 57615232], length 48
12:40:40.996128 IP 192.168.11.10.32103 > 192.168.11.11.3260: Flags [P.], seq 721:769, ack 624, win 514, options [nop,nop,TS val 57615437 ecr 674970422], length 48
12:40:41.097860 IP 192.168.11.11.3260 > 192.168.11.10.32103: Flags [.], ack 769, win 23296, options [nop,nop,TS val 674970524 ecr 57615437], length 0
^C
45 packets captured
52 packets received by filter
0 packets dropped by kernel

#11 Updated by Josh Paetzel over 4 years ago

  • Status changed from 15 to Investigation
  • Assignee changed from Josh Paetzel to Alexander Motin

Alexander,

Is there enough information here for you to take a look at this?

#12 Updated by Alexander Motin over 4 years ago

tcpdump output in provided fashion is useless, except reporting that some link is still connected. It could be more useful if you written full raw data from the beginning of copy, adding `-s 0 -w file_name` to the command, and then uploaded the file here.

`ctladm dumpooa` output tells that CTL is busy with copying, that according to huge command execution times is not going very well. Have you looked on status of your pool and disks in some tools like `gstat -I 1s` or `zpool iostat -v 1`? Is there any disc activity? Does gstat report active queue length above zero for any disks? Does `ctladm dumpooa` output changes with time except command execution times?

If there is no any disc activity and process is completely stuck, then output of `procstat -akk` could be useful.

#13 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Target version changed from 9.10.2-U1 to 9.10.2-U2

#14 Updated by Alexander Motin over 4 years ago

  • Status changed from Investigation to 15

Nothing after a month. Will close it at some point if no more data available.

#15 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Status changed from 15 to Closed: Cannot reproduce

A month is long enough to wait for a response. Lets close it for now.

#16 Updated by Jan Eagleman over 4 years ago

Kris Moore wrote:

A month is long enough to wait for a response. Lets close it for now.

The reason why I haven't updated this is because I haven't been able to reproduce this issue, the first time (after Alexander Motin his response) I tried to clone a VM from a template (and making sure I was logging everything) but then it worked flawlessly.

#17 Updated by Jan Eagleman over 4 years ago

I did however reinstall the entire vSphere environment (had some issues with the heartbeat from ESXi to vCenter)

#18 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

No worries! If you do manage to reproduce it again down the road, let us know!

#19 Updated by Jan Eagleman over 4 years ago

  • File dump added

So I cloned another VM, and suddenly it began to hang. There is 0 disk activity when this happens, I check iostat and gstat for a while, but nothing showed up on the disks that run the iscsi share.

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
currahee                                1.00T  4.43T      0      0      0      0
  mirror                                 514G  2.22T      0      0      0      0
    gptid/2a71ea82-edc6-11e5-8db4-0050569a3b4a      -      -      0      0      0      0
    gptid/2b366928-edc6-11e5-8db4-0050569a3b4a      -      -      0      0      0      0
  mirror                                 514G  2.22T      0      0      0      0
    gptid/2bf79e44-edc6-11e5-8db4-0050569a3b4a      -      -      0      0      0      0
    gptid/2cb76576-edc6-11e5-8db4-0050569a3b4a      -      -      0      0      0      0
logs                                        -      -      -      -      -      -
  gptid/a453ec7b-84d4-11e6-8934-0050569a3b4a     8K  24.9G      0      0      0      0
cache                                       -      -      -      -      -      -
  gptid/a5c67d57-84d5-11e6-8934-0050569a3b4a   125G   200G      0      0      0      0
--------------------------------------  -----  -----  -----  -----  -----  -----
[root@freenas] ~# date
Mon Jan 23 16:41:34 CET 2017
[root@freenas] ~# ctladm dumpooa
Dumping OOA queues
LUN 0 tag 0x131182b4 RTR: EXTENDED COPY. CDB:  (798836 ms)
LUN 0 tag 0x131182b5 RTR: EXTENDED COPY. CDB:  (798836 ms)
LUN 0 tag 0x131182b6 RTR: EXTENDED COPY. CDB:  (798836 ms)
LUN 0 tag 0x131182b7 RTR: EXTENDED COPY. CDB:  (798836 ms)
LUN 0 tag 0x131182b8 RTR: EXTENDED COPY. CDB:  (798836 ms)
LUN 0 tag 0x131182b9 RTR: EXTENDED COPY. CDB:  (798836 ms)
LUN 0 tag 0x131182ba RTR: EXTENDED COPY. CDB:  (798836 ms)
LUN 0 tag 0x131182bb RTR: EXTENDED COPY. CDB:  (798836 ms)
OOA queues dump done
[root@freenas] ~# date
Mon Jan 23 16:42:30 CET 2017
[root@freenas] ~# ctladm dumpooa
Dumping OOA queues
LUN 0 tag 0x131182b4 RTR: EXTENDED COPY. CDB:  (854423 ms)
LUN 0 tag 0x131182b5 RTR: EXTENDED COPY. CDB:  (854423 ms)
LUN 0 tag 0x131182b6 RTR: EXTENDED COPY. CDB:  (854423 ms)
LUN 0 tag 0x131182b7 RTR: EXTENDED COPY. CDB:  (854423 ms)
LUN 0 tag 0x131182b8 RTR: EXTENDED COPY. CDB:  (854423 ms)
LUN 0 tag 0x131182b9 RTR: EXTENDED COPY. CDB:  (854423 ms)
LUN 0 tag 0x131182ba RTR: EXTENDED COPY. CDB:  (854423 ms)
LUN 0 tag 0x131182bb RTR: EXTENDED COPY. CDB:  (854423 ms)
OOA queues dump done
[root@freenas] ~# date
Mon Jan 23 16:44:51 CET 2017
[root@freenas] ~# ctladm dumpooa
Dumping OOA queues
LUN 0 tag 0x131182b4 RTR: EXTENDED COPY. CDB:  (994899 ms)
LUN 0 tag 0x131182b5 RTR: EXTENDED COPY. CDB:  (994899 ms)
LUN 0 tag 0x131182b6 RTR: EXTENDED COPY. CDB:  (994899 ms)
LUN 0 tag 0x131182b7 RTR: EXTENDED COPY. CDB:  (994899 ms)
LUN 0 tag 0x131182b8 RTR: EXTENDED COPY. CDB:  (994899 ms)
LUN 0 tag 0x131182b9 RTR: EXTENDED COPY. CDB:  (994899 ms)
LUN 0 tag 0x131182ba RTR: EXTENDED COPY. CDB:  (994899 ms)
LUN 0 tag 0x131182bb RTR: EXTENDED COPY. CDB:  (994899 ms)
OOA queues dump done
dT: 1.005s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| nvd0
    0      0      0      0    0.0      0      0    0.0    0.0| nvd0p1
    0      0      0      0    0.0      0      0    0.0    0.0| nvd0p2
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/a453ec7b-84d4-11e6-8934-0050569a3b4a
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/a5c67d57-84d5-11e6-8934-0050569a3b4a
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    0      0      0      0    0.0      0      0    0.0    0.0| da1
    0      0      0      0    0.0      0      0    0.0    0.0| da2
    0      0      0      0    0.0      0      0    0.0    0.0| da3
    0      0      0      0    0.0      0      0    0.0    0.0| da4
    0      0      0      0    0.0      0      0    0.0    0.0| da5
    0      0      0      0    0.0      0      0    0.0    0.0| da6
    0      0      0      0    0.0      0      0    0.0    0.0| da7
    0      0      0      0    0.0      0      0    0.0    0.0| da8
    0      0      0      0    0.0      0      0    0.0    0.0| cd0
    0      0      0      0    0.0      0      0    0.0    0.0| da0p1
    0      0      0      0    0.0      0      0    0.0    0.0| da0p2
    0      0      0      0    0.0      0      0    0.0    0.0| da1p1
    0      0      0      0    0.0      0      0    0.0    0.0| da1p2
    0      0      0      0    0.0      0      0    0.0    0.0| da2p1
    0      0      0      0    0.0      0      0    0.0    0.0| da2p2
    0      0      0      0    0.0      0      0    0.0    0.0| da3p1
    0      0      0      0    0.0      0      0    0.0    0.0| da3p2
    0      0      0      0    0.0      0      0    0.0    0.0| da4p1
    0      0      0      0    0.0      0      0    0.0    0.0| da4p2
    0      0      0      0    0.0      0      0    0.0    0.0| da5p1
    0      0      0      0    0.0      0      0    0.0    0.0| da5p2
    0      0      0      0    0.0      0      0    0.0    0.0| da6p1
    0      0      0      0    0.0      0      0    0.0    0.0| da6p2
    0      0      0      0    0.0      0      0    0.0    0.0| da7p1
    0      0      0      0    0.0      0      0    0.0    0.0| da7p2
    0      0      0      0    0.0      0      0    0.0    0.0| da8p1
    0      0      0      0    0.0      0      0    0.0    0.0| da8p2
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/323f62d9-4749-11e6-bb0d-0050569a3b4a
    0      0      0      0    0.0      0      0    0.0    0.0| da2p1.eli
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/2a71ea82-edc6-11e5-8db4-0050569a3b4a
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/2b366928-edc6-11e5-8db4-0050569a3b4a
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/2bf79e44-edc6-11e5-8db4-0050569a3b4a
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/2cb76576-edc6-11e5-8db4-0050569a3b4a
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/8626de7e-ed1c-11e5-8db4-0050569a3b4a
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/2e0a7956-3d66-11e6-82b2-0050569a3b4a
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/2fc045aa-79dd-11e6-8409-0050569a3b4a
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/30433e6b-79dd-11e6-8409-0050569a3b4a
    0      0      0      0    0.0      0      0    0.0    0.0| da3p1.eli
    0      0      0      0    0.0      0      0    0.0    0.0| da4p1.eli
    0      0      0      0    0.0      0      0    0.0    0.0| da6p1.eli
    0      0      0      0    0.0      0      0    0.0    0.0| da7p1.eli
    0      0      0      0    0.0      0      0    0.0    0.0| da8p1.eli
    0      0      0      0    0.0      0      0    0.0    0.0| da1p1.eli
    0      0      0      0    0.0      0      0    0.0    0.0| da5p1.eli
    
    [root@freenas] ~# procstat -akk
  PID    TID COMM             TDNAME           KSTACK
    0 100000 kernel           swapper          mi_switch+0xe1 sleepq_timedwait+0x3a _sleep+0x26e swapper+0x28f btext+0x2c
    0 100011 kernel           firmware taskq   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100015 kernel           thread taskq     mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100017 kernel           kqueue taskq     mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100018 kernel           acpi_task_0      mi_switch+0xe1 sleepq_wait+0x3a msleep_spin_sbt+0x1a3 taskqueue_thread_loop+0xfd fork_exit+0x9a fork_trampoline+0xe
    0 100019 kernel           acpi_task_1      mi_switch+0xe1 sleepq_wait+0x3a msleep_spin_sbt+0x1a3 taskqueue_thread_loop+0xfd fork_exit+0x9a fork_trampoline+0xe
    0 100020 kernel           acpi_task_2      mi_switch+0xe1 sleepq_wait+0x3a msleep_spin_sbt+0x1a3 taskqueue_thread_loop+0xfd fork_exit+0x9a fork_trampoline+0xe
    0 100022 kernel           aiod_bio taskq   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100032 kernel           mps0 taskq       mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100035 kernel           vmx0 taskq       mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100037 kernel           nvme taskq       mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100039 kernel           vmx1 taskq       mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100043 kernel           system_taskq     mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100044 kernel           dbu_evict        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100048 kernel           CAM taskq        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100050 kernel           nvd taskq        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100051 kernel           iw_cm_wq         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100052 kernel           mlx5_core_wq     mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100053 kernel           rdma_cm          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100054 kernel           ib_addr          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100055 kernel           ib_cm_0          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100056 kernel           ib_cm_1          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100057 kernel           ib_cm_2          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100058 kernel           ib_cm_3          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100059 kernel           ib_cm_4          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100060 kernel           ib_cm_5          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100061 kernel           ib_cm_6          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100062 kernel           ib_cm_7          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100063 kernel           ib_cm_8          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100064 kernel           ib_cm_9          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100065 kernel           ib_cm_10         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100066 kernel           ib_cm_11         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100067 kernel           ib_cm_12         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100068 kernel           ib_cm_13         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100069 kernel           ib_cm_14         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100070 kernel           ib_cm_15         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100071 kernel           ib_cm_16         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100072 kernel           ib_cm_17         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100073 kernel           ib_cm_18         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100074 kernel           ib_cm_19         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100075 kernel           ib_cm_20         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100076 kernel           ib_cm_21         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100077 kernel           ib_cm_22         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100078 kernel           ib_cm_23         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100079 kernel           ib_cm_24         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100080 kernel           ib_cm_25         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100081 kernel           ib_cm_26         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100082 kernel           ib_cm_27         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100083 kernel           ib_cm_28         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100084 kernel           ib_cm_29         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100085 kernel           ib_cm_30         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100086 kernel           ib_cm_31         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100087 kernel           ib_cm_32         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100088 kernel           ib_cm_33         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100089 kernel           ib_cm_34         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100090 kernel           ib_cm_35         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100091 kernel           ib_cm_36         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100092 kernel           ib_cm_37         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100093 kernel           ib_cm_38         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100094 kernel           ib_cm_39         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100095 kernel           ib_cm_40         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100096 kernel           ib_cm_41         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100097 kernel           ib_cm_42         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100098 kernel           ib_cm_43         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100099 kernel           ib_cm_44         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100100 kernel           ib_cm_45         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100101 kernel           ib_cm_46         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100102 kernel           ib_cm_47         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100103 kernel           ib_cm_48         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100104 kernel           ib_cm_49         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100105 kernel           ib_cm_50         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100106 kernel           ib_cm_51         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100107 kernel           ib_cm_52         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100108 kernel           ib_cm_53         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100109 kernel           ib_cm_54         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100110 kernel           ib_cm_55         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100111 kernel           ib_cm_56         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100112 kernel           ib_cm_57         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100113 kernel           ib_cm_58         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100114 kernel           ib_cm_59         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100115 kernel           ib_cm_60         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100116 kernel           ib_cm_61         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100117 kernel           ib_cm_62         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100118 kernel           ib_cm_63         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100119 kernel           ib_mcast         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100120 kernel           ib_inform        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100121 kernel           local_sa         mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100122 kernel           mlx4             mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100130 kernel           zio_null_issue   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100134 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100139 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100140 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100145 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100146 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100147 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100148 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100165 kernel           zio_null_intr    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100173 kernel           zio_read_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100174 kernel           zio_read_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100175 kernel           zio_read_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100176 kernel           zio_read_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100177 kernel           zio_read_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100178 kernel           zio_read_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100179 kernel           zio_read_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100180 kernel           zio_read_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100181 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100182 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100183 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100184 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100185 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100186 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100187 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100188 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100189 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100190 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100191 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100192 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100193 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100194 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100195 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100196 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100197 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100198 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100199 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100200 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100201 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100202 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100203 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100204 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100205 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100206 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100207 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100208 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100209 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100210 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100211 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100212 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100213 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100214 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100215 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100216 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100217 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100218 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100219 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100220 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100221 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100222 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100223 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100224 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100225 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100226 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100227 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100228 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100229 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100230 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100231 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100232 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100233 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100234 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100235 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100236 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100237 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100238 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100239 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100240 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100241 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100242 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100243 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100244 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100245 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100246 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100247 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100248 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100249 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100250 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100251 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100252 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100253 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100254 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100255 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100256 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100257 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100258 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100259 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100260 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100261 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100262 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100263 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100264 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100265 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100266 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100267 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100268 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100269 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100270 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100271 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100272 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100273 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100274 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100275 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100276 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100277 kernel           zio_write_issue  mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100278 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100279 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100280 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100281 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100282 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100283 kernel           zio_write_intr_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100284 kernel           zio_write_intr_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100285 kernel           zio_write_intr_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100286 kernel           zio_write_intr_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100287 kernel           zio_write_intr_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100288 kernel           zio_write_intr_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100289 kernel           zio_write_intr_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100290 kernel           zio_write_intr_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100291 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100292 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100293 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100294 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100295 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100296 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100297 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100298 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100299 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100300 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100301 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100302 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100303 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100304 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100305 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100306 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100307 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100308 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100309 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100310 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100311 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100312 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100313 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100314 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100315 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100316 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100317 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100318 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100319 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100320 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100321 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100322 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100323 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100324 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100325 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100326 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100327 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100328 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100329 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100330 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100331 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100332 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100333 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100334 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100335 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100336 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100337 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100338 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100339 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100340 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100341 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100342 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100343 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100344 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100345 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100346 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100347 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100348 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100349 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100350 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100351 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100352 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100353 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100354 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100355 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100356 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100359 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100360 kernel           zio_null_issue   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100361 kernel           zio_null_intr    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100362 kernel           zio_read_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100363 kernel           zio_read_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100364 kernel           zio_read_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100365 kernel           zio_read_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100366 kernel           zio_read_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100367 kernel           zio_read_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100368 kernel           zio_read_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100369 kernel           zio_read_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100370 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100371 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100372 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100373 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100374 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100375 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100376 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100377 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100378 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100379 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100380 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100381 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100382 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100383 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100384 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100385 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100386 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100387 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100388 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100389 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100390 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100391 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100392 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100393 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100394 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100395 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100396 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100397 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100398 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100399 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100400 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100401 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100402 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100403 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100404 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100405 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100406 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100407 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100408 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100409 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100410 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100411 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100412 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100413 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100414 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100415 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100416 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100417 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100418 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100419 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100420 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100421 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100422 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100423 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100424 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100425 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100426 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100427 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100428 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100429 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100430 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100431 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100432 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100433 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100434 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100435 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100436 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100437 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100438 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100439 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100440 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100441 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100442 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100443 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100444 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100445 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100446 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100447 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100448 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100449 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100450 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100451 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100452 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100453 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100454 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100455 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100456 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100457 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100458 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100459 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100460 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100461 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100462 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100463 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100464 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100465 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100466 kernel           zio_write_issue  mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100467 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100468 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100469 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100470 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100471 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100472 kernel           zio_write_intr_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100473 kernel           zio_write_intr_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100474 kernel           zio_write_intr_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100475 kernel           zio_write_intr_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100476 kernel           zio_write_intr_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100477 kernel           zio_write_intr_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100478 kernel           zio_write_intr_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100479 kernel           zio_write_intr_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100480 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100481 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100482 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100483 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100484 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100485 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100486 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100487 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100488 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100489 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100490 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100491 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100492 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100493 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100494 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100495 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100496 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100497 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100498 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100499 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100500 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100501 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100502 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100503 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100504 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100505 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100506 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100507 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100508 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100509 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100510 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100511 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100512 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100513 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100514 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100515 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100516 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100517 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100518 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100519 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100520 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100521 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100522 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100523 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100524 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100525 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100526 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100527 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100528 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100529 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100530 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100531 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100532 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100533 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100534 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100535 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100536 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100537 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100538 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100539 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100540 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100541 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100542 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100543 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100544 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100545 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100546 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100547 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100548 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100549 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100550 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100551 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100552 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100553 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100554 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100555 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100556 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100557 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100558 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100559 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100560 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100561 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100562 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100563 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100564 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100565 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100566 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100567 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100568 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100569 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100570 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100571 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100572 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100573 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100574 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100575 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100576 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100577 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100578 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100579 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100580 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100581 kernel           zio_free_intr    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100582 kernel           zio_claim_issue  mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100583 kernel           zio_claim_intr   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100584 kernel           zio_ioctl_issue  mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100585 kernel           zio_ioctl_intr   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100587 kernel           metaslab_group_t mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100588 kernel           zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100589 kernel           cblk0 taskq_0    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100593 kernel           cblk0 taskq_1    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100600 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100610 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100618 kernel           dtrace_taskq     mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 100619 kernel           system_taskq     mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101084 kernel           zio_null_issue   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101085 kernel           zio_null_intr    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101086 kernel           zio_read_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101087 kernel           zio_read_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101088 kernel           zio_read_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101089 kernel           zio_read_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101090 kernel           zio_read_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101091 kernel           zio_read_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101092 kernel           zio_read_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101093 kernel           zio_read_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101094 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101095 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101096 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101097 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101098 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101099 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101100 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101101 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101102 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101103 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101104 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101105 kernel           zio_read_intr_0_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101106 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101107 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101108 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101109 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101110 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101111 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101112 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101113 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101114 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101115 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101116 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101117 kernel           zio_read_intr_1_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101118 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101119 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101120 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101121 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101122 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101123 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101124 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101125 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101126 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101127 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101128 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101129 kernel           zio_read_intr_2_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101130 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101131 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101132 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101133 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101134 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101135 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101136 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101137 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101138 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101139 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101140 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101141 kernel           zio_read_intr_3_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101142 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101143 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101144 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101145 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101146 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101147 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101148 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101149 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101150 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101151 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101152 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101153 kernel           zio_read_intr_4_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101154 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101155 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101156 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101157 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101158 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101159 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101160 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101161 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101162 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101163 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101164 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101165 kernel           zio_read_intr_5_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101166 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101167 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101168 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101169 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101170 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101171 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101172 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101173 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101174 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101175 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101176 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101177 kernel           zio_read_intr_6_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101178 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101179 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101180 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101181 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101182 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101183 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101184 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101185 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101186 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101187 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101188 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101189 kernel           zio_read_intr_7_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101190 kernel           zio_write_issue  mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101191 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101192 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101193 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101194 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101195 kernel           zio_write_issue_ mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101196 kernel           zio_write_intr_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101197 kernel           zio_write_intr_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101198 kernel           zio_write_intr_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101199 kernel           zio_write_intr_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101200 kernel           zio_write_intr_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101201 kernel           zio_write_intr_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101202 kernel           zio_write_intr_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101203 kernel           zio_write_intr_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101204 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101205 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101206 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101207 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101208 kernel           zio_write_intr_h mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101209 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101210 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101211 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101212 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101213 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101214 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101215 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101216 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101217 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101218 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101219 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101220 kernel           zio_free_issue_0 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101221 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101222 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101223 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101224 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101225 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101226 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101227 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101228 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101229 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101230 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101231 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101232 kernel           zio_free_issue_1 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101233 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101234 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101235 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101236 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101237 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101238 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101239 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101240 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101241 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101242 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101243 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101244 kernel           zio_free_issue_2 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101245 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101246 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101247 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101248 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101249 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101250 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101251 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101252 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101253 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101254 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101255 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101256 kernel           zio_free_issue_3 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101257 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101258 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101259 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101260 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101261 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101262 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101263 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101264 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101265 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101266 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101267 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101268 kernel           zio_free_issue_4 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101269 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101270 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101271 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101272 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101273 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101274 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101275 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101276 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101277 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101278 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101279 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101280 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101281 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101282 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101283 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101284 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101285 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101286 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101287 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101288 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101289 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101290 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101291 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101292 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101293 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101294 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101295 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101296 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101297 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101298 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101299 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101300 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101301 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101302 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101303 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101304 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101305 kernel           zio_free_intr    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101306 kernel           zio_claim_issue  mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101307 kernel           zio_claim_intr   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101308 kernel           zio_ioctl_issue  mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101309 kernel           zio_ioctl_intr   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101311 kernel           metaslab_group_t mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101312 kernel           metaslab_group_t mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101313 kernel           zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101352 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101353 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101354 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101355 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101356 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101357 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101358 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101359 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101360 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101361 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101362 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101363 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101364 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101366 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101367 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101368 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101369 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101370 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101371 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101372 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101373 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101374 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101375 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101376 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101377 kernel           zil_clean        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101846 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101847 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101848 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101849 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101850 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101851 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101852 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101853 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101854 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101855 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101856 kernel           zio_free_issue_5 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101857 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101858 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101859 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101860 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101861 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101862 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101863 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101864 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101865 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101866 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101867 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101868 kernel           zio_free_issue_6 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101869 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101870 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101871 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101872 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101873 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101874 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101875 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101876 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101877 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101878 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101879 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101880 kernel           zio_free_issue_7 mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101881 kernel           zio_free_intr    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101882 kernel           zio_claim_issue  mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101883 kernel           zio_claim_intr   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101884 kernel           zio_ioctl_issue  mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101885 kernel           zio_ioctl_intr   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101887 kernel           metaslab_group_t mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101888 kernel           metaslab_group_t mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101889 kernel           metaslab_group_t mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101890 kernel           zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101911 kernel           cblk0 taskq_2    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101912 kernel           cblk0 taskq_3    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101913 kernel           cblk0 taskq_4    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101914 kernel           cblk0 taskq_5    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101915 kernel           cblk0 taskq_6    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101916 kernel           cblk0 taskq_7    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101917 kernel           cblk0 taskq_8    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101918 kernel           cblk0 taskq_9    mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101919 kernel           cblk0 taskq_10   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101920 kernel           cblk0 taskq_11   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101921 kernel           cblk0 taskq_12   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 101922 kernel           cblk0 taskq_13   mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 taskqueue_thread_loop+0xd5 fork_exit+0x9a fork_trampoline+0xe
    0 102004 kernel           cfiscsimt        mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x17d cfiscsi_maintenance_thread+0xbe fork_exit+0x9a fork_trampoline+0xe
    0 102005 kernel           cfiscsitx        mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x17d icl_send_thread+0x8c fork_exit+0x9a fork_trampoline+0xe
    0 102006 kernel           cfiscsirx        mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x17d icl_receive_thread+0xa1 fork_exit+0x9a fork_trampoline+0xe
    1 100002 init             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_wait6+0x5de sys_wait4+0x72 amd64_syscall+0x40f Xfast_syscall+0xfb
    2 100012 crypto           -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 crypto_proc+0x359 fork_exit+0x9a fork_trampoline+0xe
    3 100013 crypto returns   -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 crypto_ret_proc+0x216 fork_exit+0x9a fork_trampoline+0xe
    4 100021 cam              doneq0           mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 xpt_done_td+0x8e fork_exit+0x9a fork_trampoline+0xe
    4 100049 cam              scanner          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 xpt_scanner_thread+0x7c fork_exit+0x9a fork_trampoline+0xe
    5 100025 ctl              work0            mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x17d vmem_xalloc+0x568 vmem_alloc+0x3d kmem_malloc+0x33 uma_large_malloc+0x46 malloc+0x44 tpc_process+0x1884 ctl_extended_copy_lid1+0x4be ctl_datamove+0x1ad ctl_extended_copy_lid1+0x121 ctl_work_thread+0x439 fork_exit+0x9a fork_trampoline+0xe
    5 100026 ctl              lun              mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 ctl_lun_thread+0x68 fork_exit+0x9a fork_trampoline+0xe
    5 100027 ctl              thresh           mi_switch+0xe1 sleepq_timedwait+0x3a _sleep+0x26e pause_sbt+0x6e ctl_thresh_thread+0x5d fork_exit+0x9a fork_trampoline+0xe
    6 100031 mpt_recovery0    -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 mpt_recovery_thread+0xa6 fork_exit+0x9a fork_trampoline+0xe
    7 100045 zfskern          arc_reclaim_thre mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x19e arc_reclaim_thread+0x282 fork_exit+0x9a fork_trampoline+0xe
    7 100046 zfskern          arc_user_evicts_ mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x19e arc_user_evicts_thread+0x17d fork_exit+0x9a fork_trampoline+0xe
    7 100047 zfskern          l2arc_feed_threa mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x19e l2arc_feed_thread+0xd73 fork_exit+0x9a fork_trampoline+0xe
    7 100131 zfskern          txg_thread_enter mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x17d txg_quiesce_thread+0x16b fork_exit+0x9a fork_trampoline+0xe
    7 100132 zfskern          txg_thread_enter mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x19e txg_sync_thread+0x160 fork_exit+0x9a fork_trampoline+0xe
    7 100586 zfskern          trim freenas-boo mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x19e trim_thread+0x126 fork_exit+0x9a fork_trampoline+0xe
    7 100598 zfskern          txg_thread_enter mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x17d txg_quiesce_thread+0x16b fork_exit+0x9a fork_trampoline+0xe
    7 100599 zfskern          txg_thread_enter mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x19e txg_sync_thread+0x160 fork_exit+0x9a fork_trampoline+0xe
    7 101310 zfskern          trim silver      mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x19e trim_thread+0x126 fork_exit+0x9a fork_trampoline+0xe
    7 101349 zfskern          txg_thread_enter mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x17d txg_quiesce_thread+0x16b fork_exit+0x9a fork_trampoline+0xe
    7 101350 zfskern          txg_thread_enter mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x19e txg_sync_thread+0x160 fork_exit+0x9a fork_trampoline+0xe
    7 101886 zfskern          trim currahee    mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x19e trim_thread+0x126 fork_exit+0x9a fork_trampoline+0xe
    8 100123 pagedaemon       -                mi_switch+0xe1 sleepq_timedwait+0x3a _sleep+0x26e vm_pageout+0x1f5 fork_exit+0x9a fork_trampoline+0xe
    8 100129 pagedaemon       uma              mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 uma_reclaim_worker+0x81 fork_exit+0x9a fork_trampoline+0xe
    9 100124 vmdaemon         -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 vm_daemon+0xe0 fork_exit+0x9a fork_trampoline+0xe
   10 100001 audit            -                mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x17d audit_worker+0xa3 fork_exit+0x9a fork_trampoline+0xe
   11 100003 idle             idle: cpu0       mi_switch+0xe1 critical_exit+0x7a intr_event_handle+0x106 intr_execute_handlers+0x48 lapic_handle_intr+0x3f Xapic_isr1+0xa4 acpi_cpu_idle+0x15a cpu_idle_acpi+0x3f cpu_idle+0x90 sched_idletd+0x1d5 fork_exit+0x9a fork_trampoline+0xe
   12 100004 intr             swi4: clock      mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100005 intr             swi1: netisr 0   mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100006 intr             swi3: vm
   12 100016 intr             swi5: fast taskq
   12 100023 intr             swi6: task queue mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100024 intr             swi6: Giant task mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100028 intr             irq14: ata0
   12 100029 intr             irq15: ata1      mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100030 intr             irq17: mpt0      mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100033 intr             irq256: mps0     mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100034 intr             irq257: vmx0     mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100036 intr             irq16: nvme0     mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100038 intr             irq258: vmx1     mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100040 intr             irq1: atkbd0     mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100041 intr             irq12: psm0
   12 100042 intr             swi0: uart uart
   13 100007 ng_queue         ng_queue0        mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 ngthread+0x9c fork_exit+0x9a fork_trampoline+0xe
   14 100008 geom             g_event          mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_run_events+0x4d fork_exit+0x9a fork_trampoline+0xe
   14 100009 geom             g_up             mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_io_schedule_up+0xbc g_up_procbody+0x6d fork_exit+0x9a fork_trampoline+0xe
   14 100010 geom             g_down           mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_io_schedule_down+0x4c g_down_procbody+0x6d fork_exit+0x9a fork_trampoline+0xe
   15 100014 rand_harvestq    -                mi_switch+0xe1 sleepq_timedwait+0x3a msleep_spin_sbt+0x191 random_kthread+0x276 fork_exit+0x9a fork_trampoline+0xe
   16 100125 pagezero         -                mi_switch+0xe1 sleepq_timedwait+0x3a _sleep+0x26e vm_pagezero+0x98 fork_exit+0x9a fork_trampoline+0xe
   17 100126 bufdaemon        -                mi_switch+0xe1 sleepq_timedwait+0x3a _sleep+0x26e buf_daemon+0x78 fork_exit+0x9a fork_trampoline+0xe
   18 100127 syncer           -                mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x19e sched_sync+0x6ce fork_exit+0x9a fork_trampoline+0xe
   19 100128 vnlru            -                mi_switch+0xe1 sleepq_timedwait+0x3a _sleep+0x26e vnlru_proc+0x48 fork_exit+0x9a fork_trampoline+0xe
  230 100603 g_mp_kt          -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_multipath_kt+0x6c fork_exit+0x9a fork_trampoline+0xe
  788 100612 ftcleanup        -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 fasttrap_pid_cleanup_cb+0x66 fork_exit+0x9a fork_trampoline+0xe
 1008 100614 HgfsKReqWorker   -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a HgfsKReqWorker+0x7e fork_exit+0x9a fork_trampoline+0xe
 1251 100617 devd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 1776 100613 Timer            -                mi_switch+0xe1 sleepq_wait+0x3a rtR0SemEventWait+0x252 rtTimerThread+0xfe rtThreadMain+0x2c rtThreadNativeMain+0x16 fork_exit+0x9a fork_trampoline+0xe
 1911 100135 syslog-ng        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_wait6+0x5de sys_wait4+0x72 amd64_syscall+0x40f Xfast_syscall+0xfb
 1912 100604 syslog-ng        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 kern_kevent_fp+0x399 kern_kevent+0x9f sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb
 1919 100606 rpcbind          -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
 2462 100602 g_eli[0] da5p1   -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_eli_worker+0x1b4 fork_exit+0x9a fork_trampoline+0xe
 2464 100609 g_eli[0] da6p1   -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_eli_worker+0x1b4 fork_exit+0x9a fork_trampoline+0xe
 2466 100611 g_eli[0] da7p1   -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_eli_worker+0x1b4 fork_exit+0x9a fork_trampoline+0xe
 2468 100615 g_eli[0] da8p1   -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_eli_worker+0x1b4 fork_exit+0x9a fork_trampoline+0xe
 2470 100151 g_eli[0] da1p1   -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_eli_worker+0x1b4 fork_exit+0x9a fork_trampoline+0xe
 2472 100152 g_eli[0] da2p1   -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_eli_worker+0x1b4 fork_exit+0x9a fork_trampoline+0xe
 2474 100357 g_eli[0] da3p1   -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_eli_worker+0x1b4 fork_exit+0x9a fork_trampoline+0xe
 2476 100358 g_eli[0] da4p1   -                mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x287 g_eli_worker+0x1b4 fork_exit+0x9a fork_trampoline+0xe
 2524 100605 ctld             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a seltdwait+0xae kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 2711 101923 mountd           -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a seltdwait+0xae kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 2717 101924 nfsd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a seltdwait+0xae kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 2719 100616 nfsd             nfsd: master     mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e svc_run_internal+0x46d svc_run+0x182 nfsrvd_nfsd+0x242 nfssvc_nfsd+0x107 sys_nfssvc+0x9c amd64_syscall+0x40f Xfast_syscall+0xfb
 2719 101927 nfsd             nfsd: service    mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a svc_run_internal+0x86e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
 2719 101928 nfsd             nfsd: service    mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a svc_run_internal+0x86e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
 2719 101929 nfsd             nfsd: service    mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a svc_run_internal+0x86e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
 2721 101925 rpc.statd        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 2724 101926 rpc.lockd        NLM: master      mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e svc_run_internal+0x46d svc_run+0x182 sys_nlm_syscall+0x736 amd64_syscall+0x40f Xfast_syscall+0xfb
 2801 101932 ntpd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a seltdwait+0xae kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 3080 102010 smbd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
 3976 101935 nmbd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
 3979 101934 smbd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
 3982 101933 smbd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
 3983 101938 winbindd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
 4034 100590 vmtoolsd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
 4190 101937 sshd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a seltdwait+0xae kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 4704 101947 smartd           -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 kern_nanosleep+0x10e sys_nanosleep+0x51 amd64_syscall+0x40f Xfast_syscall+0xfb
 4728 101943 sh               -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_wait6+0x5de sys_wait4+0x72 amd64_syscall+0x40f Xfast_syscall+0xfb
 4836 101949 nginx            -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_sigsuspend+0xf4 sys_sigsuspend+0x31 amd64_syscall+0x40f Xfast_syscall+0xfb
 4842 101948 winbindd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
 4844 101942 mdnsd            -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 4899 101952 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 4929 101956 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 4929 101957 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_sem_wait+0x4db __umtx_op_sem_wait+0x73 amd64_syscall+0x40f Xfast_syscall+0xfb
 4929 101958 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_sem_wait+0x4db __umtx_op_sem_wait+0x73 amd64_syscall+0x40f Xfast_syscall+0xfb
 4929 101959 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_sem_wait+0x4db __umtx_op_sem_wait+0x73 amd64_syscall+0x40f Xfast_syscall+0xfb
 4929 101960 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_sem_wait+0x4db __umtx_op_sem_wait+0x73 amd64_syscall+0x40f Xfast_syscall+0xfb
 4929 101961 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_sem_wait+0x4db __umtx_op_sem_wait+0x73 amd64_syscall+0x40f Xfast_syscall+0xfb
 4946 101954 dbus-daemon      -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a seltdwait+0xae kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 100592 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 101953 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 kern_nanosleep+0x10e sys_nanosleep+0x51 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 101964 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 101965 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 101966 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 101967 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 101968 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 101969 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 101970 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 101971 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 101972 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x40f Xfast_syscall+0xfb
 5088 101973 collectd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 umtxq_sleep+0x125 do_wait+0x387 __umtx_op_wait_uint_private+0x83 amd64_syscall+0x40f Xfast_syscall+0xfb
 5169 101939 daemon           -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_sigtimedwait+0x24c sys_sigwait+0x49 amd64_syscall+0x40f Xfast_syscall+0xfb
 5171 101962 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d umtxq_sleep+0x125 do_sem_wait+0x4db __umtx_op_sem_wait+0x73 amd64_syscall+0x40f Xfast_syscall+0xfb
 5171 101979 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a seltdwait+0xae kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 5171 101980 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a seltdwait+0xae kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 5171 101981 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a seltdwait+0xae kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
 5795 101984 winbindd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
 5918 101991 zfsd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a seltdwait+0xae kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
 5928 101993 cron             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 kern_nanosleep+0x10e sys_nanosleep+0x51 amd64_syscall+0x40f Xfast_syscall+0xfb
 5980 101982 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a tty_wait+0x1c ttydisc_read+0xf4 ttydev_read+0x86 devfs_read_f+0xd8 dofileread+0x95 kern_readv+0x68 sys_read+0x63 amd64_syscall+0x40f Xfast_syscall+0xfb
 5981 101951 getty            -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a tty_wait+0x1c ttydisc_read+0x2d4 ttydev_read+0x86 devfs_read_f+0xd8 dofileread+0x95 kern_readv+0x68 sys_read+0x63 amd64_syscall+0x40f Xfast_syscall+0xfb
 5982 101997 getty            -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a tty_wait+0x1c ttydisc_read+0x2d4 ttydev_read+0x86 devfs_read_f+0xd8 dofileread+0x95 kern_readv+0x68 sys_read+0x63 amd64_syscall+0x40f Xfast_syscall+0xfb
 5983 101998 getty            -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a tty_wait+0x1c ttydisc_read+0x2d4 ttydev_read+0x86 devfs_read_f+0xd8 dofileread+0x95 kern_readv+0x68 sys_read+0x63 amd64_syscall+0x40f Xfast_syscall+0xfb
 5984 101999 getty            -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a tty_wait+0x1c ttydisc_read+0x2d4 ttydev_read+0x86 devfs_read_f+0xd8 dofileread+0x95 kern_readv+0x68 sys_read+0x63 amd64_syscall+0x40f Xfast_syscall+0xfb
 5985 102000 getty            -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a tty_wait+0x1c ttydisc_read+0x2d4 ttydev_read+0x86 devfs_read_f+0xd8 dofileread+0x95 kern_readv+0x68 sys_read+0x63 amd64_syscall+0x40f Xfast_syscall+0xfb
 5986 102001 getty            -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a tty_wait+0x1c ttydisc_read+0x2d4 ttydev_read+0x86 devfs_read_f+0xd8 dofileread+0x95 kern_readv+0x68 sys_read+0x63 amd64_syscall+0x40f Xfast_syscall+0xfb
 5987 102002 getty            -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x17a tty_wait+0x1c ttydisc_read+0x2d4 ttydev_read+0x86 devfs_read_f+0xd8 dofileread+0x95 kern_readv+0x68 sys_read+0x63 amd64_syscall+0x40f Xfast_syscall+0xfb
 7251 102022 smbd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
12409 101940 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
12409 101941 python2.7        -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
12411 100857 bash             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_wait6+0x5de sys_wait4+0x72 amd64_syscall+0x40f Xfast_syscall+0xfb
12857 101996 tcpdump          -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 bpfread+0x1c7 devfs_read_f+0xd8 dofileread+0x95 kern_readv+0x68 sys_read+0x63 amd64_syscall+0x40f Xfast_syscall+0xfb
12858 101995 sshd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_select+0x8fa sys_select+0x54 amd64_syscall+0x40f Xfast_syscall+0xfb
12860 101977 csh              -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _sleep+0x27d kern_sigsuspend+0xf4 sys_sigsuspend+0x31 amd64_syscall+0x40f Xfast_syscall+0xfb
14184 102021 sleep            -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 kern_nanosleep+0x10e sys_nanosleep+0x51 amd64_syscall+0x40f Xfast_syscall+0xfb
14185 101978 procstat         -                <running>
19112 102020 winbindd         -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
51294 102003 smbd             -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x19e seltdwait+0xa4 kern_poll+0x464 sys_poll+0x61 amd64_syscall+0x40f Xfast_syscall+0xfb
54364 101989 nginx            -                mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _sleep+0x238 kern_kevent_fp+0x399 kern_kevent+0x9f sys_kevent+0x12a amd64_syscall+0x40f Xfast_syscall+0xfb

#20 Updated by Jan Eagleman over 4 years ago

Can someone change the status

#21 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Status changed from Closed: Cannot reproduce to Unscreened

Looks like he managed to repro it. Re-opening for another look once mav gets back from vacation / travel.

#22 Updated by Alexander Motin over 4 years ago

  • Status changed from Unscreened to Investigation

I see where it stuck -- on attempt to allocate (possibly large) memory buffer to do XCOPY:

5 100025 ctl work0 mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x17d vmem_xalloc+0x568 vmem_alloc+0x3d kmem_malloc+0x33 uma_large_malloc+0x46 malloc+0x44 tpc_process+0x1884 ctl_extended_copy_lid1+0x4be ctl_datamove+0x1ad ctl_extended_copy_lid1+0x121 ctl_work_thread+0x439 fork_exit+0x9a fork_trampoline+0xe

My guess is that memory on your system got so fragmented that it appeared to be difficult. Normal scenario after that is that ZFS should drop ARC content until fragmentation fixed sufficiently. It may take some time, but should not be fatal. I need to look what could go wrong and what can be done about it.

As workaround, increasing size of kernel virtual memory with loader tunable should probably help. Try to enable system autotuner, if it is not done yet.

#23 Updated by Jan Eagleman over 4 years ago

I did however add a few tunables myself, not sure if these are related:

http://i.imgur.com/Gh4rxYA.png

#24 Updated by Alexander Motin over 4 years ago

I am talking about vm.kmem_size, which should be set to 1.25 - 1.5 of hw.physmem, as it is set by autotuner.

#25 Updated by Jan Eagleman over 4 years ago

Just asking if my tunables can cause any problems with the allocation of memory when cloning a VM.

#26 Updated by Alexander Motin over 4 years ago

Jan Eagleman wrote:

Just asking if my tunables can cause any problems with the allocation of memory when cloning a VM.

None of them are seriously related to memory allocation.

#27 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Target version changed from 9.10.2-U2 to 9.10.3

#28 Updated by Alexander Motin over 4 years ago

  • Related to Bug #19892: connection error; dropping connection added

#29 Updated by Alexander Motin over 4 years ago

  • Status changed from Investigation to Fix In Progress
  • Seen in changed from 9.10.1-U4 to 9.10.2-U2

I expect this change should mitigate the issue in next release: https://svnweb.freebsd.org/changeset/base/313910

#30 Updated by Alexander Motin over 4 years ago

  • Status changed from Fix In Progress to Ready For Release

#31 Updated by Jan Eagleman over 4 years ago

When will this be released?

#32 Updated by Alexander Motin over 4 years ago

May be in a month. Sooner only in nightly builds (should already be there).

#33 Avatar?id=14398&size=24x24 Updated by Kris Moore over 4 years ago

  • Target version changed from 9.10.3 to 11.0

#36 Updated by Jan Eagleman over 4 years ago

Why has this changed to version 11.0?

#37 Updated by Alexander Motin over 4 years ago

Because 9.10.3 was renamed to 11.0, respecting switch to FreeBSD 11.

#38 Updated by Vaibhav Chauhan about 4 years ago

  • Target version changed from 11.0 to 11.0-RC

#39 Updated by Vaibhav Chauhan about 4 years ago

  • Status changed from Ready For Release to Resolved

#40 Updated by Dru Lavigne over 3 years ago

  • File deleted (dump)

Also available in: Atom PDF