Bug #27755

FreeNAS 11.1 panic when destination for ZFS replication

Added by Craig Shiels over 2 years ago. Updated over 2 years ago.

No priority
Benno Rice
Target version:
Seen in:
Reason for Closing:
Cannot Reproduce
Reason for Blocked:
Needs QA:
Needs Doc:
Needs Merging:
Needs Automation:
Support Suite Ticket:
Hardware Configuration:
ChangeLog Required:


When doing a replication from FreeNAS 9.3 to 11.1 and where the target volume has been enabled for encryption, ZFS reliably panics:

--- trap 0x17
[repeats of zio_execute, zio_vdev_io_start, zio_execute, vdev_queue_io_done]

There is a massive amount of recursion of zio_execute ... vdev_queue_io_done. Maybe 4 or 5 console screens worth.

panic2.png (838 KB) panic2.png Craig Shiels, 01/11/2018 10:40 PM
panic1.png (1.5 MB) panic1.png Craig Shiels, 01/11/2018 10:51 PM


#1 Updated by Craig Shiels over 2 years ago

On the sending side, the replication is configured with these options:

Recursively replicate child dataset's snapshots: Yes
Delete stale snapshots on remote system: Yes
Limit: 0
Begin: 00:00:00
End: 23:59:00
Enabled: Yes
Remote port: 22
Dedicated User enabled: Yes
Dedicated User: root
Encryption Cipher: Fast

#2 Updated by Dru Lavigne over 2 years ago

  • Assignee changed from Release Council to Benno Rice
  • Target version set to 11.2-BETA1

Benno: please indicate what type of info Craig should attach to assist you in pinpointing the issue.

#3 Updated by Craig Shiels over 2 years ago

For what it is worth, this problem happens reliably when the replication gets through sending about 300GB of data of one of the datasets - this can be anywhere between 1h30 - 2h10 in duration (depending on what else the source is doing.) On watching the system reboot, there are messages printed on the console by zfs about it removing a lot of blocks as it winds back some transactions (and the 300GB of data transferred is then lost). None of these messages from ZFS make it into syslog.

The target system that is panic'ing isn't "production" (yet), so I could "upgrade" it to a version of FreeNAS with better debugging symbols to get more meaningful information when the system panic's.

The panic, of course, is due to the kernel running out of stack space.

#4 Updated by Benno Rice over 2 years ago

A debug/diagnostic tarball would be helpful. At the very least the full call stack would be useful just so I can see the exact path it's taking.

#5 Updated by Craig Shiels over 2 years ago


Screen shot of the backtrace at the point of panic attached.

Just lost ~1.7TB of replicated data because of this bug.

I cannot find anything useful in the log files anywhere.

#6 Updated by Craig Shiels over 2 years ago


Of course, you need the other end of the stack trace.

In the middle, it's just the same functions over and over again.

#7 Updated by Craig Shiels over 2 years ago

How do I run a FreeNAS kernel that will show function arguments in the stack trace on panic?

What's the FreeNAS way of enabling savecore?

#8 Updated by Craig Shiels over 2 years ago

Ah, found

Tracing pid 0 tid 102325 td 0xfffff800991ed000
kdb_enter() at kdb_enter+0x3b/frame 0xfffffe064abb8d80
vpanic() at vpanic+0x1a3/frame 0xfffffe064abb8e00
panic() at panic+0x43/frame 0xfffffe064abb8e60
dblfault_handler() at dblfault_handler+0x1de/frame 0xfffffe064abb8f30
Xdblfault() at Xdblfault+0xac/frame 0xfffffe064abb8f30
--- trap 0x17, rip = 0xffffffff8043ddaa, rsp = 0xfffffe066952cf90, rbp = 0xfffffe066952d450 ---
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x2a/frame 0xfffffe066952d450
vdev_queue_io_done() at vdev_queue_io_done+0xc8/frame 0xfffffe066952d4a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952d4d0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952d520
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952d580
zio_execute() at zio_execute+0xac/frame 0xfffffe066952d5d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952d620
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952d650
zio_execute() at zio_execute+0xac/frame 0xfffffe066952d6a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952d700
zio_execute() at zio_execute+0xac/frame 0xfffffe066952d750
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952d7a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952d7d0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952d820
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952d880
zio_execute() at zio_execute+0xac/frame 0xfffffe066952d8d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952d920
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952d950
zio_execute() at zio_execute+0xac/frame 0xfffffe066952d9a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952da00
zio_execute() at zio_execute+0xac/frame 0xfffffe066952da50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952daa0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952dad0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952db20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952db80
zio_execute() at zio_execute+0xac/frame 0xfffffe066952dbd0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952dc20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952dc50
zio_execute() at zio_execute+0xac/frame 0xfffffe066952dca0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952dd00
zio_execute() at zio_execute+0xac/frame 0xfffffe066952dd50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952dda0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952ddd0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952de20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952de80
zio_execute() at zio_execute+0xac/frame 0xfffffe066952ded0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952df20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952df50
zio_execute() at zio_execute+0xac/frame 0xfffffe066952dfa0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952e000
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e050
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952e0a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952e0d0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e120
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952e180
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e1d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952e220
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952e250
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e2a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952e300
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e350
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952e3a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952e3d0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e420
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952e480
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e4d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952e520
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952e550
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e5a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952e600
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e650
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952e6a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952e6d0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e720
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952e780
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e7d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952e820
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952e850
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e8a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952e900
zio_execute() at zio_execute+0xac/frame 0xfffffe066952e950
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952e9a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952e9d0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952ea20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952ea80
zio_execute() at zio_execute+0xac/frame 0xfffffe066952ead0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952eb20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952eb50
zio_execute() at zio_execute+0xac/frame 0xfffffe066952eba0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952ec00
zio_execute() at zio_execute+0xac/frame 0xfffffe066952ec50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952eca0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952ecd0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952ed20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952ed80
zio_execute() at zio_execute+0xac/frame 0xfffffe066952edd0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952ee20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952ee50
zio_execute() at zio_execute+0xac/frame 0xfffffe066952eea0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952ef00
zio_execute() at zio_execute+0xac/frame 0xfffffe066952ef50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952efa0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952efd0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f020
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952f080
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f0d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952f120
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952f150
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f1a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952f200
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f250
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952f2a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952f2d0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f320
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952f380
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f3d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952f420
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952f450
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f4a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952f500
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f550
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952f5a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952f5d0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f620
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952f680
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f6d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952f720
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952f750
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f7a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952f800
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f850
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952f8a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952f8d0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f920
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952f980
zio_execute() at zio_execute+0xac/frame 0xfffffe066952f9d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952fa20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952fa50
zio_execute() at zio_execute+0xac/frame 0xfffffe066952faa0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952fb00
zio_execute() at zio_execute+0xac/frame 0xfffffe066952fb50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952fba0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952fbd0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952fc20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952fc80
zio_execute() at zio_execute+0xac/frame 0xfffffe066952fcd0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952fd20
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952fd50
zio_execute() at zio_execute+0xac/frame 0xfffffe066952fda0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952fe00
zio_execute() at zio_execute+0xac/frame 0xfffffe066952fe50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe066952fea0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe066952fed0
zio_execute() at zio_execute+0xac/frame 0xfffffe066952ff20
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe066952ff80
zio_execute() at zio_execute+0xac/frame 0xfffffe066952ffd0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe0669530020
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe0669530050
zio_execute() at zio_execute+0xac/frame 0xfffffe06695300a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe0669530100
zio_execute() at zio_execute+0xac/frame 0xfffffe0669530150
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe06695301a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe06695301d0
zio_execute() at zio_execute+0xac/frame 0xfffffe0669530220
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe0669530280
zio_execute() at zio_execute+0xac/frame 0xfffffe06695302d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe0669530320
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe0669530350
zio_execute() at zio_execute+0xac/frame 0xfffffe06695303a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe0669530400
zio_execute() at zio_execute+0xac/frame 0xfffffe0669530450
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe06695304a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe06695304d0
zio_execute() at zio_execute+0xac/frame 0xfffffe0669530520
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe0669530580
zio_execute() at zio_execute+0xac/frame 0xfffffe06695305d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe0669530620
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe0669530650
zio_execute() at zio_execute+0xac/frame 0xfffffe06695306a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe0669530700
zio_execute() at zio_execute+0xac/frame 0xfffffe0669530750
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe06695307a0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe06695307d0
zio_execute() at zio_execute+0xac/frame 0xfffffe0669530820
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe0669530880
zio_execute() at zio_execute+0xac/frame 0xfffffe06695308d0
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe0669530920
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe0669530950
zio_execute() at zio_execute+0xac/frame 0xfffffe06695309a0
zio_vdev_io_start() at zio_vdev_io_start+0x2ad/frame 0xfffffe0669530a00
zio_execute() at zio_execute+0xac/frame 0xfffffe0669530a50
vdev_queue_io_done() at vdev_queue_io_done+0x115/frame 0xfffffe0669530aa0
zio_vdev_io_done() at zio_vdev_io_done+0x89/frame 0xfffffe0669530ad0
zio_execute() at zio_execute+0xac/frame 0xfffffe0669530b20
taskqueue_run_locked() at taskqueue_run_locked+0x147/frame 0xfffffe0669530b80
taskqueue_thread_loop() at taskqueue_thread_loop+0xb8/frame 0xfffffe0669530bb0
fork_exit() at fork_exit+0x85/frame 0xfffffe0669530bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0669530bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:0:kdb.enter.default> show allpcpu
Current CPU: 14

#9 Updated by Craig Shiels over 2 years ago

cpuid = 14
dynamic pcpu = 0xfffffe06c8eb2800
curthread = 0xfffff800991ed000: pid 0 "zio_free_intr"
curpcb = 0xfffffe0669530cc0
fpcurthread = none
idlethread = 0xfffff8000a406000: tid 100017 "idle: cpu14"
curpmap = 0xffffffff82179378
tssp = 0xffffffff821ad0c0
commontssp = 0xffffffff821ad0c0
rsp0 = 0xfffffe0669530cc0
gs32p = 0xffffffff821b3918
ldt = 0xffffffff821b3958
tss = 0xffffffff821b3948
curvnet = 0

#10 Updated by Craig Shiels over 2 years ago

102620 D - 0xfffff80046d42500 [zil_clean]
db:0:kdb.enter.default> alltrace

Tracing command python3.6 pid 4137 tid 102652 td 0xfffff800992045c0
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe064abb1d40
ipi_nmi_handler() at ipi_nmi_handler+0x45/frame 0xfffffe064abb1d60
trap() at trap+0x3a/frame 0xfffffe064abb1f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe064abb1f20
--- trap 0x13, rip = 0xffffffff8047e6c0, rsp = 0xfffffe06698d15b8, rbp = 0xfffffe06698d1630 ---
zio_wait() at zio_wait/frame 0xfffffe06698d1630
dmu_read_uio_dnode() at dmu_read_uio_dnode+0x37/frame 0xfffffe06698d16a0
dmu_read_uio_dbuf() at dmu_read_uio_dbuf+0x3b/frame 0xfffffe06698d16d0
zfs_freebsd_read() at zfs_freebsd_read+0x660/frame 0xfffffe06698d1780
VOP_READ_APV() at VOP_READ_APV+0x83/frame 0xfffffe06698d17b0
vn_read() at vn_read+0x195/frame 0xfffffe06698d1830
vn_io_fault1() at vn_io_fault1+0x169/frame 0xfffffe06698d1970
vn_io_fault() at vn_io_fault+0x18c/frame 0xfffffe06698d19e0
dofileread() at dofileread+0xba/frame 0xfffffe06698d1a20
kern_readv() at kern_readv+0x68/frame 0xfffffe06698d1a70
sys_read() at sys_read+0x86/frame 0xfffffe06698d1ac0
amd64_syscall() at amd64_syscall+0xa4a/frame 0xfffffe06698d1bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe06698d1bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8019671aa, rsp = 0x7ffffffb5e78, rbp = 0x7ffffffb5eb0 ---


Tracing command zfskern pid 16 tid 102330 td 0xfffff800992005c0
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe064ab99d40
ipi_nmi_handler() at ipi_nmi_handler+0x45/frame 0xfffffe064ab99d60
trap() at trap+0x3a/frame 0xfffffe064ab99f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe064ab99f20
--- trap 0x13, rip = 0xffffffff80ab20d2, rsp = 0xfffffe0669549760, rbp = 0xfffffe0669549770 ---
lock_delay() at lock_delay+0x42/frame 0xfffffe0669549770
_sx_xlock_hard() at _sx_xlock_hard+0x559/frame 0xfffffe0669549820
vdev_queue_io() at vdev_queue_io+0x1a3/frame 0xfffffe0669549870
zio_vdev_io_start() at zio_vdev_io_start+0x24c/frame 0xfffffe06695498d0
zio_execute() at zio_execute+0xac/frame 0xfffffe0669549920
zio_nowait() at zio_nowait+0xcb/frame 0xfffffe0669549960
trim_map_commit() at trim_map_commit+0x2d9/frame 0xfffffe0669549a10
trim_map_commit() at trim_map_commit+0xe6/frame 0xfffffe0669549ac0
trim_map_commit() at trim_map_commit+0xe6/frame 0xfffffe0669549b70
trim_thread() at trim_thread+0x9f/frame 0xfffffe0669549bb0
fork_exit() at fork_exit+0x85/frame 0xfffffe0669549bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0669549bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

#11 Updated by Craig Shiels over 2 years ago

Fatal double fault
rip 0xffffffff8043ddaa rsp 0xfffffe066952cf90 rbp 0xfffffe066952d450
rax 0xfffff80099ac9000 rdx 0x267c024e rbx 0xfffff80099ee6000
rcx 0x63d0 rsi 0xfffff80099ac9570 rdi 0xfffff800b0980a70
r8 0xfffff800b06c5a70 r9 0xfffff800a13b9660 r10 0
r11 0x1 r12 0xfffff80099ac9448 r13 0xfffff800991ed000
r14 0xfffff80099ac9000 r15 0xfffff80099ac9570 rflags 0x10286
cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
fsbase 0x800644128 gsbase 0xffffffff821bd380 kgsbase 0
cpuid = 14; apic id = 16
panic: double fault
cpuid = 14
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe064abb8d80
vpanic() at vpanic+0x186/frame 0xfffffe064abb8e00
panic() at panic+0x43/frame 0xfffffe064abb8e60
dblfault_handler() at dblfault_handler+0x1de/frame 0xfffffe064abb8f30
Xdblfault() at Xdblfault+0xac/frame 0xfffffe064abb8f30
--- trap 0x17, rip = 0xffffffff8043ddaa, rsp = 0xfffffe066952cf90, rbp = 0xfffffe066952d450 ---

#12 Updated by Craig Shiels over 2 years ago

kgdb /boot/kernel-debug/kernel vmcore.# provides no joy :*(

#13 Updated by Craig Shiels over 2 years ago

For the crashes that have generated a textdump.tar.#.gz file, there is no longer a vmcore.#.gz file.
For the vmcore.#.gz that exists:

(gdb) target kvm /data/crash/vmcore.#
Cannot find a valid PCB

How do I get FreeNAS to keep the crash dump?

#14 Updated by Dru Lavigne over 2 years ago

  • Status changed from Unscreened to Investigation

#15 Updated by Craig Shiels over 2 years ago

I've just seen this panic when the system tried to unwind a transaction log on the disk as part of the pool import.

#16 Updated by Dru Lavigne over 2 years ago

  • Status changed from Investigation to Not Started

#17 Avatar?id=13649&size=24x24 Updated by Ben Gadd over 2 years ago

  • Due date set to 03/09/2018

#18 Updated by Craig Shiels over 2 years ago

This panic has stopped occurring. I think the cause of this panic is bad data on the source system. How that happens I cannot say.

#19 Updated by Dru Lavigne over 2 years ago

  • Status changed from Not Started to Closed
  • Target version changed from 11.2-BETA1 to N/A
  • Reason for Closing set to Cannot Reproduce

Craig: I'll close out for now. If it reoccurs, please attach the new trace to this ticket.

Also available in: Atom PDF