Project

General

Profile

Bug #19463

pool import is slow due to many ZVOLs/snapshots

Added by Vaibhav Chauhan about 3 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Important
Assignee:
Alexander Motin
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

so updating from 9.10.1-U4 to 9.10.2-INTERNAL2, presents you with following.
Machine in question,
https://stable-shark.ixsystems.com/ (root/meh)
mode of update: FreeNAS webui updater using update-int.ixsystems.com
IMPI: 10.212.3.8 (admin/ADMIN)

steps to reproduce:
rollback https://stable-shark.ixsystems.com/ to U4 release, apply update to FreeNAS using webui updater using update-int.ixsystems.com. you should keep IMPI session handy, IMPI: 10.212.3.8 (admin/ADMIN)

1. takes a while to import a single pool > 3-4 min

Associated revisions

Revision 3688764d (diff)
Added by Alexander Motin almost 3 years ago

Postpone ZVOL media/block size caching till first open. At least on FreeBSD there are no legal way to access media or get its size without opening device/provider first. Postponing this caching allows to skip several disk seeks per ZVOL/snapshot during import. For HDD pool with 1 ZVOL in dev mode with 1000 snapshots this reduces pool import time from 40 to 10 seconds. Ticket: #19463 (cherry picked from commit f780634303e61bc4b311a2781f9e7c48cdcba072)

Revision 3688764d (diff)
Added by Alexander Motin almost 3 years ago

Postpone ZVOL media/block size caching till first open. At least on FreeBSD there are no legal way to access media or get its size without opening device/provider first. Postponing this caching allows to skip several disk seeks per ZVOL/snapshot during import. For HDD pool with 1 ZVOL in dev mode with 1000 snapshots this reduces pool import time from 40 to 10 seconds. Ticket: #19463 (cherry picked from commit f780634303e61bc4b311a2781f9e7c48cdcba072)

Revision 47645f1c (diff)
Added by Alexander Motin almost 3 years ago

Postpone ZVOL media/block size caching till first open. At least on FreeBSD there are no legal way to access media or get its size without opening device/provider first. Postponing this caching allows to skip several disk seeks per ZVOL/snapshot during import. For HDD pool with 1 ZVOL in dev mode with 1000 snapshots this reduces pool import time from 40 to 10 seconds. Ticket: #19463 (cherry picked from commit f780634303e61bc4b311a2781f9e7c48cdcba072) (cherry picked from commit 3688764d45b1f5eedf1d62fe47e0a8940ff85820)

Revision 47645f1c (diff)
Added by Alexander Motin almost 3 years ago

Postpone ZVOL media/block size caching till first open. At least on FreeBSD there are no legal way to access media or get its size without opening device/provider first. Postponing this caching allows to skip several disk seeks per ZVOL/snapshot during import. For HDD pool with 1 ZVOL in dev mode with 1000 snapshots this reduces pool import time from 40 to 10 seconds. Ticket: #19463 (cherry picked from commit f780634303e61bc4b311a2781f9e7c48cdcba072) (cherry picked from commit 3688764d45b1f5eedf1d62fe47e0a8940ff85820)

Revision 7f048630 (diff)
Added by Alexander Motin almost 3 years ago

Postpone ZVOL media/block size caching till first open. At least on FreeBSD there are no legal way to access media or get its size without opening device/provider first. Postponing this caching allows to skip several disk seeks per ZVOL/snapshot during import. For HDD pool with 1 ZVOL in dev mode with 1000 snapshots this reduces pool import time from 40 to 10 seconds. Ticket: #19463 (cherry picked from commit f780634303e61bc4b311a2781f9e7c48cdcba072)

Revision 7f048630 (diff)
Added by Alexander Motin almost 3 years ago

Postpone ZVOL media/block size caching till first open. At least on FreeBSD there are no legal way to access media or get its size without opening device/provider first. Postponing this caching allows to skip several disk seeks per ZVOL/snapshot during import. For HDD pool with 1 ZVOL in dev mode with 1000 snapshots this reduces pool import time from 40 to 10 seconds. Ticket: #19463 (cherry picked from commit f780634303e61bc4b311a2781f9e7c48cdcba072)

History

#1 Updated by William Grzybowski about 3 years ago

  • Status changed from Unscreened to Screened

Does it happen every time you boot?

#2 Updated by Vaibhav Chauhan about 3 years ago

I cannot say this for certain as this happened right during update-reboot process.

#3 Updated by William Grzybowski about 3 years ago

  • Status changed from Screened to Unscreened
  • Assignee changed from William Grzybowski to Alexander Motin

mav, I am unsure what to do with this ticket. Regardless this seem out of depth for me since from his description, this is a pure ZFS thing.

Feel free to close if you don't think it is worth investigating.

Thank you!

#4 Updated by Alexander Motin about 3 years ago

  • Status changed from Unscreened to Screened

I don't think this problem is related to the new version update. I've reverted to previous boot environment of 9.10.1-U4 and got the same pool import time of several minutes. I haven't proven yet what is the import delay cause, but I found huge delays of `zpool list t snapshot`, caused by their number - there are 86762 snapshots on that pool, and attempt to list them first time causes huge delays due to disk seek time bottleneck.

#5 Updated by Alexander Motin about 3 years ago

  • Status changed from Screened to Closed: Behaves correctly

Most of time spent during the pool import is spent on creation of devfs entries for 3300 zvol snapshots. Creation of each one require respective zvol properties to be read, performance of which is trivially limited by seek time of the disks. I don't think we can do much about this, rather then tell to not create so many snapshots (every 5 minutes for 2 weeks), especially for zvols.

#6 Updated by Alexander Motin almost 3 years ago

  • Subject changed from pool import during boot process is slow to pool import is slow due to many ZVOLs/snapshots
  • Status changed from Closed: Behaves correctly to Fix In Progress

I've found a way to improve pool import time in case of many ZVOLs/snapshots by several times (from 40 to 10 seconds on my test setup with 1 ZVOL and 1000 snapshots).

#7 Updated by Alexander Motin almost 3 years ago

  • Status changed from Fix In Progress to 19
  • Priority changed from Expected to Important

#8 Updated by Alexander Motin almost 3 years ago

  • Status changed from 19 to Ready For Release

#9 Updated by Dru Lavigne almost 2 years ago

  • Status changed from Ready For Release to Resolved

Also available in: Atom PDF