Project

General

Profile

Bug #40712

Reduce taskqueue congestion caused by ZFS ZIO pipeline activity

Added by Alexander Motin over 1 year ago. Updated over 1 year ago.

Status:
Done
Priority:
No priority
Assignee:
Alexander Motin
Category:
OS
Target version:
Seen in:
Severity:
Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

ZFS has concept of parent-child ZIOs to guide I/O requests through different stack layers. It is typical to have at least 3 ZIO per I/O. In many cases such submissions and completions are handled asynchronously via taskqueue(9) mechanism, that makes the last to be called multiple times per each I/O, that creates heavy lock congestion burning much CPU time and increasing latency at high IOPS no matter how many other measures are taken.

Associated revisions

Revision c99ff0d8 (diff)
Added by Alexander Motin over 1 year ago

Reduce taskq and context-switch cost of zio pipe When doing a read from disk, ZFS creates 3 ZIO's: a zio_null(), the logical zio_read(), and then a physical zio. Currently, each of these results in a separate taskq_dispatch(zio_execute). On high-read-iops workloads, this causes a significant performance impact. By processing all 3 ZIO's in a single taskq entry, we reduce the overhead on taskq locking and context switching. We accomplish this by allowing zio_done() to return a "next zio to execute" to zio_execute(). This results in a ~12% performance increase for random reads, from 96,000 iops to 108,000 iops (with recordsize=8k, on SSD's). Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed by: George Wilson <george.wilson@delphix.com> Signed-off-by: Matthew Ahrens <mahrens@delphix.com> External-issue: DLPX-59292 Closes #7736 zfsonlinux/zfs@62840030a7dceaee013ddbcc1eebcfc7922edf7c Ticket: #40712 (cherry picked from commit 592eca31793c2701bb921e959e0cf7894f9d076f)

History

#1 Updated by Alexander Motin over 1 year ago

  • Status changed from Unscreened to Ready for Testing

ZoL patch merged through FreeBSD head to freenas/11-stable branch: https://github.com/freenas/os/commit/c99ff0d882d586f6952e9dde17a0cf571e7f2f07

Testing may just include basic ZFS operation under load and in different scenarios. Performance testing we can leave to upstream and nap@.

#2 Updated by Bonnie Follweiler over 1 year ago

  • Status changed from Ready for Testing to Passed Testing
  • Needs QA changed from Yes to No

#4 Updated by Dru Lavigne over 1 year ago

  • Status changed from Passed Testing to Done

Also available in: Atom PDF