Project

General

Profile

Bug #83580

Improve speed of py-libzfs when retrieving many datasets or snapshots

Added by William Grzybowski 5 months ago. Updated 5 months ago.

Status:
Done
Priority:
No priority
Assignee:
Waqar Ahmed
Category:
Middleware
Target version:
Seen in:
Severity:
Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Currently py-libzfs is extremely slow with a huge set of datasets or snapshots

It can be compared with CLI version:

"zfs get -t filesystem all"

vs

"[i.__getstate__() for i in libzfs.ZFS.datasets]"

and

"zfs get -t snapshot all"

vs

"[i.__getstate__() for i in libzfs.ZFS.snapshots]"

I suspect thats because we are retrieving each nvlist at a time and and building the python object for it.
We need the investigate if that can be optimized. One idea is to try and grab a chunk of nvlist at once (pure cython code, not mixed with python, and transform it later).

Another idea would be to serialize all of the ahead in pure cython (or even C) and only convert to python at the final stage, since when most of the times when we want a list of all datasets/snapshots we will only need it serialized anyway.

Associated revisions

Revision 10105f3f (diff)
Added by Waqar Ahmed 5 months ago

Improve datasets retrieval speed

This commit introduces a change to make sure that we are able to retrieve datasets faster then the current method we have for it. With the number of datasets going higher, the time being taken to retrieve their properties gets even higher. Changes introduces here make sure that the new speed is on level with the zfs cli.
Ticket: #83580

Revision 31ab6853 (diff)
Added by Waqar Ahmed 5 months ago

Improve datasets retrieval speed

This commit introduces a change to make sure that we are able to retrieve datasets faster then the current method we have for it. With the number of datasets going higher, the time being taken to retrieve their properties gets even higher. Changes introduces here make sure that the new speed is on level with the zfs cli.
Ticket: #83580

Revision cda2328b (diff)
Added by Waqar Ahmed 5 months ago

Improve snapshots retrieval speed

This commit introduces a change to make sure that we are able to retrieve snapshots faster then the current method we have for it. With the number of snapshots going higher, the time being taken to retrieve their properties gets even higher. Changes introduces here make sure that the new speed is on level with the zfs cli.
Ticket: #83580

Revision c53ca99c (diff)
Added by Waqar Ahmed 5 months ago

Improve datasets retrieval speed

This commit introduces a change to make sure that we are able to retrieve datasets faster then the current method we have for it. With the number of datasets going higher, the time being taken to retrieve their properties gets even higher. Changes introduces here make sure that the new speed is on level with the zfs cli.
Ticket: #83580

Revision 0038fde9 (diff)
Added by Waqar Ahmed 5 months ago

Improve snapshots retrieval speed

This commit introduces a change to make sure that we are able to retrieve snapshots faster then the current method we have for it. With the number of snapshots going higher, the time being taken to retrieve their properties gets even higher. Changes introduces here make sure that the new speed is on level with the zfs cli.
Ticket: #83580

History

#1 Updated by Waqar Ahmed 5 months ago

  • Status changed from Unscreened to Ready for Testing

#2 Updated by Dru Lavigne 5 months ago

  • Subject changed from Investigate possibility of increase speed of py-libzfs for a lot of snapshots/datasets to Improve speed of py-libzfs when retrieving many datasets or snapshots
  • Needs Doc changed from Yes to No
  • Needs Merging changed from Yes to No

#4 Updated by Dru Lavigne 5 months ago

  • Status changed from Ready for Testing to Done
  • Target version changed from 11.3-BETA1 to 11.3-ALPHA1
  • Needs QA changed from Yes to No

Also available in: Atom PDF