Project

General

Profile

Bug #10840

zfs-snmp using wrong/missing sysctl variables for arc/l2arc

Added by Matthew Dailey about 5 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Expected
Assignee:
Suraj Ravichandran
Category:
Middleware
Target version:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

In the code for /usr/local/bin/freenas-snmp/zfs-snmp, there are two items that are not being reported properly: zfsArcMeta and zfsL2ArcSize

zfsArcMeta (.1.3.6.1.4.1.25359.1.2.2.0) calls zfs_arc_meta(), that in turn calls "sysctl kstat.zfs.misc.arcstats.arc_meta_used". It looks like this variable doesn't exist in FreeBSD:
  1. sysctl kstat.zfs.misc.arcstats.arc_meta_used
    sysctl: unknown oid 'kstat.zfs.misc.arcstats.arc_meta_used'
    It looks like this is kept track of in /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c as arc_meta_used but not actually exposed via kstat. This should probably be removed from the MIB until/unless FreeBSD exposes this via kstat.
zfsL2ArcSize (.1.3.6.1.4.1.25359.1.3.5.0) calls zfs_l2arc_size(), that calls kstat("kstat.zfs.misc.arcstats.l2_size"). Since the kstat() function already adds kstat. to the oid, this is actually calling:
  1. sysctl kstat.kstat.zfs.misc.arcstats.l2_size
    sysctl: unknown oid 'kstat.kstat.zfs.misc.arcstats.l2_size'
    The solution is just to fix this line:

--- zfs-snmp.orig 2015-07-09 08:58:22.510818000 -0400
+++ zfs-snmp 2015-08-03 10:50:34.283279448 -0400
@ -221,7 +221,7 @

def zfs_l2arc_size(oid):
    # KB
- return ('gauge', kstat("kstat.zfs.misc.arcstats.l2_size") / 1024)
+ return ('gauge', kstat("zfs.misc.arcstats.l2_size") / 1024)
def zfs_l2arc_hits(oid):
zfs-snmp-l2arc.patch (327 Bytes) zfs-snmp-l2arc.patch Matthew Dailey, 08/03/2015 08:00 AM

Associated revisions

Revision 7b727829 (diff)
Added by Suraj Ravichandran about 5 years ago

Partial rework of zfs-snmp stats for utilizing py-libzfs instead of zpool parsing. Various other improvements include: 1. s/kstat("kstat.zfs.misc.arcstats.l2_size")/kstat("zfs.misc.arcstats.l2_size"). 2. Added new OID for Dataset level stats. 3. Made a bunch of zfsFilesystem --> zfsPool naming changes. 4. Changed MIB Syntax for pool health to string instead of integer. 5. Made pool size MIBs use actual pool 'size' property. 6. Made zvol size MIBs use zvol 'volsize' property. Thanks to Matthew Dailey for suggesting these changes (and even providing patches for them, unfortunately I could not use all his patches since I went the py-libzfs way but much appreciated). Ticket: #10840 Ticket: #10807

Revision ec1866b7 (diff)
Added by Suraj Ravichandran about 5 years ago

Partial rework of zfs-snmp stats for utilizing py-libzfs instead of zpool parsing. Various other improvements include: 1. s/kstat("kstat.zfs.misc.arcstats.l2_size")/kstat("zfs.misc.arcstats.l2_size"). 2. Added new OID for Dataset level stats. 3. Made a bunch of zfsFilesystem --> zfsPool naming changes. 4. Changed MIB Syntax for pool health to string instead of integer. 5. Made pool size MIBs use actual pool 'size' property. 6. Made zvol size MIBs use zvol 'volsize' property. Thanks to Matthew Dailey for suggesting these changes (and even providing patches for them, unfortunately I could not use all his patches since I went the py-libzfs way but much appreciated). Ticket: #10840 Ticket: #10807 (cherry picked from commit 7b7278296839be06e10fac35b5b083773eabbd1d)

Revision 6712a377 (diff)
Added by Suraj Ravichandran about 5 years ago

Partial rework of zfs-snmp stats for utilizing py-libzfs instead of zpool parsing. Various other improvements include: 1. s/kstat("kstat.zfs.misc.arcstats.l2_size")/kstat("zfs.misc.arcstats.l2_size"). 2. Added new OID for Dataset level stats. 3. Made a bunch of zfsFilesystem --> zfsPool naming changes. 4. Changed MIB Syntax for pool health to string instead of integer. 5. Made pool size MIBs use actual pool 'size' property. 6. Made zvol size MIBs use zvol 'volsize' property. Thanks to Matthew Dailey for suggesting these changes (and even providing patches for them, unfortunately I could not use all his patches since I went the py-libzfs way but much appreciated). Ticket: #10840 Ticket: #10807 (cherry picked from commit 7b7278296839be06e10fac35b5b083773eabbd1d)

History

#1 Updated by William Grzybowski about 5 years ago

  • Assignee changed from William Grzybowski to Suraj Ravichandran

#2 Updated by Matthew Dailey about 5 years ago

Should have used a pre for the patch:

--- zfs-snmp.orig    2015-07-09 08:58:22.510818000 -0400
+++ zfs-snmp    2015-08-03 10:50:34.283279448 -0400
@@ -221,7 +221,7 @@

 def zfs_l2arc_size(oid):
     # KB
-    return ('gauge', kstat("kstat.zfs.misc.arcstats.l2_size") / 1024)
+    return ('gauge', kstat("zfs.misc.arcstats.l2_size") / 1024)

 def zfs_l2arc_hits(oid):

Attaching as a file, too, just in case. :-)

#3 Updated by Suraj Ravichandran about 5 years ago

  • Status changed from Unscreened to Fix In Progress

#4 Updated by Suraj Ravichandran about 5 years ago

@Matthew:

Thanks for spotting the zfs_l2arc_size related s/kstat.zfs.misc.arcstats.l2_size/zfs.misc.arcstats.l2_size

As for the zfsArcMeta (.1.3.6.1.4.1.25359.1.2.2.0) did you try that systctl on freenas? Because here is my output of trying it on one of the nightlies (freenas 9.3 nightlies) and i am pretty sure it works on stable too:

[root@freenas93] /usr/local/bin/freenas-snmp# sysctl kstat.zfs.misc.arcstats.arc_meta_used
kstat.zfs.misc.arcstats.arc_meta_used: 28322080

#5 Updated by Suraj Ravichandran about 5 years ago

  • Status changed from Fix In Progress to 19
  • Priority changed from No priority to Expected
  • Target version set to Unspecified
  • % Done changed from 0 to 100

#6 Updated by Matthew Dailey about 5 years ago

@Suraj:

I swore I had tried this on FreeNAS, but I think perhaps I only tried on TrueNAS. I'm getting inconsistent results between these, as well as on a couple of "pure" FreeBSD systems.

FreeNAS 9.3 (cloned from git master on 7/29):

freenas-test# uname -a
FreeBSD freenas-test 9.3-RELEASE-p21 FreeBSD 9.3-RELEASE-p21 #0 r281084+5c217d3: Wed Jul 29 09:56:43 EDT 2015     root@schultz-freebsd93:/usr/home/mdailey/freenas/objs/os-base/amd64/usr/home/mdailey/freenas/FreeBSD/src/sys/FREENAS.amd64  amd64
freenas-test# sysctl kstat.zfs.misc.arcstats.arc_meta_used
kstat.zfs.misc.arcstats.arc_meta_used: 37043592

TrueNAS 9.3 (TrueNAS-9.3-STABLE-201506300138):

vmnas1# uname -a
FreeBSD vmnas1 9.3-RELEASE-p16 FreeBSD 9.3-RELEASE-p16 #2 r281084+50d25c4: Thu Jun 25 11:32:48 PDT 2015     root@build3.ixsystems.com:/tank/home/jkh/build/TN/objs/os-base/amd64/tank/home/jkh/build/TN/FreeBSD/src/sys/TRUENAS.amd64  amd64
vmnas1# sysctl kstat.zfs.misc.arcstats.arc_meta_used
sysctl: unknown oid 'kstat.zfs.misc.arcstats.arc_meta_used'

FreeBSD 9.3:

schultz-freebsd93# uname -a
FreeBSD schultz-freebsd93 9.3-RELEASE-p21 FreeBSD 9.3-RELEASE-p21 #0: Tue Jul 28 09:02:48 UTC 2015     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
schultz-freebsd93# sysctl kstat.zfs.misc.arcstats.arc_meta_used
sysctl: unknown oid 'kstat.zfs.misc.arcstats.arc_meta_used'

FreeBSD 10.1:

root@pail:~# uname -a
FreeBSD pail 10.1-RELEASE-p6 FreeBSD 10.1-RELEASE-p6 #0: Tue Feb 24 19:00:21 UTC 2015     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
root@pail:~# sysctl kstat.zfs.misc.arcstats.arc_meta_used
sysctl: unknown oid 'kstat.zfs.misc.arcstats.arc_meta_used': No such file or directory

I'm not sure what could be going on here, unless FreeNAS is doing something in the kernel compile that none of the others are.

#7 Updated by Suraj Ravichandran about 5 years ago

Matthew Dailey wrote:

@Suraj:

I'm not sure what could be going on here, unless FreeNAS is doing something in the kernel compile that none of the others are.

Yes FreeNAS does that a lot!. The upcoming TrueNAS and FreeNAS Updates should resolve this issue (of the discrepancy in that sysctl working in FreeNAS and not in TrueNAS)

#8 Updated by Xin Li about 5 years ago

Matthew Dailey wrote:

@Suraj:

I swore I had tried this on FreeNAS, but I think perhaps I only tried on TrueNAS. I'm getting inconsistent results between these, as well as on a couple of "pure" FreeBSD systems.

[...]

I'm not sure what could be going on here, unless FreeNAS is doing something in the kernel compile that none of the others are.

FreeNAS's ZFS is more up-to-date with FreeBSD -CURRENT.

This kstat variable was introduced in FreeBSD r277583 (stable/10) and r275748 (head) and will be available in FreeBSD 10.2, and before that it was called vfs.zfs.arc_meta* due to a change in ARC to expose more useful data.

#9 Updated by Suraj Ravichandran about 5 years ago

  • Status changed from 19 to Ready For Release
  • ChangeLog Entry updated (diff)

#10 Updated by Matthew Dailey about 5 years ago

@Xin
Thanks so much for the explanation and even providing the specific revs - this makes perfect sense and explains what I'm seeing.

@Suraj
Thanks for your work on this. I'll look forward to trying out your new py-libzfs based code and hopefully to seeing this (as well as the kstat change) in the TrueNAS train.

#11 Updated by Jordan Hubbard about 5 years ago

  • Status changed from Ready For Release to Resolved

#12 Avatar?id=14398&size=24x24 Updated by Kris Moore about 4 years ago

  • Target version changed from Unspecified to N/A

Also available in: Atom PDF