Project

General

Profile

Bug #27334

Pass SNMP Auxiliary parameters values to running configuration

Added by Mike Ely over 1 year ago. Updated over 1 year ago.

Status:
Done
Priority:
Expected
Assignee:
Vladimir Vinogradenko
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

After update, SNMP monitoring stopped working on all FreeNAS hosts. After debugging determined that the aux params were being ignored.

Contents of /usr/local/etc/snmpd.conf before update:

agentAddress udp:161,udp6:161,unix:/var/run/snmpd.sock
sysLocation [value filled in by user]
sysContact [ditto]
sysDescr Hardware: amd64 Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz running at 2100 Software: FreeBSD 11.0-STABLE

pass .1.3.6.1.4.1.25359.1 /usr/local/bin/freenas-snmp/zfs-snmp
rocommunity "public" default
extend 'fnalert' /mnt/tank/user-custom/bin/alert-wrapper

load 30.0 15.0 10.0

proc cron

After update:
agentAddress udp:161,udp6:161,unix:/var/run/snmpd.sock
sysLocation [value filled in by user]
sysContact [ditto]
sysDescr Hardware: amd64 Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz running at 2100 Software: FreeBSD 11.1-STABLE (revision 199506)

master agentx
rocommunity "public" default

Notice anything missing? This broke our monitoring. I've temporarily patched things back by editing the snmpd.conf by hand (which won't survive a reboot) and restarting the daemon, but this is a pretty significant bug.


Related issues

Has duplicate FreeNAS - Bug #27379: SNMP auxiliary parameters in the legacy&new GUI don't get saved to snmpd.confClosed: Duplicate2017-12-22
Has duplicate FreeNAS - Bug #27871: Pass SNMP Auxiliary parameters values to running configurationClosed2018-01-19
Has duplicate FreeNAS - Bug #28441: Adding auxiliary parameters breaks snmpd.Closed2018-02-12

Associated revisions

Revision db48ee0a (diff)
Added by Vladimir Vinogradenko over 1 year ago

fix(snmp): Fix SNMP "Auxiliary parameters" values not being passed into working config
(lost in b8bf7e3de2fd3821010685dbcb3c215ec067fbb1)

Ticket: #27334

Revision a4ca3400 (diff)
Added by Vladimir Vinogradenko over 1 year ago

fix(snmp): implement _reload_snmp so it config would be regenerated after service update

Ticket: #27334

Revision 54fdb084 (diff)
Added by Vladimir Vinogradenko over 1 year ago

fix(snmp): Fix snmpd logging

TIcket: #27334

Revision 7235b3f1 (diff)
Added by Vladimir Vinogradenko over 1 year ago

fix(snmp): Fix SNMP "Auxiliary parameters" values not being passed into working config
(lost in b8bf7e3de2fd3821010685dbcb3c215ec067fbb1)

Ticket: #27334

Revision c512939b (diff)
Added by Vladimir Vinogradenko over 1 year ago

fix(snmp): implement _reload_snmp so it config would be regenerated after service update

Ticket: #27334

Revision 3faa1fc6 (diff)
Added by Vladimir Vinogradenko over 1 year ago

fix(snmp): Fix snmpd logging

TIcket: #27334

Revision c29d71fe (diff)
Added by Vladimir Vinogradenko over 1 year ago

fix(snmp): Fix passing multiline SNMP options

Ticket: #27334

Revision 2158c812 (diff)
Added by Vladimir Vinogradenko over 1 year ago

fix(snmp): Fix passing multiline SNMP options

Ticket: #27334

Revision 13d6e60b (diff)
Added by Vladimir Vinogradenko over 1 year ago

fix(snmp): Fix passing multiline SNMP options

Ticket: #27334

Revision 1b298045 (diff)
Added by Vladimir Vinogradenko over 1 year ago

fix(snmp): Fix passing multiline SNMP options

Ticket: #27334

Revision 15b732cc (diff)
Added by Vladimir Vinogradenko over 1 year ago

fix(snmp): Fix passing multiline SNMP options

Ticket: #27334

History

#1 Updated by Dru Lavigne over 1 year ago

  • Status changed from Unscreened to 15

Mike: please attach a debug (System -> Advanced -> Save Debug).

#2 Updated by Mike Ely over 1 year ago

  • File debug-a-20171220091607.tgz added
  • Private changed from No to Yes

Dru Lavigne wrote:

Mike: please attach a debug (System -> Advanced -> Save Debug).

Debug attached. Please note that I've manually edited /usr/local/etc/snmpd.conf to match values needed by us (with full knowledge that this will get wiped on a reboot).

#3 Updated by Mike Ely over 1 year ago

As an aside, please consider providing an alternate means for sending debug files which a user does not want publicly available. I've had to mark this bug private since Redmine does not have a facility to support marking individual attachments private:
http://www.redmine.org/issues/12677

#4 Updated by Dru Lavigne over 1 year ago

  • Status changed from 15 to Unscreened

William: please load balance.

#5 Updated by Mike Ely over 1 year ago

Additional:

The "Log level" drop down doesn't seem to be honored either. I've tried setting it to Debug and running snmpwalk, but nothing shows in /var/log/messages as would be expected.

#6 Updated by Dru Lavigne over 1 year ago

  • Assignee changed from Release Council to William Grzybowski

#7 Updated by William Grzybowski over 1 year ago

  • Assignee changed from William Grzybowski to Vladimir Vinogradenko

#8 Updated by William Grzybowski over 1 year ago

  • Has duplicate Bug #27379: SNMP auxiliary parameters in the legacy&new GUI don't get saved to snmpd.conf added

#9 Updated by Mike Ely over 1 year ago

  • File deleted (debug-a-20171220091607.tgz)

#10 Updated by Mike Ely over 1 year ago

  • Private changed from Yes to No

I've removed the debug file so this bug can be made public again. If Redmine were able to create private attachments this would not have been necessary.

Bug maintainer: please give an alternate path for me to send the debug file your way once you're ready to work this issue.

#11 Updated by Vladimir Vinogradenko over 1 year ago

  • Status changed from Unscreened to Needs Developer Review
  • Assignee changed from Vladimir Vinogradenko to William Grzybowski
  • Target version set to 11.1-U1

Fixed.

snmpd messages will appear in /var/log/daemon.log

#12 Updated by William Grzybowski over 1 year ago

  • Status changed from Needs Developer Review to Reviewed by Developer
  • Assignee changed from William Grzybowski to Vladimir Vinogradenko

#13 Updated by Vladimir Vinogradenko over 1 year ago

  • Status changed from Reviewed by Developer to Ready For Release

#14 Updated by Dru Lavigne over 1 year ago

  • Subject changed from SNMP "Auxiliary parameters" values not being passed into working config to Pass SNMP Auxiliary parameters values to running configuration

#15 Updated by Dru Lavigne over 1 year ago

  • Status changed from Ready For Release to Resolved

#16 Updated by Mike Ely over 1 year ago

  • Seen in changed from 11.1 to 11.1-U1

Updated today, and the bug persists in 11.1-U1. Not fixed at all. Sorry.

Tested on multiple arrays including the one we bought from IX Systems. All are failing to pass aux params from the webUI to the actual working SNMP daemon, so I have to edit the file by hand and manually restart the service, which gets wiped on every reboot. Still.

Please provide a private upload link so I can send the debug file.

#17 Avatar?id=14398&size=24x24 Updated by Kris Moore over 1 year ago

  • Status changed from Resolved to Unscreened
  • Target version changed from 11.1-U1 to 11.1-U2

#18 Updated by Dru Lavigne over 1 year ago

  • Has duplicate Bug #27871: Pass SNMP Auxiliary parameters values to running configuration added

#19 Updated by Dru Lavigne over 1 year ago

  • Status changed from Unscreened to Not Started

#20 Updated by Vladimir Vinogradenko over 1 year ago

  • Status changed from Not Started to Blocked
  • Reason for Blocked set to Waiting for feedback

Mike, sorry, I am not able to repeat this on 11.1-U1. Parameters are passed.

Did you do re-saving SNMP options in UI? Because it won't regenerate config file without doing it.

#21 Updated by Mike Ely over 1 year ago

  • Reason for Blocked deleted (Waiting for feedback)

That was a worthy experiment. I just tried slightly modifying one of the params in the UI, saving the file, and then rebooting the array. Unfortunately that did not work: the array came online with the default snmpd.conf:

agentAddress udp:161,udp6:161,unix:/var/run/snmpd.sock
sysLocation proc cron
sysContact unknown@localhost
sysDescr Hardware: amd64 Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz running at 2100 Software: FreeBSD 11.1-STABLE (revision 199506)

master agentx
rocommunity "" default

Please indicate next steps or provide a non-Redmine location to upload the debug file from the array (since Redmine stupidly does not support private attachments to issues).

#22 Updated by Vladimir Vinogradenko over 1 year ago

  • Reason for Blocked set to Waiting for feedback

Mike, sadly debug log won't help here.

Was file modified after changing params in the UI (before system reboot)?

What file are we talking about. /etc/local/snmpd.conf?

Could you please upload your copy of /etc/ix.rc.d/ix-snmpd?

#23 Updated by Mike Ely over 1 year ago

  • Reason for Blocked deleted (Waiting for feedback)

Vladimir Vinogradenko wrote:

Was file modified after changing params in the UI (before system reboot)?

I just tested this, and what happens is when I modify the params in the UI and save, the console echoes out "Received TERM or STOP signal... shutting down..."
and the file /etc/local/snmpd.conf is reset back to defaults. All my params get wiped at that time.

What file are we talking about. /etc/local/snmpd.conf?

Yes.

Could you please upload your copy of /etc/ix.rc.d/ix-snmpd?

#!/bin/sh                                                                                                                                              
#                                                                                                                                                      
# $FreeBSD$                                                                                                                                                    
#

# PROVIDE: ix-snmpd
# REQUIRE: FILESYSTEMS NETWORKING
# REQUIRE: openssh
# BEFORE: snmpd

. /etc/rc.subr

snmp_config=/etc/local/snmpd.conf

generate_snmpconf()
{
        local IFS="|" 
        local f="snmp_location snmp_contact snmp_community snmp_v3 snmp_v3_username snmp_v3_authtype snmp_v3_password snmp_v3_privproto snmp_v3_privpassphrase snmp_options" 
        eval local $f
        local sf=$(var_to_sf $f)
        RO_FREENAS_CONFIG=$(ro_sqlite ${name} 2> /tmp/${name}.fail && rm /tmp/${name}.fail)
        trap 'rm -f ${RO_FREENAS_CONFIG}' EXIT

        umask 077
        hw_machine=`/sbin/sysctl -n hw.machine`
        hw_model=`/sbin/sysctl -n hw.model`
        hw_clockrate=`/sbin/sysctl -n hw.clockrate`
        kern_ostype=`/sbin/sysctl -n kern.ostype`
        kern_osrelease=`/sbin/sysctl -n kern.osrelease`
        kern_osrevision=`/sbin/sysctl -n kern.osrevision`

        ${FREENAS_SQLITE_CMD} ${RO_FREENAS_CONFIG} "SELECT $sf FROM services_snmp ORDER BY -id LIMIT 1" | \
        while eval read -r $f; do
                cat > "${snmp_config}" << EOF
agentAddress udp:161,udp6:161,unix:/var/run/snmpd.sock
sysLocation ${snmp_location:-unknown}
sysContact ${snmp_contact:-unknown@localhost}
sysDescr Hardware: ${hw_machine} ${hw_model} running at ${hw_clockrate} Software: ${kern_ostype} ${kern_osrelease} (revision ${kern_osrevision})

master agentx
EOF

                if [ ${snmp_v3} -eq 1 ]; then

                        if [ "${snmp_v3_username}" != "" -a "${snmp_v3_password}" != "" ]; then
                                echo -n "createUser ${snmp_v3_username} ${snmp_v3_authtype} \"${snmp_v3_password}\"" >> ${snmp_config}
                                if [ -n "${snmp_v3_privproto}" -a -n "${snmp_v3_privpassphrase}" ]; then
                                        echo " ${snmp_v3_privproto} \"${snmp_v3_privpassphrase}\"" >> ${snmp_config}
                                else
                                        echo >> ${snmp_config}
                                fi
                                echo "rwuser ${snmp_v3_username}" >> ${snmp_config}
                        fi

                else
                        echo "rocommunity \"${snmp_community}\" default" >> ${snmp_config}
                fi

                echo "${snmp_options}" >> ${snmp_config}
        done
}

name="ix-snmpd" 
start_cmd='generate_snmpconf'
stop_cmd=':'

load_rc_config $name

#24 Updated by Mike Ely over 1 year ago

For what it's worth, here's a slightly redacted copy of what we're putting in the aux params section:

extend 'fnalert' /mnt/tank/custom-scripts/bin/alert-wrapper

load 40.2 20.0 18.0

proc cron

The syslocation, contact, and community fields above that are filled in (although those params don't get properly saved either).

#25 Updated by Vladimir Vinogradenko over 1 year ago

Mike, one more debug command please:

echo "select * from services_snmp" | sqlite3 /data/freenas-v1.db

#26 Updated by Mike Ely over 1 year ago

Vladimir Vinogradenko wrote:

Mike, one more debug command please:

[...]

The output looks like what we want in snmpd.conf:

1|Redacted Our Syslocation|redacted@redacted.example|0|0|communityname||||||extend 'fnalert' /mnt/tank/custom-scripts/bin/alert-wrapper

load 40.2 20.0 18.0

proc cron|3

#27 Updated by Vladimir Vinogradenko over 1 year ago

  • Status changed from Blocked to Done

Mike, thank you for your help, hope this is finally fixed now.

#28 Updated by Dru Lavigne over 1 year ago

  • Needs Doc changed from Yes to No
  • Needs Merging changed from Yes to No

#29 Updated by Mike Ely over 1 year ago

  • Priority changed from No priority to Expected
  • QA Status Test Fails FreeNAS added
  • QA Status deleted (Not Tested)

Vladimir Vinogradenko wrote:

Mike, thank you for your help, hope this is finally fixed now.

Did I miss something here? You asked for more information, I provided that, and you say it's fixed?

It is not fixed. When SNMPD is restarted by FreeNAS either through the WebUI or a system reboot, snmpd.conf is set to the default values from /etc/ix.rc.d/ix-snmpd and not the values which are being stored in the sqlite database.

We have four arrays, including one that we purchased new from IXSystems, and they are all exhibiting the same behavior. Please reopen this bug and please do not close it while the issue persists.

#30 Updated by Dru Lavigne over 1 year ago

Mike: the code that was committed 10 hours ago (on this ticket) should resolve the issue and will be available in the next update (target version).

#31 Updated by Mike Ely over 1 year ago

Dru Lavigne wrote:

Mike: the code that was committed 10 hours ago (on this ticket) should resolve the issue and will be available in the next update (target version).

Ah, so. Sorry, apparently email notifications don't include linked patch notifications.

#32 Updated by Mike Ely over 1 year ago

  • QA Status Not Tested added
  • QA Status deleted (Test Fails FreeNAS)

Is there any way to make the changes to ...ix.rc.d/ix-snmpd persist across reboots prior to the patch install? I can confirm that the changed ix-snmpd does fix the problem of changes to webUI nulling out snmpd.conf but the manual changes to ix-snmpd are wiped on reboot.

#33 Avatar?id=13649&size=24x24 Updated by Ben Gadd over 1 year ago

  • Due date set to 02/12/2018

Due date updated to reflect the code freeze for 11.1U2.

#34 Avatar?id=13649&size=24x24 Updated by Ben Gadd over 1 year ago

  • Severity set to New

#35 Updated by Dru Lavigne over 1 year ago

  • Has duplicate Bug #28441: Adding auxiliary parameters breaks snmpd. added

Also available in: Atom PDF