Project

General

Profile

Bug #42018

Prevent hold on file descriptors that stalls plugin installation

Added by Martin Wilke over 1 year ago. Updated over 1 year ago.

Status:
Done
Priority:
No priority
Assignee:
Brandon Schneider
Category:
Middleware
Target version:
Severity:
Medium
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Sometimes iocage stops with <stdin> and nothing happened. Example https://builds.ixsystems.com/jenkins/job/iocage%20-%20plugin-single-test/1/console I did abound it after 1 hour.

History

#1 Updated by Brandon Schneider over 1 year ago

  • Subject changed from iocage - stalling builds to iocage plugins stalling jenkins when they hold onto stdin or stdout.
  • Category changed from Console to Middleware
  • Status changed from Unscreened to In Progress
  • Target version changed from Backlog to 11.2-BETA3
  • Seen in changed from Unspecified to Master - FreeNAS Nightlies
  • Severity changed from New to Medium

#2 Updated by Brandon Schneider over 1 year ago

iocage PR: https://github.com/freenas/iocage/pull/18
DESC: Some plugins (quasselcore, bacula-server) hold onto these descriptors and that stalls the plugin installation.
RISK: Low
ACCEPTANCE: Plugins still install, and those plugins no longer stall

#3 Updated by Dru Lavigne over 1 year ago

  • Subject changed from iocage plugins stalling jenkins when they hold onto stdin or stdout. to Prevent hold on file descriptors that stalls plugin installation
  • Needs Doc changed from Yes to No

#4 Updated by Brandon Schneider over 1 year ago

  • Status changed from In Progress to Ready for Testing

#5 Updated by Brandon Schneider over 1 year ago

  • Needs Merging changed from Yes to No

#6 Updated by Jurgen Segaert over 1 year ago

Hi Brandon,
This change seems to have some weird side effects, i.e. not all of the output that is issued in post_install.sh is displayed. Example:
The last lines of post_install.sh of the gitlab plugin are as follows:
https://github.com/freenas/iocage-plugin-gitlab/blob/master/post_install.sh

echo "Starting nginx..." 
service nginx start
echo "Starting gitlab..." 
service gitlab start
echo "Starting gltlab pages..." 
service gitlab_pages start
echo "Starting sshd..." 
service sshd start

echo "Database Name: $DB" 
echo "Database User: $USER" 
echo "Database Password: $PASS" 
echo "Please open the URL to set your password, Login Name is root." 

... but when installing the plugin from the command line:
iocage fetch --plugins -n gitlab ip4_addr="em0|192.168.0.210/24" --branch master

...some of the information that is echo'ed, e.g. in this example the user and password, doesn't show:
(...)
Starting nginx...
Performing sanity check on nginx configuration:
Starting nginx.
Starting gitlab...
Regenerate Gitlab Gemfile.lock
Regenerate Gitaly Gemfile.lock
Starting GitLab Unicorn
Starting GitLab Sidekiq
Starting GitLab Workhorse
Starting Gitaly
...
The GitLab Unicorn web server with pid 15867 is running.
The GitLab Sidekiq job dispatcher with pid 15938 is running.
The GitLab Workhorse with pid 15902 is running.
Gitaly with pid 15900 is running.
GitLab and all its components are up and running.
Starting gltlab pages...
Starting sshd...
Generating RSA host key.
2048 SHA256:H5cpfCpV0MScQKWtP4MneTwe5J47ThaG/vRs7SKbjGE root@gitlab (RSA)
Generating ECDSA host key.
256 SHA256:W+I+QKD3TWNBRZXZgsTHJbEekDr/CZF9qPUaXHk0Gto root@gitlab (ECDSA)
Generating ED25519 host key.
256 SHA256:afm2KzX57V04wfFV/inVD1EFrQmdM84+mMrCIlR/p00 root@gitlab (ED25519)
Performing sanity check on sshd configuration.
Starting sshd.
Database Name: gitlabhq_production

Admin Portal:
http://192.168.0.210

This seems to be non-deterministic; different runs will output a different set of lines, but there's always something missing.

#7 Updated by Brandon Schneider over 1 year ago

  • Status changed from Ready for Testing to In Progress

Odd I hadn't seen any missing output on my tests. Will see what's happening.

Thanks Jurgen!

#8 Updated by Brandon Schneider over 1 year ago

Jurgen: Are you able to try the commit https://github.com/iocage/iocage/commit/99ba2a55e6da56a15db2af5a2959dc941b0559b3 and see if it resolves your issue before I merge it into the FreeNAS tree? Unless you're on an SDK image, you'll need to comment out line 10 of the Makefile and lines 1-2 of the requirements-dev.txt before the makefile will install for you (assuming you test this on FreeNAS, I'm not sure which machine you test your PR's on)

If it's easier, we can merge it into the FreeNAS tree and you can wait for a nightly to roll with it, however, I cannot reproduce the missing output, before or after these commits. So I'm suspecting possibly CPU speed is playing into this, what CPU are you using?

- Brandon

My output:

root@freenas:~ # iocage clean -jf ; iocage fetch -P -n gitlab dhcp=on bpf=yes vnet=on --branch master
Cleaning iocage/jails
All iocage jail datasets have been destroyed.
Plugin: GitLab
  Official Plugin: True
  Using RELEASE: 11.2-RELEASE
  Using Branch: master
  Post-install Artifact: https://github.com/freenas/iocage-plugin-gitlab.git
  These pkgs will be installed:
    - www/gitlab
    - databases/postgresql95-server
    - databases/postgresql95-contrib
    - www/nginx

Testing SRV response to iocage-plugins
Testing DNSSEC response to iocage-plugins

Installing plugin packages:
  - www/gitlab... 
  - databases/postgresql95-server... 
  - databases/postgresql95-contrib... 
  - www/nginx... 

Fetching artifact... 

Running post_install.sh

Command output:
gitlab_enable:  -> YES
gitlab_pages_enable:  -> YES
postgresql_enable:  -> YES
redis_enable:  -> YES
nginx_enable:  -> YES
sshd_enable: NO -> YES
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /usr/local/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/local/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start

LOG:  ending log output to stderr
HINT:  Future log output will go to log destination "syslog".
CREATE ROLE
CREATE DATABASE
ALTER ROLE
CREATE EXTENSION
LOG:  ending log output to stderr
HINT:  Future log output will go to log destination "syslog".
redis does not exist in /etc/rc.d or the local startup
directories (/usr/local/etc/rc.d), or is not executable
Starting nginx...
Performing sanity check on nginx configuration:
Starting nginx.
Starting gitlab...
gitlab does not exist in /etc/rc.d or the local startup
directories (/usr/local/etc/rc.d), or is not executable
Starting gltlab pages...
gitlab_pages does not exist in /etc/rc.d or the local startup
directories (/usr/local/etc/rc.d), or is not executable
Starting sshd...
Generating RSA host key.
2048 SHA256:wbpV6UJUKTmJmFPhgPXY2LdlQN0nW5WY7/gQuvKV6ac root@gitlab (RSA)
Generating ECDSA host key.
256 SHA256:qRST/k9l0Q/fTDOX5DRPlKa4XidZBZJFVM3Hz5zPXmA root@gitlab (ECDSA)
Generating ED25519 host key.
256 SHA256:3jLttIILRaun3v/u1ADCsvtEbN8+d36+TSeLemzD1NI root@gitlab (ED25519)
Performing sanity check on sshd configuration.
Starting sshd.
Database Name: gitlabhq_production
Database User: git
Database Password: Rb5ryG2hfEAbSq7S
Please open the URL to set your password, Login Name is root.

Admin Portal:
http://192.168.122.63

#9 Updated by Jurgen Segaert over 1 year ago

Thank you, I'll run a few tests later today after I'm done with my day job.

#10 Updated by Brandon Schneider over 1 year ago

  • Status changed from In Progress to Ready for Testing

Jurgen made a change that fixes the behavior he saw over in iocage/iocage. I brought that into the PR, which means I cannot reproduce this, and neither can he anymore. Yay!

Marking back as RFT as the PR is merged.

#11 Updated by Jurgen Segaert over 1 year ago

Thank you so much for jumping on my issue!
I tested the patch on the nightlies in a VM, so performance may indeed be a factor for this to occur.

#12 Updated by Dru Lavigne over 1 year ago

  • Status changed from Ready for Testing to Done
  • Needs QA changed from Yes to No

Also available in: Atom PDF