Prevent hold on file descriptors that stalls plugin installation
Sometimes iocage stops with <stdin> and nothing happened. Example https://builds.ixsystems.com/jenkins/job/iocage%20-%20plugin-single-test/1/console I did abound it after 1 hour.
#1 Updated by Brandon Schneider about 2 years ago
- Subject changed from iocage - stalling builds to iocage plugins stalling jenkins when they hold onto stdin or stdout.
- Category changed from Console to Middleware
- Status changed from Unscreened to In Progress
- Target version changed from Backlog to 11.2-BETA3
- Seen in changed from Unspecified to Master - FreeNAS Nightlies
- Severity changed from New to Medium
#6 Updated by Jurgen Segaert about 2 years ago
This change seems to have some weird side effects, i.e. not all of the output that is issued in post_install.sh is displayed. Example:
The last lines of post_install.sh of the gitlab plugin are as follows:
echo "Starting nginx..." service nginx start echo "Starting gitlab..." service gitlab start echo "Starting gltlab pages..." service gitlab_pages start echo "Starting sshd..." service sshd start echo "Database Name: $DB" echo "Database User: $USER" echo "Database Password: $PASS" echo "Please open the URL to set your password, Login Name is root."
... but when installing the plugin from the command line:
iocage fetch --plugins -n gitlab ip4_addr="em0|192.168.0.210/24" --branch master
...some of the information that is echo'ed, e.g. in this example the user and password, doesn't show:
(...) Starting nginx... Performing sanity check on nginx configuration: Starting nginx. Starting gitlab... Regenerate Gitlab Gemfile.lock Regenerate Gitaly Gemfile.lock Starting GitLab Unicorn Starting GitLab Sidekiq Starting GitLab Workhorse Starting Gitaly ... The GitLab Unicorn web server with pid 15867 is running. The GitLab Sidekiq job dispatcher with pid 15938 is running. The GitLab Workhorse with pid 15902 is running. Gitaly with pid 15900 is running. GitLab and all its components are up and running. Starting gltlab pages... Starting sshd... Generating RSA host key. 2048 SHA256:H5cpfCpV0MScQKWtP4MneTwe5J47ThaG/vRs7SKbjGE root@gitlab (RSA) Generating ECDSA host key. 256 SHA256:W+I+QKD3TWNBRZXZgsTHJbEekDr/CZF9qPUaXHk0Gto root@gitlab (ECDSA) Generating ED25519 host key. 256 SHA256:afm2KzX57V04wfFV/inVD1EFrQmdM84+mMrCIlR/p00 root@gitlab (ED25519) Performing sanity check on sshd configuration. Starting sshd. Database Name: gitlabhq_production Admin Portal: http://192.168.0.210
This seems to be non-deterministic; different runs will output a different set of lines, but there's always something missing.
#8 Updated by Brandon Schneider about 2 years ago
Jurgen: Are you able to try the commit https://github.com/iocage/iocage/commit/99ba2a55e6da56a15db2af5a2959dc941b0559b3 and see if it resolves your issue before I merge it into the FreeNAS tree? Unless you're on an SDK image, you'll need to comment out line 10 of the Makefile and lines 1-2 of the requirements-dev.txt before the makefile will install for you (assuming you test this on FreeNAS, I'm not sure which machine you test your PR's on)
If it's easier, we can merge it into the FreeNAS tree and you can wait for a nightly to roll with it, however, I cannot reproduce the missing output, before or after these commits. So I'm suspecting possibly CPU speed is playing into this, what CPU are you using?
root@freenas:~ # iocage clean -jf ; iocage fetch -P -n gitlab dhcp=on bpf=yes vnet=on --branch master Cleaning iocage/jails All iocage jail datasets have been destroyed. Plugin: GitLab Official Plugin: True Using RELEASE: 11.2-RELEASE Using Branch: master Post-install Artifact: https://github.com/freenas/iocage-plugin-gitlab.git These pkgs will be installed: - www/gitlab - databases/postgresql95-server - databases/postgresql95-contrib - www/nginx Testing SRV response to iocage-plugins Testing DNSSEC response to iocage-plugins Installing plugin packages: - www/gitlab... - databases/postgresql95-server... - databases/postgresql95-contrib... - www/nginx... Fetching artifact... Running post_install.sh Command output: gitlab_enable: -> YES gitlab_pages_enable: -> YES postgresql_enable: -> YES redis_enable: -> YES nginx_enable: -> YES sshd_enable: NO -> YES The files belonging to this database system will be owned by user "pgsql". This user must also own the server process. The database cluster will be initialized with locale "C". The default text search configuration will be set to "english". Data page checksums are disabled. creating directory /usr/local/pgsql/data ... ok creating subdirectories ... ok selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting dynamic shared memory implementation ... posix creating configuration files ... ok creating template1 database in /usr/local/pgsql/data/base/1 ... ok initializing pg_authid ... ok initializing dependencies ... ok creating system views ... ok loading system objects' descriptions ... ok creating collations ... ok creating conversions ... ok creating dictionaries ... ok setting privileges on built-in objects ... ok creating information schema ... ok loading PL/pgSQL server-side language ... ok vacuuming database template1 ... ok copying template1 to template0 ... ok copying template1 to postgres ... ok syncing data to disk ... ok Success. You can now start the database server using: /usr/local/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start LOG: ending log output to stderr HINT: Future log output will go to log destination "syslog". CREATE ROLE CREATE DATABASE ALTER ROLE CREATE EXTENSION LOG: ending log output to stderr HINT: Future log output will go to log destination "syslog". redis does not exist in /etc/rc.d or the local startup directories (/usr/local/etc/rc.d), or is not executable Starting nginx... Performing sanity check on nginx configuration: Starting nginx. Starting gitlab... gitlab does not exist in /etc/rc.d or the local startup directories (/usr/local/etc/rc.d), or is not executable Starting gltlab pages... gitlab_pages does not exist in /etc/rc.d or the local startup directories (/usr/local/etc/rc.d), or is not executable Starting sshd... Generating RSA host key. 2048 SHA256:wbpV6UJUKTmJmFPhgPXY2LdlQN0nW5WY7/gQuvKV6ac root@gitlab (RSA) Generating ECDSA host key. 256 SHA256:qRST/k9l0Q/fTDOX5DRPlKa4XidZBZJFVM3Hz5zPXmA root@gitlab (ECDSA) Generating ED25519 host key. 256 SHA256:3jLttIILRaun3v/u1ADCsvtEbN8+d36+TSeLemzD1NI root@gitlab (ED25519) Performing sanity check on sshd configuration. Starting sshd. Database Name: gitlabhq_production Database User: git Database Password: Rb5ryG2hfEAbSq7S Please open the URL to set your password, Login Name is root. Admin Portal: http://192.168.122.63
#10 Updated by Brandon Schneider about 2 years ago
- Status changed from In Progress to Ready for Testing
Jurgen made a change that fixes the behavior he saw over in iocage/iocage. I brought that into the PR, which means I cannot reproduce this, and neither can he anymore. Yay!
Marking back as RFT as the PR is merged.