Project

General

Profile

Bug #22859

Speed up the installation of base-os

Added by William Grzybowski over 1 year ago. Updated 12 months ago.

Status:
Resolved
Priority:
Important
Assignee:
Sean Fagan
Category:
OS
Target version:
Sprint:
Severity:
New
Backlog Priority:
Reason for Closing:
Reason for Blocked:
Needs QA:
No
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Install and update seems to take a long time, especially now with the new WebUI where it has a lot of small files.

The python process seems to be CPU bound, its at 100% CPU usage. We need to improve that one way or another.
Using multiprocessing module might be the way to go.

Associated revisions

Revision acdd7f88 (diff)
Added by Sean Fagan over 1 year ago

sqlite3 still uses transactions in executemany(), it appears; setting
isolation_level to DEFERRED for this speeds up the installation of base-os
significantly.

Ticket: #22859

History

#1 Updated by William Grzybowski over 1 year ago

  • Description updated (diff)

#2 Updated by Sean Fagan over 1 year ago

  • Status changed from Unscreened to Screened
  • Priority changed from Blocks Until Resolved to Important

It is dependent on processor speed, installation source, and installation target.

This is not blocking until resolved, since it's been shipping this way for over two years.

#3 Updated by Sean Fagan over 1 year ago

There are two relatively easy things to check.

  1. Make sure the version of freenas-pkgtools is up to date, since one of the things it does is put the target BE into async mode while doing the update/install.
  2. If that's not enough of an improvement, perhaps use pigz -- it made a significant difference in the package creation process. But then again, the build servers have lots of CPU cores.

#4 Updated by William Grzybowski over 1 year ago

We have been shipping for two years but the size of the image and the number of files has increased in 9.10.3. Thus we are noticing a bigger difference now.

1. We are using the latest of freenas-pkgtools in master builds.
2. Adding pigz might be something but it could prove to be harder to use given how the tar file is read in files extract.

I mentioned multiprocessing because we could extract multiple files at once (within same dir) since most of the overhead seems to be from python objects as well calculating sha256 and reading the file (compressed).
There are downsides as well but well, just wanted to give some idea. We have quite a few people complaining how long it takes to install/upgrade.

#5 Avatar?id=14398&size=24x24 Updated by Kris Moore over 1 year ago

This is also impacting QA. The longest bulk of time spent is just updating / extracting files and any extra speed we can eek out of this will be much appreciated.

#6 Updated by William Grzybowski over 1 year ago

There is also the option of writing a C module for a core of the install/update. e.g. looping through the files and ExtractEntry.

#7 Updated by Sean Fagan over 1 year ago

While I can build fn9_head, it doesn't boot, at all, under Fusion. Grub simply doesn't boot.

#8 Updated by William Grzybowski over 1 year ago

Sean Fagan wrote:

While I can build fn9_head, it doesn't boot, at all, under Fusion. Grub simply doesn't boot.

How is fn9_head related?

#9 Updated by Sean Fagan over 1 year ago

Seen in: Master - FreeNAS Nightlies

Is that not fn9_head?

#10 Updated by William Grzybowski over 1 year ago

Sean Fagan wrote:

Seen in: Master - FreeNAS Nightlies

Is that not fn9_head?

No, freenas9 profile.

#11 Updated by Sean Fagan over 1 year ago

Well at least that means I don't have to update my OS 8-).

#12 Updated by Sean Fagan over 1 year ago

So, without any changes (other than to keep track of time), the install on a VM on my Mac takes 233 seconds. That's not particularly slow -- as I would generally expect, it's faster than Corral is.

Now, of course, that's on a VM with a super-fast SSD, so installing to an actual thumb drive is going to be considerably slower.

Will time an update next.

#13 Updated by Sean Fagan over 1 year ago

And 3m12s to update to 9.10-MASTER-201703301241.

#14 Avatar?id=14398&size=24x24 Updated by Kris Moore over 1 year ago

  • Target version changed from 9.10.3 to 9.10.4

Moving this to 9.10.4 - William did you have some specific code in mind that could be "optimized" for performance?

#15 Updated by William Grzybowski over 1 year ago

Kris Moore wrote:

Moving this to 9.10.4 - William did you have some specific code in mind that could be "optimized" for performance?

They are already described in #4 and #6

#16 Avatar?id=14398&size=24x24 Updated by Kris Moore over 1 year ago

  • Target version changed from 9.10.4 to 11.1

#17 Updated by Sean Fagan over 1 year ago

  • Status changed from Screened to 19

commit:acdd7f88492938a2ae888305d4d434a742d9c648 improves installation speed of base-os significantly (well, the post-installation gap is significantly smaller now). There's not a lot to be done about the fact that there are almost 80,000 filesystem objects in base-os right now, most of which are significantly smaller than the blocksize used by both the filesystem and usb devices. Doing a checksum of the files right after being written does not appear to be a limiting factor, since from running iostat it's clear that I/O is simply not being done for large periods -- that's what caching and asynchronous writes get you.

(Now, switching to something other than gzip may help, but has compatibility issues; another possibility is to decompress the package file after verification but before extraction. I've not benchmarked either of those, as getting rid of the very significant delay between the package being installed and it going onto the next package seemed like a good start.)

#18 Updated by Dru Lavigne about 1 year ago

  • Status changed from 19 to 46

Sean: is this code ready to be merged? If not, will it be in time for 11.1?

#19 Updated by Sean Fagan about 1 year ago

  • Assignee changed from Sean Fagan to Vaibhav Chauhan

I honestly thought it was already merged. VB, again, what's the next step to get it in?

#20 Updated by Vaibhav Chauhan about 1 year ago

  • Status changed from 46 to Ready For Release

this change had a typo which was fixed in `a02b27d94994040d85ad6b6821edafa1cb8f4906`

#21 Updated by William Grzybowski about 1 year ago

  • Assignee changed from Vaibhav Chauhan to Sean Fagan

#22 Updated by Dru Lavigne about 1 year ago

  • Subject changed from Slow install/update to Speed up the installation of base-os

#23 Updated by Dru Lavigne about 1 year ago

  • Target version changed from 11.1 to 11.1-BETA1

#24 Updated by Dru Lavigne 12 months ago

  • Status changed from Ready For Release to Resolved

#25 Updated by Rishabh Chauhan 12 months ago

I just upgraded the VM from FreeNas 11 u4 to FreeNas PreRelease(Beta1)...
System specs: 4096MB RAM, Hard-disks: 20GB, 40GB, 40GB
Update and reboot time: 11min 51sec

After talking to Sean, I found that previously it was taking +30min for the same process.., Hence Test Passed

#26 Updated by Rishabh Chauhan 12 months ago

  • Needs QA changed from Yes to No
  • QA Status Test Passes FreeNAS added
  • QA Status deleted (Not Tested)

Also available in: Atom PDF