Project

General

Profile

Bug #25136

Long RSYNC jobs cause unstable system and "indefinite wait buffer" messages in console

Added by Gaelan Lloyd about 3 years ago. Updated about 3 years ago.

Status:
Closed: User Config Issue
Priority:
No priority
Assignee:
Alexander Motin
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

- Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
- SuperMicro X9SRL-F
- 32 GB ECC RAM

ChangeLog Required:
No

Description

Whenever I try to do a large RSYNC job on my FreeNAS machine the web GUI will become intermittently unresponsive and the system console fills up with messages like this:

Jul 13 22:25:03 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101740, size: 32768
Jul 13 22:25:03 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2097213, size: 28672
Jul 13 22:25:23 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101740, size: 32768
Jul 13 22:25:23 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2097213, size: 28672
Jul 13 22:25:43 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101740, size: 32768
Jul 13 22:25:43 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2097213, size: 28672
Jul 13 22:26:03 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101662, size: 4096
Jul 13 22:26:04 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2097226, size: 4096
Jul 13 22:26:32 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2098054, size: 8192
Jul 13 22:26:52 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2098054, size: 8192
Jul 13 22:27:12 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2098054, size: 8192
Jul 13 22:27:32 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2098054, size: 8192
Jul 13 22:28:04 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101675, size: 4096
Jul 13 22:28:24 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101675, size: 4096
Jul 13 22:28:44 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101675, size: 4096
Jul 13 22:29:04 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101675, size: 4096
Jul 13 22:29:36 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101748, size: 4096
Jul 13 22:29:56 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101748, size: 4096
Jul 13 22:30:16 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101748, size: 4096
Jul 13 22:30:36 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101748, size: 4096
Jul 13 22:31:22 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2109797, size: 12288
Jul 13 22:31:42 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2109797, size: 12288
Jul 13 22:32:02 gkfs swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2109797, size: 12288

In some cases the server will become completely unresponsive, even the actual keyboard and monitor console on the server itself.

I've attached images of the reporting charts showing that the system isn't being taxed too hard IMO. I have no idea what those messages mean, the closest I've seen is something having to do with ZFS swap space (https://lists.freebsd.org/pipermail/freebsd-fs/2012-April/014029.html).

The operation that I'm performing is an RSYNC job from the internal ZFS hard drive pool to an external USB drive that I've formatted with FreeNAS as ZFS. I'm doing a backup of files from the internal HDD pool to the external USB drive. I've tried the same operation with the external USB drive formatted as UFS and had the same thing happen.

memory.png (17 KB) memory.png Gaelan Lloyd, 07/13/2017 10:45 PM
swap.png (10.5 KB) swap.png Gaelan Lloyd, 07/13/2017 10:45 PM
cpu.png (15.1 KB) cpu.png Gaelan Lloyd, 07/13/2017 10:45 PM
system-load.png (12.5 KB) system-load.png Gaelan Lloyd, 07/13/2017 10:45 PM
11769
11770
11771
11772

History

#1 Updated by Dru Lavigne about 3 years ago

Gaelan: please attach a debug (System -> Advanced -> Save Debug). We'll mark the ticket as private until a dev has a chance to review the debug.

#2 Updated by Gaelan Lloyd about 3 years ago

Hi Dru, thanks for your fast reply. This system contains sensitive material and I'm not comfortable sharing a full debug at the moment. Are there any specific files that I can attach from the debug so that I can redact out the parts I need to keep private? Sorry to be a hassle.

#3 Updated by Dru Lavigne about 3 years ago

  • Assignee changed from Release Council to Alexander Motin
  • Private changed from No to Yes

Sasha, can you let Gaelan which debug files would be useful for your diagnosis? I'll mark the ticket as private til you get a chance to review them.

#4 Updated by Alexander Motin about 3 years ago

  • Status changed from Unscreened to Closed: User Config Issue

Those messages are reported when swap daemon can't read requested pages from disk within quite long 20 seconds. It may be a combination of slow disk, some USB problems causing delays, USB interface not supporting command queuing, and the fact of swap used in general. I am not sure we can do much here. You on the other side can do some things:
- Reformat external disk without swap partition (you can temporary set size to zero in FreeNAS Advanced settings);
- Explicitly limit ZFS ARC size via vfs.zfs.arc_max loader tunable to some lower value to avoid swapping;
- Update to modern FreeNAS 11.0;
- Look what is going on with your disks load and delays with `gstat -I 1s -p`.

#5 Updated by Dru Lavigne about 3 years ago

  • Target version set to N/A
  • Private changed from Yes to No

#6 Updated by Gaelan Lloyd about 3 years ago

Hi Alexander, I have upgraded to 11.0-U2. Still having issues and can't find ways to simply format the HDD without swap.

To sum up what I'm trying to do, I just want to be able to plug in an external HDD to the server and have it do an rsync job to back up the data in my pool to the external HDD.

I can't find any instructions that are current on how to do this, and the only options that I get when formatting the drive with the volume manager are to create a ZFS partition (with swap, I'm assuming).

What's the best way I can go about doing backups to a single, USB external HDD? Any instructions you can send would be really helpful.

Also available in: Atom PDF