Project

General

Profile

Bug #8995

Adopt FreeBSD RPC improvements

Added by Xin Li over 5 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Nice to have
Assignee:
Xin Li
Category:
OS
Target version:
Seen in:
Severity:
New
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

Changes ca3bdd1..31d5a6b (trueos|fce777e and trueos|31d5a6b) in trueos masters/9.3 trunk should be considered for a SU.

Changeset fce777e fixes a problem when signal is delivered to a nfsd thread, they would exit.

Changeset 31d5a6b improves responsiveness and reliability for NFS.

Associated revisions

Revision b4d8df81 (diff)
Added by kib over 5 years ago

Current reaction of the nfsd worker threads to any signal is exit. This is not correct at least for the stop requests. Check for stop conditions and suspend threads if requested. Reported and tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week (cherry picked from commit e83a8b4fa1e916a1dfd6ffe59024ef83cd10f294) (cherry picked from commit fce777e3498485465faf5a792f786ea74f52a3a0) Ticket: #8995

Revision b4d8df81 (diff)
Added by kib over 5 years ago

Current reaction of the nfsd worker threads to any signal is exit. This is not correct at least for the stop requests. Check for stop conditions and suspend threads if requested. Reported and tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week (cherry picked from commit e83a8b4fa1e916a1dfd6ffe59024ef83cd10f294) (cherry picked from commit fce777e3498485465faf5a792f786ea74f52a3a0) Ticket: #8995

Revision 6efc936e (diff)
Added by wollman over 5 years ago

Fix overflow bugs in and remove obsolete limit from kernel RPC implementation. The kernel RPC code, which is responsible for the low-level scheduling of incoming NFS requests, contains a throttling mechanism that prevents too much kernel memory from being tied up by NFS requests that are being serviced. When the throttle is engaged, the RPC layer stops servicing incoming NFS sockets, resulting ultimately in backpressure on the clients (if they're using TCP). However, this is a very heavy-handed mechanism as it prevents all clients from making any requests, regardless of how heavy or light they are. (Thus, when engaged, the throttle often prevents clients from even mounting the filesystem.) The throttle mechanism applies specifically to requests that have been received by the RPC layer (from a TCP or UDP socket) and are queued waiting to be serviced by one of the nfsd threads; it does not limit the amount of backlog in the socket buffers. The original implementation limited the total bytes of queued requests to the minimum of a quarter of (nmbclusters * MCLBYTES) and 45 MiB. The former limit seems reasonable, since requests queued in the socket buffers and replies being constructed to the requests in progress will all require some amount of network memory, but the 45 MiB limit is plainly ridiculous for modern memory sizes: when running 256 service threads on a busy server, 45 MiB would result in just a single maximum-sized NFS3PROC_WRITE queued per thread before throttling. Removing this limit exposed integer-overflow bugs in the original computation, and related bugs in the routines that actually account for the amount of traffic enqueued for service threads. The old implementation also attempted to reduce accounting overhead by batching updates until each queue is fully drained, but this is prone to livelock, resulting in repeated accumulate-throttle-drain cycles on a busy server. Various data types are changed to long or unsigned long; explicit 64-bit types are not used due to the unavailability of 64-bit atomics on many 32-bit platforms, but those platforms also cannot support nmbclusters large enough to cause overflow. This code (in a 10.1 kernel) is presently running on production NFS servers at CSAIL. Summary of this revision: * Removes 45 MiB limit on requests queued for nfsd service threads * Fixes integer-overflow and signedness bugs * Avoids unnecessary throttling by not deferring accounting for completed requests Differential Revision: https://reviews.freebsd.org/D2165 Reviewed by: rmacklem, mav MFC after: 30 days Relnotes: yes Sponsored by: MIT Computer Science & Artificial Intelligence Laboratory (cherry picked from commit 299467e040e301d18e614fc1f65329181f32e687) (cherry picked from commit 31d5a6b8c958dd10bc5f4ac66a10a760565f2e12) Ticket: #8995

Revision 6efc936e (diff)
Added by wollman over 5 years ago

Fix overflow bugs in and remove obsolete limit from kernel RPC implementation. The kernel RPC code, which is responsible for the low-level scheduling of incoming NFS requests, contains a throttling mechanism that prevents too much kernel memory from being tied up by NFS requests that are being serviced. When the throttle is engaged, the RPC layer stops servicing incoming NFS sockets, resulting ultimately in backpressure on the clients (if they're using TCP). However, this is a very heavy-handed mechanism as it prevents all clients from making any requests, regardless of how heavy or light they are. (Thus, when engaged, the throttle often prevents clients from even mounting the filesystem.) The throttle mechanism applies specifically to requests that have been received by the RPC layer (from a TCP or UDP socket) and are queued waiting to be serviced by one of the nfsd threads; it does not limit the amount of backlog in the socket buffers. The original implementation limited the total bytes of queued requests to the minimum of a quarter of (nmbclusters * MCLBYTES) and 45 MiB. The former limit seems reasonable, since requests queued in the socket buffers and replies being constructed to the requests in progress will all require some amount of network memory, but the 45 MiB limit is plainly ridiculous for modern memory sizes: when running 256 service threads on a busy server, 45 MiB would result in just a single maximum-sized NFS3PROC_WRITE queued per thread before throttling. Removing this limit exposed integer-overflow bugs in the original computation, and related bugs in the routines that actually account for the amount of traffic enqueued for service threads. The old implementation also attempted to reduce accounting overhead by batching updates until each queue is fully drained, but this is prone to livelock, resulting in repeated accumulate-throttle-drain cycles on a busy server. Various data types are changed to long or unsigned long; explicit 64-bit types are not used due to the unavailability of 64-bit atomics on many 32-bit platforms, but those platforms also cannot support nmbclusters large enough to cause overflow. This code (in a 10.1 kernel) is presently running on production NFS servers at CSAIL. Summary of this revision: * Removes 45 MiB limit on requests queued for nfsd service threads * Fixes integer-overflow and signedness bugs * Avoids unnecessary throttling by not deferring accounting for completed requests Differential Revision: https://reviews.freebsd.org/D2165 Reviewed by: rmacklem, mav MFC after: 30 days Relnotes: yes Sponsored by: MIT Computer Science & Artificial Intelligence Laboratory (cherry picked from commit 299467e040e301d18e614fc1f65329181f32e687) (cherry picked from commit 31d5a6b8c958dd10bc5f4ac66a10a760565f2e12) Ticket: #8995

Revision 24e8c7c7 (diff)
Added by kib over 5 years ago

Current reaction of the nfsd worker threads to any signal is exit. This is not correct at least for the stop requests. Check for stop conditions and suspend threads if requested. Reported and tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week (cherry picked from commit e83a8b4fa1e916a1dfd6ffe59024ef83cd10f294) (cherry picked from commit fce777e3498485465faf5a792f786ea74f52a3a0) Ticket: #8995 (cherry picked from commit b4d8df81f6a1bf743da4d6f5f495a8e3747915bd)

Revision 24e8c7c7 (diff)
Added by kib over 5 years ago

Current reaction of the nfsd worker threads to any signal is exit. This is not correct at least for the stop requests. Check for stop conditions and suspend threads if requested. Reported and tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week (cherry picked from commit e83a8b4fa1e916a1dfd6ffe59024ef83cd10f294) (cherry picked from commit fce777e3498485465faf5a792f786ea74f52a3a0) Ticket: #8995 (cherry picked from commit b4d8df81f6a1bf743da4d6f5f495a8e3747915bd)

Revision a7c9e621 (diff)
Added by wollman over 5 years ago

Fix overflow bugs in and remove obsolete limit from kernel RPC implementation. The kernel RPC code, which is responsible for the low-level scheduling of incoming NFS requests, contains a throttling mechanism that prevents too much kernel memory from being tied up by NFS requests that are being serviced. When the throttle is engaged, the RPC layer stops servicing incoming NFS sockets, resulting ultimately in backpressure on the clients (if they're using TCP). However, this is a very heavy-handed mechanism as it prevents all clients from making any requests, regardless of how heavy or light they are. (Thus, when engaged, the throttle often prevents clients from even mounting the filesystem.) The throttle mechanism applies specifically to requests that have been received by the RPC layer (from a TCP or UDP socket) and are queued waiting to be serviced by one of the nfsd threads; it does not limit the amount of backlog in the socket buffers. The original implementation limited the total bytes of queued requests to the minimum of a quarter of (nmbclusters * MCLBYTES) and 45 MiB. The former limit seems reasonable, since requests queued in the socket buffers and replies being constructed to the requests in progress will all require some amount of network memory, but the 45 MiB limit is plainly ridiculous for modern memory sizes: when running 256 service threads on a busy server, 45 MiB would result in just a single maximum-sized NFS3PROC_WRITE queued per thread before throttling. Removing this limit exposed integer-overflow bugs in the original computation, and related bugs in the routines that actually account for the amount of traffic enqueued for service threads. The old implementation also attempted to reduce accounting overhead by batching updates until each queue is fully drained, but this is prone to livelock, resulting in repeated accumulate-throttle-drain cycles on a busy server. Various data types are changed to long or unsigned long; explicit 64-bit types are not used due to the unavailability of 64-bit atomics on many 32-bit platforms, but those platforms also cannot support nmbclusters large enough to cause overflow. This code (in a 10.1 kernel) is presently running on production NFS servers at CSAIL. Summary of this revision: * Removes 45 MiB limit on requests queued for nfsd service threads * Fixes integer-overflow and signedness bugs * Avoids unnecessary throttling by not deferring accounting for completed requests Differential Revision: https://reviews.freebsd.org/D2165 Reviewed by: rmacklem, mav MFC after: 30 days Relnotes: yes Sponsored by: MIT Computer Science & Artificial Intelligence Laboratory (cherry picked from commit 299467e040e301d18e614fc1f65329181f32e687) (cherry picked from commit 31d5a6b8c958dd10bc5f4ac66a10a760565f2e12) Ticket: #8995 (cherry picked from commit 6efc936e7b6631e991ca2277673ba14ab163072f)

Revision a7c9e621 (diff)
Added by wollman over 5 years ago

Fix overflow bugs in and remove obsolete limit from kernel RPC implementation. The kernel RPC code, which is responsible for the low-level scheduling of incoming NFS requests, contains a throttling mechanism that prevents too much kernel memory from being tied up by NFS requests that are being serviced. When the throttle is engaged, the RPC layer stops servicing incoming NFS sockets, resulting ultimately in backpressure on the clients (if they're using TCP). However, this is a very heavy-handed mechanism as it prevents all clients from making any requests, regardless of how heavy or light they are. (Thus, when engaged, the throttle often prevents clients from even mounting the filesystem.) The throttle mechanism applies specifically to requests that have been received by the RPC layer (from a TCP or UDP socket) and are queued waiting to be serviced by one of the nfsd threads; it does not limit the amount of backlog in the socket buffers. The original implementation limited the total bytes of queued requests to the minimum of a quarter of (nmbclusters * MCLBYTES) and 45 MiB. The former limit seems reasonable, since requests queued in the socket buffers and replies being constructed to the requests in progress will all require some amount of network memory, but the 45 MiB limit is plainly ridiculous for modern memory sizes: when running 256 service threads on a busy server, 45 MiB would result in just a single maximum-sized NFS3PROC_WRITE queued per thread before throttling. Removing this limit exposed integer-overflow bugs in the original computation, and related bugs in the routines that actually account for the amount of traffic enqueued for service threads. The old implementation also attempted to reduce accounting overhead by batching updates until each queue is fully drained, but this is prone to livelock, resulting in repeated accumulate-throttle-drain cycles on a busy server. Various data types are changed to long or unsigned long; explicit 64-bit types are not used due to the unavailability of 64-bit atomics on many 32-bit platforms, but those platforms also cannot support nmbclusters large enough to cause overflow. This code (in a 10.1 kernel) is presently running on production NFS servers at CSAIL. Summary of this revision: * Removes 45 MiB limit on requests queued for nfsd service threads * Fixes integer-overflow and signedness bugs * Avoids unnecessary throttling by not deferring accounting for completed requests Differential Revision: https://reviews.freebsd.org/D2165 Reviewed by: rmacklem, mav MFC after: 30 days Relnotes: yes Sponsored by: MIT Computer Science & Artificial Intelligence Laboratory (cherry picked from commit 299467e040e301d18e614fc1f65329181f32e687) (cherry picked from commit 31d5a6b8c958dd10bc5f4ac66a10a760565f2e12) Ticket: #8995 (cherry picked from commit 6efc936e7b6631e991ca2277673ba14ab163072f)

History

#1 Updated by Jordan Hubbard over 5 years ago

  • Assignee set to Xin Li

#2 Updated by Xin Li over 5 years ago

  • Status changed from Unscreened to Ready For Release

#3 Updated by Josh Paetzel over 5 years ago

  • ChangeLog Entry updated (diff)

#4 Updated by Jordan Hubbard over 5 years ago

  • Status changed from Ready For Release to Resolved

#5 Avatar?id=14398&size=24x24 Updated by Kris Moore about 4 years ago

  • Target version changed from Unspecified to N/A

Also available in: Atom PDF