Project

General

Profile

Bug #25383

Slow scp transfer speeds using ssh cipher chacha20-poly1305@openssh.com

Added by Lucas Burson about 1 year ago. Updated 10 months ago.

Status:
Closed: Insufficient Info
Priority:
No priority
Assignee:
Alexander Motin
Category:
OS
Target version:
Seen in:
Sprint:
Severity:
New
Backlog Priority:
Reason for Closing:
Reason for Blocked:
Needs QA:
Yes
Needs Doc:
Yes
Needs Merging:
Yes
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:

Board: Supermicro X10SDV-4C+-TLN4F-O Mini-ITX Intel Xeon D-1518 DDR4 Motherboard and CPU
Memory: 2x Crucial - 16GB (1 x 16GB) DDR4-2133 Memory
Network controller: Intel i350-AM2 Gigabit Ethernet

ChangeLog Required:
No

Description

On a 1Gbps network I see 30MB/s scp transfers when using cipher , which is extremely slow. Using a different cipher results in the expected ~112MB/s.
I'm not sure if this is a FreeNAS issue or BSD's openssh.
I see this as a significant issue since, for many users, ssh transfers will default to and users will migrate to "faster" NAS software.

See this forum post for my original investigation:
https://forums.freenas.org/index.php?threads/very-slow-scp-transfer-speeds-to-baremetal-freenas-but-fast-to-vm-in-freenas-bhyve.56390/

Workaround:
  • Remove from the openssh cipher list.
  • Removing that cipher increases transfer rate to 50MB/s. Use these systctls to get the FULL 1Gbps rates:
    sysctl kern.ipc.maxsockbuf=33554432
    sysctl net.inet.tcp.recvbuf_auto=1
    sysctl net.inet.tcp.sendbuf_auto=1
    sysctl net.inet.tcp.sendbuf_max=33554432
    sysctl net.inet.tcp.recvbuf_max=33554432
    sysctl net.inet.tcp.recvspace=4194304
    sysctl net.inet.tcp.sendspace=4194304
    
    

History

#1 Updated by Dru Lavigne about 1 year ago

  • Assignee changed from Release Council to Alexander Motin

#2 Updated by Alexander Motin about 1 year ago

  • Status changed from Unscreened to Screened
  • Target version set to 11.1

Have you tried to look on scp CPU usage in `top` while using different ciphers? I suspect that on your CPU chacha20 is slower then AES because it is not accelerated by hardware (AES accelerated by AES-NI CPU instructions), while CPU frequency is not very high to do it in software.

#3 Updated by Alexander Motin about 1 year ago

  • Status changed from Screened to 15

#4 Updated by Lucas Burson about 1 year ago

Alexander Motin wrote:

Have you tried to look on scp CPU usage in `top` while using different ciphers? I suspect that on your CPU chacha20 is slower then AES because it is not accelerated by hardware (AES accelerated by AES-NI CPU instructions), while CPU frequency is not very high to do it in software.

CPU usage of chacha20 is 15/0 (usr/sys), and AES is 9/15. The kernel is doing more of the work with AES, is this an indication of hardware acceleration?

#5 Updated by Alexander Motin about 1 year ago

scp is single-threaded application. I asked about percent of CPU time used by it specifically to identify the bottleneck. Higher total CPU usage can be just a result of higher throughput.

#6 Updated by Lucas Burson about 1 year ago

Okay. Let me know if you'd like other metrics, I'm happy to help. I have a Fedora26 VM running on the FreeNAS node (in bhyve) which performs at 112MB/s but I didn't think to check the cipher. I'll verify the cipher used report back. That should help indicate if the cipher is slow from hardware or compiled bits.

#7 Updated by Alexander Motin about 1 year ago

As I have told, I do like to see CPU time usage by the scp/ssh thread. Check for used cipher would also be useful. If it ends up in confirmation that chacha20 is just slower by itself, then I tend to close this with "Behaves correctly".

#8 Updated by Lucas Burson about 1 year ago

Copying the file to FreeNAS using causes sshd to use 100% CPU, limiting the transfer to 30MB/s.
Copying the file to Fedora 26 VM (running on FreeNAS), again using , is at 112MB/s.

#9 Updated by Lucas Burson about 1 year ago

Metrics from FreeBSD 11.1 VM.

Cipher     | top sshd | usr%/sys% | rate
chacha20   |    100   | 15/23     |  84MB/s
sha128-crt |     94   |  6/30     | 100MB/s

#10 Updated by Lucas Burson about 1 year ago

Hi, do you need more feedback on this?

#11 Updated by Dru Lavigne about 1 year ago

  • Status changed from 15 to 46

Sasha: please indicate if more info is needed.

#12 Updated by Dru Lavigne 12 months ago

  • Status changed from 46 to Investigation
  • Target version changed from 11.1 to 11.1-U1

#13 Updated by Dru Lavigne 10 months ago

  • Status changed from Investigation to 15

Lucas: is this still an issue for you on 11.1? If so, please attach a debug from an 11.1 system, created after a slow transfer.

#14 Updated by Dru Lavigne 10 months ago

  • Status changed from 15 to Closed: Insufficient Info
  • Target version changed from 11.1-U1 to N/A

Lucas: I'll close this one out for now. If it is still an issue on 11.1, please attach a new debug to this ticket.

Also available in: Atom PDF