or more exactly How can I improve network throughput between two systems mentioned below, optimizing only stuff from OSI layer 4 and below it?
I have two Nginx servers running on Ubuntu instances. Both systems use the same Ubuntu version, identical Nginx configurations, and the same network-related settings (including TCP stack parameters).
I am using the wrk benchmarking tool to generate HTTP traffic.
When I run wrk locally on the same instance as the Nginx server, I achieve approximately 520,000 requests per second. Both servers produce similar results in this configuration.
However, when I run the same load test from one Ubuntu instance against the Nginx server on the other instance, throughput drops to approximately 175,000 requests per second. The results are consistent regardless of which machine acts as the client or server.
The two machines are connected directly using a `Cat5e` cable. The distance between them is less than one meter. Both systems use 1 Gbps full-duplex Ethernet adapters. The TX and RX ring buffer sizes are set to 256, which is the maximum supported value on both NICs.
I do not observe any packet drops or errors when checking interface statistics with:
ethtool -S eno1
Based on the packet sizes, I expected throughput closer to 210,000 requests per second over the current network.
I estimated it like this.
Each HTTP/1.1 request fits into a single TCP packet of approximately 185 bytes, and each response fits into a single packet of approximately 235 bytes.
Adding
Ethernet header: 14 bytes
FCS: 4 bytes
Preamble + SFD: 8 bytes
Inter-frame gap: 12 bytes
So that on the wire:
Request ≈ 185 + 38 = 223 bytes
Response ≈ 235 + 38 = 273 bytes
iperf3 measures bitrate around 936Mbps
936Mb/(223*8) ≈ 524,663 requests
936Mb/(273*8) ≈ 428,571 requests
I took the minimum and divided by 2. ≈214,000
I am almost sure that the server process time is not greater than a few microseconds. I did TCP dump and figured it out.
