Zend certified PHP/Magento developer

slow vpn tunnels speed on high speed network

I have some VMs with minimum network speed of 1G
I need to connect each VM via a split tunnel.
While the raw speed between VMs are 1G but over the tunnels , the speed is down to 10% of the max speed (100M).
My question is that what and where are the bottlenecks?
I am expecting to see near 800M ~ 900M

Tests have been done using iperf3 and three VPN softwares

  1. SoftEther VPN (Dev version v5+)
  2. ocserv / openconnect
  3. wireguard

server

iperf3 --server --bind IP

client

iperf3 --client IP --time 1 --parallel 1

Raw ipv4 to ipv4 (from server A to server B)

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-1.00   sec  1006 MBytes  8.44 Gbits/sec  1473             sender
[  5]   0.00-1.04   sec  1002 MBytes  8.07 Gbits/sec                  receiver

SoftEther tap to tap (cascade connection from server A to sever B)

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-1.00   sec   108 MBytes   899 Mbits/sec  1205             sender
[  5]   0.00-1.05   sec   104 MBytes   833 Mbits/sec                  receiver

OpenConnect tun to tun (from server A to server B)

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-1.00   sec  97.0 MBytes   814 Mbits/sec  5311             sender
[  5]   0.00-1.04   sec  94.3 MBytes   759 Mbits/sec                  receiver

WireGuard

is twice faster near 200M

Both server Debian 11
and Kernel

Linux 206 5.10.0-23-amd64 #1 SMP Debian 5.10.179-2 (2023-07-14) x86_64 GNU/Linux