[winswitch] XPRA performance on jittery and droppy connections

[Antoine Martin]

On 18/11/2020 00:02, David Johnston via shifter-users wrote:
> Good Day, sorry for the length but I really need help! Payment offered!
The project is looking for sponsors, please contact me privately.

> I'm working on the design and planning for a large-scale XPRA deployment and there is an issue I'm trying to solve:
> 
> On connections with higher-than-normal jitter and packet loss/delays, the UI performance degradation is very severe. I'm using primarily the HTML5 client, 4.04 and 4.1, but the full fat client has the same issue. Setting quality=5 helps somewhat but looks really bad.
Yes, we have a number of tickets regarding this issue.

> However, if I RDP to something using the same connection, I get full resolution and the degradation is barely noticeable. I am using nested VPNs and packet shaping to reproduce my clients' remote access network conditions. The problem looks exactly like it does on their network so I think it's a fair reproduction.
Excellent!
The main reason why this has not been fixed yet is because it has been
very difficult to reproduce reliably.

> XPRA gets into a state where it's hardly using any bandwidth, but the session pauses greyed-out with the little spinning circle. (Presumably waiting for expected packets that it's not getting)
Correct.
The spinners show up when the client is expecting reply packets from the
server and it isn't receiving them in a timely manner.

> Sometimes I get a session timeout and it tries to re-connect. As soon as I hit it with usage again (scrolling in a window) the problem starts again and the session eventually halts; while using very little bandwidth.
> 
> XPRA debug logs have tons of backlogs and delays. The actual connection latency may only be 40ms, but it ends up waiting for multiple seconds, apparently without retrying even after the normal latency has been far exceeded.
I'm not sure what "retrying" you are referring to.
The UDP transport is unstable and should not be used, with the TCP based
ones xpra does not retry anything and simply relies on TCP for
guaranteed delivery.

> When I look at PCAPs, I can see tons of TCP dup acks and retransmits in both XPRA and RDP (expected given the conditions), but somehow RDP is able to recover better. RDP has a higher packet frequency, but fewer packets close to the MTU. I tried reducing the MTU to 800 but the same exact problem happens. RDP seems to better auto-correct for buffering, delays, and loss.
That's quite possible, and definitely something we can improve.
> I can measure the network's loss with this tool by cranking the packet size and frequency: https://packetlosstest.com/ As long as I stay below a threshold, the latency and loss are low. RDP seems to be able to find this threshold and stay under it.
> 
> What I would like to ask is:
> 
>   *   Is there a way to force the client.jitter=1000 even on non-wireless?
XPRA_NETWORK_ADAPTER_TYPE=wireless xpra attach ...
Will force wireless mode.
>   *   Is there a way to pre-emptively re-request packets more aggressively?
As per above, this doesn't make sense.
The xpra server will send them as fast as it can, until it hits
bandwidth constraints.
>   *   If I was to dig into the source to work on this, where should I dig?
The bandwidth-limit code is probably a good start, and likely the
culprit for the behaviour you are seeing:
https://xpra.org/trac/ticket/999

Simply turning it off might help, ie:
--bandwidth-limit=20Mbps --bandwidth-detection=no
> 
> If there are any takers to help me out with this, I'll give full logs and remote access to my environment and there is a possibility of payment for services rendered as well, we can negotiate.
I would prefer if you could just provide the steps so that I can
reproduce the problem reliably at my end.
Can you create a ticket or perhaps add to an existing one:
https://xpra.org/trac/ticket/2617
https://xpra.org/trac/ticket/2812

Cheers,
Antoine

> 
> Thank you so much, I'm looking forward to hearing from you.
> 
> David W Johnston
> DLS Technology Corp