From djohnston at dlstech.com Tue Nov 17 17:02:40 2020 From: djohnston at dlstech.com (David Johnston) Date: Tue, 17 Nov 2020 17:02:40 +0000 Subject: [winswitch] XPRA performance on jittery and droppy connections Message-ID: Good Day, sorry for the length but I really need help! Payment offered! I'm working on the design and planning for a large-scale XPRA deployment and there is an issue I'm trying to solve: On connections with higher-than-normal jitter and packet loss/delays, the UI performance degradation is very severe. I'm using primarily the HTML5 client, 4.04 and 4.1, but the full fat client has the same issue. Setting quality=5 helps somewhat but looks really bad. However, if I RDP to something using the same connection, I get full resolution and the degradation is barely noticeable. I am using nested VPNs and packet shaping to reproduce my clients' remote access network conditions. The problem looks exactly like it does on their network so I think it's a fair reproduction. XPRA gets into a state where it's hardly using any bandwidth, but the session pauses greyed-out with the little spinning circle. (Presumably waiting for expected packets that it's not getting) Sometimes I get a session timeout and it tries to re-connect. As soon as I hit it with usage again (scrolling in a window) the problem starts again and the session eventually halts; while using very little bandwidth. XPRA debug logs have tons of backlogs and delays. The actual connection latency may only be 40ms, but it ends up waiting for multiple seconds, apparently without retrying even after the normal latency has been far exceeded. When I look at PCAPs, I can see tons of TCP dup acks and retransmits in both XPRA and RDP (expected given the conditions), but somehow RDP is able to recover better. RDP has a higher packet frequency, but fewer packets close to the MTU. I tried reducing the MTU to 800 but the same exact problem happens. RDP seems to better auto-correct for buffering, delays, and loss. I can measure the network's loss with this tool by cranking the packet size and frequency: https://packetlosstest.com/ As long as I stay below a threshold, the latency and loss are low. RDP seems to be able to find this threshold and stay under it. What I would like to ask is: * Is there a way to force the client.jitter=1000 even on non-wireless? * Is there a way to pre-emptively re-request packets more aggressively? * If I was to dig into the source to work on this, where should I dig? If there are any takers to help me out with this, I'll give full logs and remote access to my environment and there is a possibility of payment for services rendered as well, we can negotiate. Thank you so much, I'm looking forward to hearing from you. David W Johnston DLS Technology Corp From totaam at xpra.org Wed Nov 18 03:35:04 2020 From: totaam at xpra.org (Antoine Martin) Date: Wed, 18 Nov 2020 10:35:04 +0700 Subject: [winswitch] [ANNOUNCE] Xpra 4.0.5: many fixes, none critical Message-ID: Hi, This release fixes a large number of bugs but none of them are critical. There is no urgency to update if you were not affected. Here's a short summary: * caps-lock should work properly for all keys / platforms, finally * HTML5 client fixes: for keyboard (IE), audio, fullscreen switch * socket fixes: SSL upgrades and hostname verification, OpenSSH tunnels, etc * proxy: instance zombies, shutdown, control packet errors * MS Windows: resource leak, missing notifications, keyboard issues * build or packaging fixes: Fedora 33, archlinux, Debian Bullseye, Ubuntu Groovy * Wayland clients: restore workarounds and clipboard, fix connection failures * NVENC: session leak, packaging, SDK v10 support * misc: file transfers, debugging tweaks and fixes, etc Full release notes: * fix caps lock wrongly applied to numeric keys * fix HTML5 client keyboard layout detection with Internet Explorer * fix HTML5 audio forwarding with some versions of Safari * fix HTML5 (un)fullscreen * fix ssl server hostname verification errors * fix syntax errors when using connections using nested ssh tunnels * fix socket_util import errors with some subcommands * fix http / websocket and ssl socket upgrade failures * fix server errors when ws sockets cannot be upgrade to wss * fix ssh command option not being honourd with the client launcher * fix proxy control socket becoming unresponsive after errors * fix proxy shutdown * fix proxy instance zombies on server start failures * fix sqlite authentication module not handling configuration options * fix stdout errors causing server startup or shutdown problems * fix MS Windows resource leak in system tray forwarding * fix MS Windows bubble notifications not showing on some systems * fix MS Windows client keyboard unresponsive issues * fix MS Windows and MacOS websocket servers missing mimetypes module * fix archlinux build path stripping * fix Fedora 33 package dependency issues * fix Wayland detection and workarounds * fix clipboard errors under Wayland * fix client signal listener not forwarding signal messages * fix client failing to connect due to keymap changes (ie: Wayland) * fix client not showing authentication prompt only once per connection * fix opengl debug option for saving buffers as jpeg * fix spurious "missing resolution" errors (often with HTML5 client resizing) * fix duplicated data in bug reports * fix download checksum verification (was not verified with python3 builds) * fix spurious file transfer errors with python3 builds * fix NVENC session leak due to flushing errors, support building with SDK10 * remove "numpy" dependency for builds without NVENC / NVFBC (fixes MacOS installation problems) * add new NVENC presets from SDK v10, workaround deprecation warnings * HTML5 connect page can now specify the display to connect to * avoid starting new threads for file transfers that don't need one * raise default maximum packet size to prevent connection errors with large xdg menu data * don't let bad http requests mess up the server log * prevent peek data or exception message from corrupting the log / stdout * remove dependency on "requests" package introduced in 4.0.4 * add correct packaging for Debian Bullseye and Debian Groovy Gorilla * make it possible to override the Xorg binary path detection The source: https://xpra.org/src/ Downloads: https://xpra.org/trac/wiki/Download Cheers Antoine From thomaslc at umass.edu Mon Nov 23 16:44:31 2020 From: thomaslc at umass.edu (Tom Carpenter) Date: Mon, 23 Nov 2020 11:44:31 -0500 Subject: [winswitch] xpra macOS support Message-ID: <921e1820-6e01-49e8-1781-3b133f9f2e03@umass.edu> I'm new to xpra and have been unsuccessful using it to access a macOS Catalina system. What I've tried so far is, while logged on to a Catalina system, start up xpra using "xpra shadow" and then using another system (also happens to be a macOS Catalina system) launch xpra and attempt to connect to the first by specifying the IP. So, my question is how well supported is macOS currently; I'm thinking my inexperience with xpra is the more likely issue so I'm really just trying to determine if the problem is me or if there's still work to do for macOS, with some versions of macOS being better supported than others. I'll add that I don't have the macOS firewall enabled, there's no antivirus software running on the system I'm trying to access, and on that same system "sh" was added to "System Preferences | Security & Privacy | Privacy | Screen Recording". -- Tom Carpenter Biology 221 Morrill 611 North Pleasant Street Amherst, MA 01003 (413) 577-2311 (P) (413) 545-3243 (F) From mike at cchtml.com Mon Nov 23 21:49:22 2020 From: mike at cchtml.com (Michael Cronenworth) Date: Mon, 23 Nov 2020 15:49:22 -0600 Subject: [winswitch] Disable TCP on systemd socket activation In-Reply-To: <4a30b36a-f006-2f68-1e98-4f1e48c92d91@nagafix.co.uk> References: <78d29d9d-3bdb-40f2-3cb3-012992616619@cchtml.com> <4a30b36a-f006-2f68-1e98-4f1e48c92d91@nagafix.co.uk> Message-ID: <372eb3d5-765c-67bb-77e9-cc7ba4cd9eb2@cchtml.com> On 10/23/20 4:05 AM, Antoine Martin via shifter-users wrote: > Right. > Based on your request, I have just added a simple way to make this work: > https://www.xpra.org/trac/ticket/2914 > > You can grab the latest centos8 beta builds which have this change > already included. I tested this today using the 4.0.5-10.r27999.el8_2 packages, but I do not see any change in behavior. I tried to set the new environment variable in three places: - /etc/sysconfig/xpra (XPRA_SD_SOCKET_TYPE=ssl) - /etc/xpra/conf.d/05_features.conf (env = XPRA_SD_SOCKET_TYPE=ssl) - /lib/systemd/system/xpra.service (modified ExecStart line to include --env=...) After reloading SystemD and restarting the Xpra service and socket, connections to port 14500 are still handled using plain TCP all the way past authentication to the session. I'm using a web browser (Firefox) and the HTML5 client if that makes any difference. Any I doing something wrong or did I mislead you in my enhancement request? Thanks, Michael From antoine at nagafix.co.uk Tue Nov 24 14:09:29 2020 From: antoine at nagafix.co.uk (Antoine Martin) Date: Tue, 24 Nov 2020 21:09:29 +0700 Subject: [winswitch] XPRA performance on jittery and droppy connections In-Reply-To: References: Message-ID: <35b3c471-832a-a224-b6e2-2204bbcacc46@nagafix.co.uk> On 18/11/2020 00:02, David Johnston via shifter-users wrote: > Good Day, sorry for the length but I really need help! Payment offered! The project is looking for sponsors, please contact me privately. > I'm working on the design and planning for a large-scale XPRA deployment and there is an issue I'm trying to solve: > > On connections with higher-than-normal jitter and packet loss/delays, the UI performance degradation is very severe. I'm using primarily the HTML5 client, 4.04 and 4.1, but the full fat client has the same issue. Setting quality=5 helps somewhat but looks really bad. Yes, we have a number of tickets regarding this issue. > However, if I RDP to something using the same connection, I get full resolution and the degradation is barely noticeable. I am using nested VPNs and packet shaping to reproduce my clients' remote access network conditions. The problem looks exactly like it does on their network so I think it's a fair reproduction. Excellent! The main reason why this has not been fixed yet is because it has been very difficult to reproduce reliably. > XPRA gets into a state where it's hardly using any bandwidth, but the session pauses greyed-out with the little spinning circle. (Presumably waiting for expected packets that it's not getting) Correct. The spinners show up when the client is expecting reply packets from the server and it isn't receiving them in a timely manner. > Sometimes I get a session timeout and it tries to re-connect. As soon as I hit it with usage again (scrolling in a window) the problem starts again and the session eventually halts; while using very little bandwidth. > > XPRA debug logs have tons of backlogs and delays. The actual connection latency may only be 40ms, but it ends up waiting for multiple seconds, apparently without retrying even after the normal latency has been far exceeded. I'm not sure what "retrying" you are referring to. The UDP transport is unstable and should not be used, with the TCP based ones xpra does not retry anything and simply relies on TCP for guaranteed delivery. > When I look at PCAPs, I can see tons of TCP dup acks and retransmits in both XPRA and RDP (expected given the conditions), but somehow RDP is able to recover better. RDP has a higher packet frequency, but fewer packets close to the MTU. I tried reducing the MTU to 800 but the same exact problem happens. RDP seems to better auto-correct for buffering, delays, and loss. That's quite possible, and definitely something we can improve. > I can measure the network's loss with this tool by cranking the packet size and frequency: https://packetlosstest.com/ As long as I stay below a threshold, the latency and loss are low. RDP seems to be able to find this threshold and stay under it. > > What I would like to ask is: > > * Is there a way to force the client.jitter=1000 even on non-wireless? XPRA_NETWORK_ADAPTER_TYPE=wireless xpra attach ... Will force wireless mode. > * Is there a way to pre-emptively re-request packets more aggressively? As per above, this doesn't make sense. The xpra server will send them as fast as it can, until it hits bandwidth constraints. > * If I was to dig into the source to work on this, where should I dig? The bandwidth-limit code is probably a good start, and likely the culprit for the behaviour you are seeing: https://xpra.org/trac/ticket/999 Simply turning it off might help, ie: --bandwidth-limit=20Mbps --bandwidth-detection=no > > If there are any takers to help me out with this, I'll give full logs and remote access to my environment and there is a possibility of payment for services rendered as well, we can negotiate. I would prefer if you could just provide the steps so that I can reproduce the problem reliably at my end. Can you create a ticket or perhaps add to an existing one: https://xpra.org/trac/ticket/2617 https://xpra.org/trac/ticket/2812 Cheers, Antoine > > Thank you so much, I'm looking forward to hearing from you. > > David W Johnston > DLS Technology Corp From antoine at nagafix.co.uk Tue Nov 24 14:11:59 2020 From: antoine at nagafix.co.uk (Antoine Martin) Date: Tue, 24 Nov 2020 21:11:59 +0700 Subject: [winswitch] Disable TCP on systemd socket activation In-Reply-To: <372eb3d5-765c-67bb-77e9-cc7ba4cd9eb2@cchtml.com> References: <78d29d9d-3bdb-40f2-3cb3-012992616619@cchtml.com> <4a30b36a-f006-2f68-1e98-4f1e48c92d91@nagafix.co.uk> <372eb3d5-765c-67bb-77e9-cc7ba4cd9eb2@cchtml.com> Message-ID: On 24/11/2020 04:49, Michael Cronenworth via shifter-users wrote: > On 10/23/20 4:05 AM, Antoine Martin via shifter-users wrote: >> Right. >> Based on your request, I have just added a simple way to make this work: >> https://www.xpra.org/trac/ticket/2914 >> >> You can grab the latest centos8 beta builds which have this change >> already included. > > I tested this today using the 4.0.5-10.r27999.el8_2 packages, but I do > not see any change in behavior. Version 4.0.5 is the latest stable release. Beta builds can be found here: https://xpra.org/beta/ Once the fix is verified, please comment in that ticket and I will backport this change to the stable versions. Cheers, Antoine > Any I doing something wrong or did I mislead you in my enhancement request? Just try the beta builds and you should be good to go. > > Thanks, > Michael From mike at cchtml.com Tue Nov 24 14:34:20 2020 From: mike at cchtml.com (Michael Cronenworth) Date: Tue, 24 Nov 2020 08:34:20 -0600 Subject: [winswitch] Disable TCP on systemd socket activation In-Reply-To: References: <78d29d9d-3bdb-40f2-3cb3-012992616619@cchtml.com> <4a30b36a-f006-2f68-1e98-4f1e48c92d91@nagafix.co.uk> <372eb3d5-765c-67bb-77e9-cc7ba4cd9eb2@cchtml.com> Message-ID: On 11/24/20 8:11 AM, Antoine Martin via shifter-users wrote: > Version 4.0.5 is the latest stable release. > Beta builds can be found here: > https://xpra.org/beta/ > > Once the fix is verified, please comment in that ticket and I will > backport this change to the stable versions. Duh. :( I tested the 4.1 beta packages and the feature works as intended, but I am unable to login to a session. I only appended XPRA_SD_SOCKET_TYPE=wss to /etc/sysconfig/xpra and restarted xpra. HTTP connections are dropped. HTTPS connections succeed. Attempting to login results in an authentication failure. Logs show: Error: the proxy server requires an authentication mode, client connection 'wss' does not specify one use 'none' to disable authentication Removing the new environment variable from /etc/sysconfig/xpra and a service restart allows me to login. Thanks, Michael From antoine at nagafix.co.uk Tue Nov 24 14:56:24 2020 From: antoine at nagafix.co.uk (Antoine Martin) Date: Tue, 24 Nov 2020 21:56:24 +0700 Subject: [winswitch] Disable TCP on systemd socket activation In-Reply-To: References: <78d29d9d-3bdb-40f2-3cb3-012992616619@cchtml.com> <4a30b36a-f006-2f68-1e98-4f1e48c92d91@nagafix.co.uk> <372eb3d5-765c-67bb-77e9-cc7ba4cd9eb2@cchtml.com> Message-ID: <2347044c-bf10-938a-af48-bf9755b58676@nagafix.co.uk> (..) > I tested the 4.1 beta packages and the feature works as intended, but I > am unable to login to a session. > > I only appended XPRA_SD_SOCKET_TYPE=wss to /etc/sysconfig/xpra and > restarted xpra. > > HTTP connections are dropped. HTTPS connections succeed. That's expected: the socket is now SSL enabled (wss) and it will reject non-ssl traffic. > Attempting to login results in an authentication failure. Logs show: > > Error: the proxy server requires an authentication mode, > client connection 'wss' does not specify one > use 'none' to disable authentication > > Removing the new environment variable from /etc/sysconfig/xpra and a > service restart allows me to login. Try replacing "--tcp-auth=$TCP_AUTH" with "--wss-auth=sys" We could make this the default too I guess - though it may be confusing to see an auth option defined for a socket type that isn't used by default. Cheers, Antoine > > Thanks, > Michael From mike at cchtml.com Tue Nov 24 15:24:16 2020 From: mike at cchtml.com (Michael Cronenworth) Date: Tue, 24 Nov 2020 09:24:16 -0600 Subject: [winswitch] Disable TCP on systemd socket activation In-Reply-To: <2347044c-bf10-938a-af48-bf9755b58676@nagafix.co.uk> References: <78d29d9d-3bdb-40f2-3cb3-012992616619@cchtml.com> <4a30b36a-f006-2f68-1e98-4f1e48c92d91@nagafix.co.uk> <372eb3d5-765c-67bb-77e9-cc7ba4cd9eb2@cchtml.com> <2347044c-bf10-938a-af48-bf9755b58676@nagafix.co.uk> Message-ID: <5d7cf6e2-b250-7b90-39e7-bf5ae2750768@cchtml.com> On 11/24/20 8:56 AM, Antoine Martin via shifter-users wrote: > Try replacing "--tcp-auth=$TCP_AUTH" with "--wss-auth=sys" > > We could make this the default too I guess - though it may be confusing > to see an auth option defined for a socket type that isn't used by default. Setting wss-auth allows login and everything else seems normal from starting apps to shutting down the server. Thanks. From antoine at nagafix.co.uk Tue Nov 24 17:10:58 2020 From: antoine at nagafix.co.uk (Antoine Martin) Date: Wed, 25 Nov 2020 00:10:58 +0700 Subject: [winswitch] xpra macOS support In-Reply-To: <921e1820-6e01-49e8-1781-3b133f9f2e03@umass.edu> References: <921e1820-6e01-49e8-1781-3b133f9f2e03@umass.edu> Message-ID: On 23/11/2020 23:44, Tom Carpenter via shifter-users wrote: > I'm new to xpra and have been unsuccessful using it to access a macOS > Catalina > system. What I've tried so far is, while logged on to a Catalina system, > start up > xpra using "xpra shadow" Normally, I would also also need to ask: * does the command output show anything useful? * does it show a connection attempt? But.. see below. > and then using another system (also happens to > be a macOS Catalina system) launch xpra and attempt to connect to the first by > specifying the IP. I assume that you're using the dialog GUI in "ssh" mode? > So, my question is how well supported is macOS > currently; > I'm thinking my inexperience with xpra is the more likely issue so I'm > really just trying to determine if the problem is me or if there's still work to do > for macOS, with some versions of macOS being better supported than others. Xpra "shadow" mode used to work quite well on MacOS. It used to be possible to run this simple command from any remote host: xpra shadow ssh://USER at MACOSHOST/ And have the MacOS desktop just show up in a window, but Apple has changed something yet again and so this no longer works. Note: to connect using "ssh" mode, you need to install the "PKG" images so that the "xpra" command will be found by the ssh connection. To make matters worse, it looks like some of the API calls we had been using until now no longer do anything (tested on a 10.14.x system) so the remote mouse clicks and keyboard events aren't emulated properly.. Let me take a look at this as soon as I can and I'll get back to you. Cheers, Antoine > I'll add that I don't have the macOS firewall enabled, there's no antivirus software > running on the system I'm trying to access, and on that same system "sh" was added to > "System Preferences | Security & Privacy | Privacy | Screen Recording". > > From thomaslc at umass.edu Mon Nov 30 16:38:31 2020 From: thomaslc at umass.edu (Tom Carpenter) Date: Mon, 30 Nov 2020 11:38:31 -0500 Subject: [winswitch] xpra macOS support In-Reply-To: References: <921e1820-6e01-49e8-1781-3b133f9f2e03@umass.edu> Message-ID: On 11/24/20 12:10 PM, Antoine Martin via shifter-users wrote: > On 23/11/2020 23:44, Tom Carpenter via shifter-users wrote: >> I'm new to xpra and have been unsuccessful using it to access a macOS >> Catalina >> system. What I've tried so far is, while logged on to a Catalina system, >> start up >> xpra using "xpra shadow" > Normally, I would also also need to ask: > * does the command output show anything useful? > * does it show a connection attempt? > But.. see below. > >> and then using another system (also happens to >> be a macOS Catalina system) launch xpra and attempt to connect to the first by >> specifying the IP. > I assume that you're using the dialog GUI in "ssh" mode? I tried both TCP and SSH from the GUI. I was testing at home and the system I was trying to connecting from and the system I was connecting to were both using wifi. I need to check my wifi router's settings but I'm wondering if the wifi router was the issue - I may have it configured to block connections between wifi devices. > >> So, my question is how well supported is macOS >> currently; >> I'm thinking my inexperience with xpra is the more likely issue so I'm >> really just trying to determine if the problem is me or if there's still work to do >> for macOS, with some versions of macOS being better supported than others. > Xpra "shadow" mode used to work quite well on MacOS. > It used to be possible to run this simple command from any remote host: > xpra shadow ssh://USER at MACOSHOST/ > And have the MacOS desktop just show up in a window, but Apple has > changed something yet again and so this no longer works. > > Note: to connect using "ssh" mode, you need to install the "PKG" images > so that the "xpra" command will be found by the ssh connection. > > To make matters worse, it looks like some of the API calls we had been > using until now no longer do anything (tested on a 10.14.x system) so > the remote mouse clicks and keyboard events aren't emulated properly.. > > Let me take a look at this as soon as I can and I'll get back to you. > > Cheers, > Antoine > > >> I'll add that I don't have the macOS firewall enabled, there's no antivirus software >> running on the system I'm trying to access, and on that same system "sh" was added to >> "System Preferences | Security & Privacy | Privacy | Screen Recording". >> >> > _______________________________________________ > shifter-users mailing list > shifter-users at lists.devloop.org.uk > https://lists.devloop.org.uk/mailman/listinfo/shifter-users I did a clean install of Big Sur on a different system and was able to access that from a Windows 10 system - both systems are on the ethernet network where I work. To get a working connection to the Big Sur system, I started the xpra GUI on the Big Sur system and manually started the xpra shadow server using the "Shadow" button in the GUI, i.e. I haven't been able to establish a connection to the Big Sur system by initiating the connection from the Windows 10 system when the Shadow server isn't already running on the Big Sur system. If you weren't aware of this, the first time the Shadow server is started on a Big Sur system, macOS prompts you with ???? "sh" would like to record this computer's screen. ???? Grant access to this application in Security & Privacy ???? preferences, located in System Preferences. If you click the "Open System Preferences" button, an "sh" item is added to "Security & Privacy | Privacy | Screen Recording", but if you don't place a check mark in the box to the left of "sh" in "Screen Recording", the effect of clicking the mouse buttons or pressing keys on keyboard, on the system you're connnecting from, isn't displayed on the system you're connnecting from. For example, if I position the mouse cursor of my Windows 10 system on the Apple menu in the xpra window for the Big Sur system and left click, the Apple menu opens on the Big Sur system but that activity isn't displayed in the xpra window on the Windows 10 system. The keyboard doesn't seem to work at all until the box to the left of "sh" is checked in "Screen Recording". -- Tom Carpenter Biology 221 Morrill 611 North Pleasant Street Amherst, MA 01003 (413) 577-2311 (P) (413) 545-3243 (F) From mike at cchtml.com Mon Nov 30 23:23:31 2020 From: mike at cchtml.com (Michael Cronenworth) Date: Mon, 30 Nov 2020 17:23:31 -0600 Subject: [winswitch] Random connection drops in browser client Message-ID: Hi, I have had an issue with Xpra dropping and reconnecting sessions while using applications inside the web browser client. I have experienced this for several months and logs do not show a reason. Currently I'm using Xpra 4.0.5. I can leave a session open for hours without reconnect attempts if I leave an application open and idle so I do not feel this is a network connectivity issue. It's only randomly reconnecting after clicking around applications, such as Firefox, inside a session. The randomness frequency is anywhere from once per 5 minutes to a few times per minute. It is not isolated to a single application. Server side logs show the client disconnected and then reconnected. No warnings or other errors. Client side browser logs show: websocket closed: undefined reason: null, reconnect: true, reconnect attempt: 0 connection-list ... tries to reconnect to server ... (web browser shows session for a second) connection-established update encodings: (encodings listed...) ping timeout - reconnecting ... tries to reconnect to server ... (this can cycle once, or a half-dozen times in a row, until finally....) connection-established update encodings: (encodings listed...) server connection is OK The session is then "stable" until the next reconnection episode. Any ideas as to the cause? Thanks, Michael