Wayland vs X11: What Actually Stops Working When You Switch

Wayland vs X11: What Will Stop Working When You Switch?

Secure global hotkeys stopped working after a Linux desktop upgrade – but is a push to the Wayland display server protocol to blame? It's not. The specific compatibility boundary: tools that rely on X11's ability to directly scrape the screen or listen to global keyboard events. Identifying if Wayland is active is as simple as typing this in a terminal: echo $XDG_SESSION_TYPE. Does that say x11 or wayland?

For many users, switching from X11 to Wayland is about the only change they've made. The push to adopt the Wayland display server protocol promises better security, smoother animations, and superior hardware handling. But for some users, a lot stops working. At first glance, Wayland looks like a cause of malfunction and obsolescence for Linux desktops.

In fact, the issue isn't Wayland itself. Screen recording, screen sharing, and hotkeys often fail on Wayland simply because the protocol doesn't allow applications to access the screen directly or listen to global keyboard events, the way X11 does.

Screen Sharing and Recording

Wayland lets individual applications draw content to the screen, but does not allow them to directly access screen content or screen feedback. When users try to share their screen or record screen content under Wayland, they usually hit a wall.

There is a practical path forward for screen sharing, at least. The xdg-desktop-portal provides a standardized portal for screen sharing and screen casting between Wayland applications and the host.

Screen sharing can also run on Wayland by using both the PipeWire media service and a web browser that supports it, such as the Firefox web browser or Google Chrome. PipeWire provides a single service that handles how audio and video are sent and received on your Linux desktop.

Meanwhile integrated screen recording tools remain limited although the PipeWire project offers cross-platform session management, which really could open opportunities even for cross-desktop recording. A streamlined way to share a single Wayland app (instead of the whole screen) and the file system are both on the forward-looking PipeWire backlog.

Remote Desktops and Vendor-Specific Pain Points

On Wayland, some third-party remote desktop tools such as TeamViewer and AnyDesk fail to run. GNOME offers a built-in remote desktop tool as a Wayland-friendly alternative. The GNOME Remote Desktop supplies an optional remote desktop server. Remote users can use the GNOME Boxes, the virtual machine manager for GNOME, to run virtual machines that require access to the shared desktop.

Integrating a Wayland-based remote desktop tool into your workflow can yield considerable benefits, especially for developers, IT professionals, and anyone who needs an effective and reliable remote connection. The connection is secure, and data is encrypted.

Many Linux distributions provide support for desktop compositing on Wayland, NVIDIA GPU users installing a new NVIDIA driver may miss the option to start a Wayland session after the driver installation is complete. The user interface does not indicate this is because NVIDIA does not support out-of-the-box support for Wayland in the proprietary display driver NVIDIA provides.

Even with the Wayland driver enabled, there will still be limited or no support for Wayland. GPU users without hardware accelerated video decoding may face issues with screen sharing.

The NVIDIA display driver also has to be updated to work in the Wayland session. This lack of compatibility can cause issues with NVIDIA's proprietary driver, among other patches and forwards while compiling the kernel. Users can check if they are using the proprietary driver with the command nvidia-smi or lsmod | grep nvidia. Please ensure that NVIDIA's proprietary driver version 535 or higher is installed.

When the GBM backend display server is set to the NVIDIA driver, there can be issues depending on the session configuration. If needed, the GBM_BACKEND environment variable should be set to nvidia-drm. Likewise, if the __GLX_VENDOR_LIBRARY_NAME variable isn't set to nvidia, then hardware-accelerated 3D accelerated applications will not launch.

Hotkeys and Automation

Global hotkeys are those keystrokes which control the behavior of a program that's currently running, or perform actions in response to user input. Under X11, while through-framework 32-bit hotkeys maintain their support with similar GNOME integration as before, global hotkeys may not work as under Wayland.

However, keyboard shortcuts work well: They are much more flexible and use the SuppressKeyboards option to help keep focus within the active window, and they remain assigned in the global menu's shortcut editor. Apps can still communicate to your compositor through IPC. Applications catch these IPC events to perform special operations when the hotkey is pressed. With this approach, you achieve global keyboard shortcuts on Wayland without X11's flaws.

Global hotkeys under Wayland remain limited however - the global listener is separate per compositor. Global hotkeys on Wayland add layers of workarounds to script the repeated behavior necessary for true global hotkeys. Both KDE and GNOME still offer some global keyboard shortcuts as seen in the effects.

GNOME Keyboard Shortcuts are also a GNOME-specific workaround to make the most of global hotkeys that do not work under Wayland.

How to Check your Current Desktop Session

If you suspect that your sharing, hotkeys, or viewing issues may be related to your display server, it's easy to check which session type you're currently running on. Open a terminal and type the command:

echo $XDG_SESSION_TYPE

If you are using an X11 desktop session, the display manager will return X11. In the case of a Wayland desktop session, the response should confirm as wayland.

Alternatively, you can identify the current login session with:

loginctl show-session <SESSION_ID> -p Type

When to Switch to X11 in a Single Session

Many of the tools that will draw and record the screen, enable global hotkeys, or maintain a remote display count on access to X11 behavior. These tools remain the bedrock of many users' workflows: Some tools do not have a universal portal-pipewire replacement just yet - so if your career demands tools that have yet to port to Wayland, the solution is to log into X11 for that session, and then Wayland for the rest.

At the GDM login screen, select the session from the menu before entering your password and log in, choosing the X11 session instead of the Wayland session. Ensure you're running a compatible version of GNOME or KDE Plasma. Ensure you're running on a current GNOME or KDE version to harness the presentation features and performance enhancements on Wayland.

What those accustomed to X11 will find in Wayland is that while Wayland possesses no architecture for window decoration, the Wayland protocol remains built on a session where window decorations lack. Wayland remains designed to associate the visible window decoration with the framebuffer compatible with the common Wayland protocol models for decorations.

That means, a GNOME or KDE theme-specific GNOME Shell or KDE service could implement the wayland protocol in the window state. As a result, Wayland remains a problem making Wayland frameworks configure standard-featured window design.

Sources

• Ubuntu.Fan — ubuntu.fan
• All Things How — allthings.how
• GNOME Extensions — extensions.gnome.org