Jan Grulich

Screen sharing support in WebRTC for Wayland sessions

Last time I wrote about possibility to share a screen of Plasma wayland session, using xdg-desktop-portal and our xdg-desktop-portal-kde backend implementation. Problem was that during that time, there was no application which would implement support for this, leaving my previous effort useless so far. Luckily, this should change pretty soon. I, together with my Red Hat collegues Tomáš Popela and Eike Rathke, have been working for past few weeks on bringing support for screen sharing on Wayland to web browsers. All modern browsers use WebRTC for all audio-video communication, including screen sharing, meaning that in a perfect world, just one implementation would be needed, which is not that exactly this case. Let’s go a bit into the details first.

Each system (Windows, Mac and X11) in WebRTC reimplements an abstract class called DesktopCapturer, which defines API used by applications to support screen sharing. For our wayland support, we started with a new implementation using Pipewire as the core technology used for screen content delivery and for the communication part, to request which screen to share and to obtain Pipewire stream information, we use xdg-desktop-portal, providing simple API to do so. Advantage of using xdg-desktop-portal is that it will work also in sandbox (Flatpak and Snap) and that there is support in Plasma (using xdg-desktop-portal-kde backend) and support for Gnome (using xdg-desktop-portal-gtk backend), both using same API. Using our desktop capturer implementation, WebRTC starts to communicate with xdg-desktop-portal, we set up a session associated with our request, we tell xdg-desktop-portal that we have interest in screen sharing so xdg-desktop-portals asks your backend implementation to provide a dialog to select a screen to share and once this is settled, we request to start screen sharing and backend implementation will set up Pipewire stream, sending us back file descriptor of a Pipewire remote which we can open and connect to it. Once we are connected, we finally start receiving buffers from Pipewire with screen data and providing them to applications. This so far sounds simple and that the work is basically done, but this is unfortunately not an ideal world.

We found out that e.g. Firefox uses some older copy of WebRTC, so while working on WebRTC trunk, to have our work ready for upstream, we had to modify slightly our changes for Firefox older copy in order to be able to test our changes. There is also one thing we haven’t figured out yet, the thing is that Firefox has its own dialog to select a screen (for other capturer implementations) and we were not able to avoid displaying it when our capturer is used. Another thing is Chromium, which seems to be using WebRTC in a different way when compared to Firefox as Chromium is using plugins so this is also something we have to figure out. There are probably still plenty of other things to do before all of this can be upstream, but we have made great progress on this so far. We even had couple of Bluejeans calls where both sides could share their screens, one running Gnome Wayland session and me running Plasma Wayland session.

And for those who like adventures, I have set up a Fedora COPR repository (currently building), with Firefox containing our changes so you can test it yourself, we will need testers soon or later anyway. You just need to make sure that Pipewire is running, otherwise this won’t work, portals (both xdg-desktop-portal and backend implementation) should be started automatically.

Screenshot of Firefox in action:

Screen sharing in Plasma wayland session

One of the important missing features in Plasma wayland session is without a doubt possibility to share your screen or record you screen. To support this you need help of the compositor and somehow deliver all needed information to the client (application), in ideal way something what can be used by all DEs, such as Gnome. Luckily, this has been one of the primary goals of Pipewire, together with support for Flatpak. If you haven’t heard about Pipewire, it’s a new project that wants to improve audio and video handling in Linux, supporting all the usecases handled by PulseAudio and providing same level of handling for video input and output. With Pipewire supporting this, there was recently a new API added to xdg-desktop-portal for screen cast support and also for remote desktop. Using this API, applications can now have access to your screen content on Wayland sessions or in case they are running in sandbox. With various backend implementation, like xdg-desktop-portal-kde or xdg-desktop-portal-gtk, they just need to support one API to target all desktops. Screen cast portal works the way, that the client first needs to create a session between him and xdp (xdg-desktop-portal) backend implementation, user then gets a dialog with a screen he would like to share and starts screen sharing. Once he does that, xdp backend implementation creates a Pipewire stream, sends back response to the client with stream id and then client can connect to that stream and get its content. Once he no longer requests content of the selected stream, xdp backend implementation gets information that nobody is longer connected to the created Pipewire stream and can stop sharing screen information and xdp backend implementation is again ready to accept next requests for screen sharing. This is all happening in the background so there is really no cool picture I can show, at least this dialog which you get when you request to share a screen.

I finished support for screen cast portal in xdg-desktop-portal-kde last week and currently waiting for it to pass review and be merged to master. This is also currently blocked by two not merged reviews, one adding support for sending GBM buffers from KWin and one with new Remote Access Manager interface in KWayland, both authored by Oleg Chernovskiy, for which I’m really greatful. This all will hopefully land soon enough for Plasma 5.13. Testing this is currently a bit complicated as you need everything compiled yourself and besides my testing application there is really no app using this, except maybe Gnome remote desktop, but there should be support in future for this in Krfb, Chrome or in Firefox. Hopefully soon enough.

Last thing I would like to mention is for GSoC students. We also need remote desktop portal support to have full remote desktop experience so I decided to propose this as a GSoC idea so students can choose this interesting stuff as their GSoC work.

Gnome integration for Qt based applications in Flatpak

Following blog post from Patrick Griffis about new themes support in Flatpak, we started working on supporting this new feature too. Currently wherever you start a Qt application, it would always look like a KDE application or something would be missing, like icons so you would end up with bad experience and mixed feelings. This is going to change now as we now support Gnome in form of icons, widget style and Qt platform theme and with this, when you run a Qt application in Gnome, it will look definitely better and more natively than before. We packaged regular adwaita icons which are used by default in Gnome as extension of freedesktop runtime. For widget style we use adwaita-qt style, which is a Qt style attempting to look like Gtk’s adwaita and the most important part putting this all together is QGnomePlatform, a Qt platform theme which reads your Gnome configuration and applies it to running Qt applications. QGnomePlatform also enforces Qt apps to use adwaita icons and adwaita-qt style by default so that’s another reason why it is important. Both adwaita-qt and QGnomePlatform projects are by the way authored by Martin Bříza, a collegue of mine from Red Hat so if you meet him in person somewhere buy him a beer for that he cares about Qt integration in Gnome :). Now coming to a question how to install this and make it work. Basically all you need to do is install following extensions and you shold be done:

flatpak install kderuntime org.freedesktop.Platform.Icontheme.Adwaita
flatpak install kderuntime org.kde.KStyle.Adwaita
flatpak install kderuntime org.kde.PlatformTheme.QGnomePlatform

Your Qt apps running in flatpak should then automatically pick up all of these extensions without any further modification, same way it does automatically when you run it outside the sandbox. Simply done!. I’m also aware that there are more Gtk themes, like adwaita-dark or high-contrast, both are also available in form of Qt style and we will probably package them later, but at this point it is mostly proof of concept that this can be done and works nicely. You can follow our wiki page if you want more information about runtimes, repository with applications and so on and from me it’s all for now.

Btw. below you can see okular running in flatpak and using adwaita-qt style with adwaita icons.

Telegram desktop client for flatpak #3

I have been working lately on fixing issues we had with telegram when using Qt’s flatpak platform plugin to have portals support. This was all crashing due to system tray. Qt tries to fallback to xembed when platform theme doesn’t provide any system tray implementation. Unfortunately Qt assumes that we are using xcb platform plugin (used everywhere by default) and tried to call some functions on it, but unfortunately we were using flatpak platform plugin which loads xcb only internally and because of that it was crashing. To solve that I had to implement my own xembed support into our flatpak platform plugin without using xcb specific functions. This was all happening only on Gnome, because in KDE our platform theme has it’s own implementation of system tray using SNI and xembed is not used there at all. With all those fixes I decided to make telegram to finally use flatpak platform plugin by default.

What is also new is that I created one more branch with alpha releases. Right now I have master branch with latest stable telegram in version 1.0.29 and devel branch with latest alpha release 1.0.37 bringing support for calling.

You can get it the same way as before, following instructions from previous blog post, except that now you have to specify branch you want to install.

Discover: Open desktop ratings

As I’m trying to keep improving the flatpak backend in discover I decided to add support for reviews. To do so I implemented support for GNOME’s Open Desktop Ratings which is rating/review system used by gnome-software. Result of this is now fully functional review system, where you can read user comments and ratings and submit your own reviews. We also use same mechanism as in gnome-software for generating user_hash which identifies you in odrs server and given that you are able to modify/delete your reviews from both discover and gnome-software (note that discover doesn’t support this yet). You can also vote for already existing reviews so others get feedback on how useful each review is. We also decided to use same review system in our PackageKit backend and replace current Ubuntu Popularity Contest system so not only flatpak users will benefit from this. During testing of this review support we’ve hit many UI issues related to review system causing users not to be able see reviews or write new ones which were introduced during transition to kirigami. We fixed all of them and you can look forward to improved experience in the upcoming discover release. To improve this further, we or at least I, would like to also add a new widget showing current total app ratings as of now you can only see reviews with comments only, not overall app rating, but this needs some discussion and design consideration.

Here are some screenshots to prove that this is already working:

If you want to test it, which we would like you to do, you can just compile discover from master branch (with -DBUILD_FlatpakBackend=ON cmake parameter for flatpak support). That’s all from me for now. Have a nice weekend :).

Telegram desktop client for flatpak #2

Some time ago I posted a blog post about how I packed telegram desktop client for flatpak. I’ve been updating it since then in some reasonable intervals as I don’t have time to update it more often and mostly because the telegram client’s build system breaks my build quite oftenly. Recently I discovered that someone managed to patch telegram to use system Qt libraries instead of building own patched Qt and building linking it statically. After some time I managed to adjust those patches and make them work with my build which allows me to use Qt from KDE runtimes. Here are new instructions how to get this work:

Add KDE runtimes repository and install runtimes
$ flatpak remote-add kde --from https://distribute.kde.org/kderuntime.flatpakrepo
$ flatpak install kde org.kde.Platform

And then you can install and run the telegram desktop client:
$ wget https://jgrulich.fedorapeople.org/telegram/keys/telegram.gpg
$ flatpak remote-add --gpg-import=telegram.gpg telegram-desktop https://jgrulich.fedorapeople.org/telegram/repo/
$ flatpak install telegram-desktop org.telegram.desktop
$ flatpak run org.telegram.desktop

Or install it from bundle
$ wget https://jgrulich.fedorapeople.org/telegram/telegram.flatpak
$ flatpak install --bundle telegram.flatpak

The reason I did the hard work to build it with Qt from KDE runtimes is that now you can use telegram with portals support if you run it with “-platform flatpak” parameter. Unfortunately this only makes openURI portal to work as telegram has some internal hacks or whatever to use gtk filedialog so for that reason I still allow to access user’s home directory. There is also a bug if you use telegram under KDE where it tries to use QSystemTrayIcon instead of libappindicator and unfortunately telegram’s system tray icon (the one using QSystemTrayIcon) works only with Qt 5.6.2 and in KDE runtimes we have Qt 5.7.1. The system tray icon is visible, but its context menu doesn’t work so if you want to have fully working system tray icon you have to use “–env=XDG_CURRENT_DESKTOP=gnome” flatpak parameter to force it to use libappindicator.

And that’s it. Sorry you had telegram broken for couple of days while I was fighting with it, but hopefully it will work perfectly now.

KDE Discover flatpak backend

As some of you might already know, I’ve been focusing lately on Flatpak and its integration into KDE. You can check my work on Flatpak KDE portals, which are being currently included in our KDE runtimes and repositories were migrated to KDE git so there has been made some progress since last time I talked about them. Recently I started looking into adding Flatpak support to KDE Discover, to have same support for Flatpak as Gnome has with gnome-software. From the begining it was a nightmare for me as I have never used any glib based library so that slowed me down little bit. I also went through gnome-software code to understand how flatpak integration is done there to get some inspiration. Things went well since then and I have already quite nice stuff to share with you. We currently support most common functionality, like listing available/installed flatpak applications in Discover with possibilities to install/remove/update and of course launch them. We also support flatpak bundles and flatpakref files already. If you don’t believe me then here are some screenshots:

This is quite exciting stuff for me. There is still plenty of things we need to solve and improve, as well as adding possibility to manage flatpak repositories which is quite important feature to have too. This all is already possible to try in master branch of discover, you just need to enable flatpak backend. We will keep intensively working on this and hopefully we will have fully functional flatpak support in Discover soon and ready for the next Plasma release. See you soon!!.

KDE Flatpak portals introduction

I guess you all have heard about Flatpak, Snappy and sandboxing in general. Flatpak is a new way of distributing applications. With Flatpak applications are running in sandbox, which means they are isolated from the rest of your system. With that in mind you need a way how to access some stuff outside the sandbox, like your files or have access to your hardware. To solve this problem the Flatpak developers came up with portals. Portals are high-level session bus APIs that provide access to resources to sandboxed applications. Idea is that there is one DBus service available and visible for the sandboxed application which is supposed to communicate with it to get access outside the sandbox. Then this one service communicates with backend implementations which may be different per desktop so you have in example a Gnome implementation or in my case KDE implementation. The backend then provides dialogs so users can access files or hardware outside the sandbox. To add portal support you need to add your backend implementation, which is quite easy part and if you don’t have any then one from other available ones will be used. Complicated part is to alter the framework the application is using to use the portal DBus service instead of doing what it usually do (e.g. when opening a file dialog you want to send a request over DBus instead of actually displaying the dialog). I’ve been playing with this for some time and I tried to cover the most common or required portals. Let’s go through all of this:

File chooser portal

I think this is the most common and needed portal from all of them. I’ve added a backend implementation as well as support for this to Qt in form of my own Qt platform plugin. Given you can modify Qt’s behaviour using your own Qt platform plugin then I didn’t have to modify Qt at all. The platform plugin alters FileDialog to talk around DBus instead of showing the dialog. The dialog is then shown thanks to the backend implementation and user running app in sandbox shouldn’t notice any difference. The file chooser portal supports both opening and saving files with possibilities to set properties like filters, accept label text etc. which are all already implemented in both like in the platform plugin and the backend.

App chooser portal

This portal allows the user to select an application outside the sandbox which should be used for opening a document, a file or whatever. It is also used when opening an url. On the backend side this is just a simple dialog with list of possible applications associated with given file/document type. On Qt side I added my own implementation of OpenUrl() again into the platform plugin to make it transparent the same way the file dialog works.

Print portal

I guess the most complicated and also quite important portal. I’ve been able to add just backend implementation so far which will be used for printing from gtk applications as gtk already supports printing from sandbox. The idea behind this portal is that when app requests to print a document, it calls PreparePrint() method and the backend presents classic print dialog to the user where he can configure printer, setup the page and paper and so on. This configuration is then passed back to the application where the framework is supposed to create a pdf or ps file already pre-formatted and ready for printing. This file is then passed as file descriptor again to the backend using Print() method and printed. This all works with Qt backend and gtk app, or Gnome backend and gtk app, but unfortunately not yet with Qt apps as I still don’t know how to do this without touching Qt code as this cannot be done on platform plugin level, at least from what I can see. A simple solution can be to don’t touch Qt at all and let the app print the document to file and have a simple utility sending this file through the portal to print it. As you can set application name which should be used for printing in QPrinter, then we maybe can just set this automatically when the app is running in sandbox to make this work automatically. I’m definitely open to your ideas or any help :).

Notification portal

Again very useful portal and not that complicated. I have full backend implementation presenting a notification outside the sandbox when someone calls AddNotification() method. To make this work for KDE applications automatically I had to modify KNotifications framework and implement my own flatpak plugin which replaces NotifyByPopup plugin. All this plugin does instead displaying a popup it calls the portal DBus service which then calls AddNotification() in my backend and presents notification outside the sandbox. Both the plugin and backend supports also sending back information about triggered action so you can also get feedback back to the sandboxed application.

Those are all the portals I have fully or partially covered so far. There are of course more portals designed in Flatpak portal API, like screenshot portal, inhibit portal, but for some of them we don’t have any framework/API or they are not that important so they will be added later. I also have a test application which you can run in sandbox and test all the portals.

How to use flatpak portals

In order to use flatpak portals, either my implementation or the Gnome one, you need to install xdg-desktop-portal which is the main portal DBus service visible to sandboxed applications and which calls backend implementation. For gtk implementation you need xdg-desktop-portal-gtk and for KDE/Qt implementation you need xdg-desktop-portal-kde. This is required to be installed outside the sandbox to present dialogs for file and hardware access. To add support for portals to your sandboxed application you should be fine already with gtk, with Qt and KDE you need my Qt platform plugin and my modifications made to KNotifications. To use those you need to modify flatpak manifest to include them during build so they are available in the sandbox. You can get inspiration in my test app manifest. And finally to use my platform plugin as KNotifications will be used automatically you need to start your app using flatpak run your_app -platform flatpak.

I guess that’s all from me today. Patches and improvements are warmly welcomed as well as any reported issue. If you want some information about Flatpak and KDE in general, we have setup a wiki page where you can find information about KDE runtimes and applications packaged for Flatpak. There will be also a talk at FOSDEM this year about Flatpak and KDE from Aleix Pol.

Introducing new KCM for network configuration

After several attempts trying to write a new KCM for network configuration and actually not finishing any of them, I decided to start one more time, but this time my goal was to simply transform the old editor into a bit nicer KCM and place it into system settings where this was missing for very long time. You can see my current result below.

kcm

kcm1

kcm2

This is still same editor as it was existing before as a standalone application, except the list of connections is now written in QML and is similar to the applet we have in systray. I also had to rewrite the editor widget a bit because it’s currently implemented as a dialog with a tabwidget inside where each tab is represented by one setting widget (e.g. Ipv4SettingWidget), For the new KCM we now have ConnectionEditorBase widget doing all the logic behind, like creating specific setting widgets based on connection type and so on. This widget alone doesn’t display anything and you have to actually subclass it and reimplement method taking care of layouting. This allows me to have e.g. ConnectionEditorTabWidget which just subclasses ConnectionEditorBase and reimplements addWidget() method to place setting widgets into QTabWidget. In future we can also simply write a new UI/layout on top ConnectionEditorBase widget and get rid of the tab layout.

Regarding functionality, it should be already almost on par with functionality of the editor. There are still some missing features (like import/export of VPN), but besides that I think everything else is going well. With the new KCM there are also some minor improvements, like you can now reset your not-saved changes you made to a connection. My plan is to get this into Plasma 5.9 which is supposed to be released in january so I still have plenty of time to finish missing features and address issues I made during this transition and of course time to take your comments into account and make this KCM as most usable for everyone I can :).

I’m going to Akademy 2016

Going-To-Akademy-2016

This year I’m giving a bit technical talk called “Bring NetworkManager support to your Qt applications” where I would like to share with you possibilities of NetworkManagerQt framework. I also host a BoF on Monday about Flatpak, where I plan to discuss mostly KDE Flatpak portals so anyone interested in this topic is welcome. Aaaand to be honest, I wrote this blog post just to be able to use the lovely “I’m going to Akademy” banner. See you in Berlin!!

Posted in KDE
Scroll To Top