Jan Grulich

Google

WebRTC (Chromium): Year end report

- jgrulich 6 Comments

Although Wayland screen sharing is still not yet enabled by default in Chromium, which is what I hoped to achieve this year, I think I can say we are almost there and you can expect it sooner than later. Let’s summarize what we have accomplished this year to make this change happen:

Stream restoration support

You probably remember that you had to go through two portal (xdg-desktop-portal) dialogs all the time you wanted to share your screen. We had first portal dialog to have your selected screen visible in Chromium preview dialog and yet another portal dialog to make your screen shared with the web page itself. This was quite annoying as users had to make the same selection twice. Thanks to a new addition into portal API I was able to implement stream restoration support in WebRTC to bypass the second portal dialog and have your selection instantly shared with the web page itself once you confirm it in the Chromium dialog. This was released in Chromium 105.

Tests for PipeWire (streaming) code

This is not a feature that is visible to users, but it makes an important part of the whole process. It was a long effort to make this happen as it’s something that is not trivial to test and needs some dependencies for the tests itself to run. As a first step we had to bring PipeWire and some of its dependencies into the infrastructure. As easy as it sounds, in order to add a new dependency there you have to add it in form of a CIPD (Chrome Infrastructure Package Deployment) package. This means you write recipes with information how to build and get your package. This all on a CentOS 7 based distribution where you have to work with older libraries or missing libraries and tools (e.g. Meson). This makes your packages later available in third_party directory that is available to both Chromium and WebRTC. Next step was to write the tests itself. The only way how I could test our PipeWire code, code that is all about receiving frames over PipeWire stream, was to write another “testing” stream that will be sending us frames with parameters where we will know what to expect and can verify we received what we were supposed to receive. For the tests itself I used GTest framework which is very well documented and quite comprehensive. Sadly, we were still just in the middle of the process and one would hope that it cannot get more complicated, but the opposite is true. There is a whole runtime setup we need for our tests to run. We need PipeWire and PipeWire session manager to run, otherwise we would never activate and connect our streams. To create the setup I had to write a Python wrapper script that sets all environment variables for PipeWire to find everything in the third_party directory (plugins, libspa, etc.) and run both PipeWire and PipeWire session manager in order to run our test. As a last step, because we had a script that runs the test, we had to create a mapping in the infrastructure making the script itself a launcher of our test + limit this only to x86_64 architecture as that’s the only one where we have PipeWire available as CIPD package.

Here are upstream changes that implement all above mentioned:

UX improvements in Chromium preview dialog

All these improvements were made by Alexander Cooper, Alex is Google engineer working on screen sharing stack in Chromium and WebRTC. I have been intensively working with Alex for the past year and he is my go to person when it comes to code reviews or anything related to screen sharing in Chromium. Alex made a great set of UX improvements in their preview dialog. My original implementation always automatically invoked portal dialog when screen sharing was initiated. This led into one dialog overlaping the other. In most recent Chromium version (I think starting with 107) users will be presented with Chromium preview dialog first asking to share a web tab and only once they pick to share a screen they will get presented with the portal dialog. You can also now re-request the portal dialog in case you pick a wrong screen to share.

Web Engines Hackfest in A Coruña

I travelled to A Coruña in May to attend the Web Engines hackfest and meet Alex Cooper from Google there. This was a perfect opportunity for us to meet in person and I’m really grateful for that. Even though that for me it was a bit stepping outside of my comfort zone as I went somewhere where I didn’t know anyone (besides Alex), but as I later found out, all the people there were super friendly and I’m happy that I met some new faces. We had very productive conversations with Alex and having enough time to talk in person was beneficial for both sides as we could explain to each other technical details of our backgrounds and talk about future plans and current issues.

Firefox and WebRTC rebase

Firefox upstream has been behind with our WebRTC changes for more than ~2 years. This has changed recently with Firefox 106, where they finally rebased their WebRTC version to M103 (Chromium 103). I also provided them list of additional backports they should pick up in order to have fixes for some crashes and issues we have fixed since then. Sadly, as I later found out, even though they did the rebase and all the backports, the new codebase (for screen sharing) is not in use and instead they still keep the old code and use it instead. This is because they don’t have all the dependencies in place and it’s been blocked on this issue since then, hopefully it will get figured out soon and also Firefox users can benefit from all the improvements we did over the past two years. This is not an issue for Fedora users where I created a patch for our Firefox package that enables the new code and use it instead.

Plans for the future

  • Extend PipeWire code test coverage. Currently we test only the essential part of the code, but I would like to further extend the tests with tests for all kinds of metadata we can use (damage regions, crops, mouse cursor etc.)
  • Use portal dialog as the default one. This has some requirements that need to be fulfilled before Chromium can rely on it. We need to show screen/window previous, as currently it can easily happen you pick a wrong screen (especially when you have two identical monitors) and we need a way how to invoke Chromium dialog so users can share a web tab.
  • Fix bugs. I’m currently not aware of any issue and we already fixed a bunch of them this year as people use it more often, but I expect more issues to appear as Wayland becomes more and more dominant and we finally make it enabled by default.

WebRTC: journey to make wayland screen sharing enabled by default

- jgrulich 12 Comments

While we have pretty good support for screen sharing on Wayland in WebRTC, which is included in browsers like Chromium or Firefox, it is still not enabled by default in Chromium and it is kept behind a flag. Not only you have to remember to always enable it for new configurations, but for many users it is not even something they are aware of. This has been my main focus recently and I would like to share with you steps that has been done and what are the plans for the future.

What are the changes to expect in Chromium soon?

DMA-BUF improvements/fixes:

Last year I landed proper DMA-BUF support in WebRTC, which made things way faster. It was working, but it was not perfect and there were some corner cases where it might not be working at all. Here are changes I made recently:

  • Advertise DMA-BUF support when it is really supported. Older versions of PipeWire don’t handle the new way of DMA-BUF negotiation and therefore it shouldn’t be used in such cases. Also using DMA-BUF modifiers requires some recent versions of PipeWire on both sides.
  • Implemented stream renegotiation. In situation when we fail to import a DMA-BUF with given modifier, we will drop this modifier and try to renegotiate stream parameters and go with a different modifier or fallback to shared memory buffers in case we fail completely.
  • Make sure to import DMA-BUF with correct render node. In case of multi-gpu setups, we always picked the first render node to import DMA-BUFs, but it can happen that they were actually produced by a different render node and for that reason we might fail to import them. We now try to get default EGLDisplay, which should be the same one used by the wayland compositor and we should be using same render node.

Better mouse cursor support:

Until now we had mouse cursor as part of the screen content. This means that everytime you moved with your mouse cursor, we had to update whole image and that is very inefficient. The API in WebRTC allows you to implement MouseCursorMonitor which can be used to track mouse changes only and each platform can have both MouseCursorMonitor and DesktopCapturer implementations combined in DesktopAndCursorComposer to get complete image and this all works automatically like a magic. Unlike X11 implementation, our only option is to get everything from one PipeWire stream we connect to and there was no way how to make it shared from DesktopCapturer implementation so it can be used by MouseCursorMonitor implementation. I had to split DesktopCapturer to have xdg-desktop-portal and PipeWire separate implementations. Code for PipeWire is now a SharedScreenCastStream class which is being shared through DesktopCaptureOptions. This is set of parameters associated with each capturer instance and luckily this is also passed to MouseCursorMonitor so we can have access to already initialized PipeWire stream and get the cursor data from there. Implement MouseCursorMonitor with SharedScreenCastStream was then piece of cake.

List of merge requests:

This should again significantly improve performance of screen sharing, because moving with a mouse over a static screen content doesn’t need full screen content update.

Misc:

Last but not least, I’m now in touch with Google developers who help me to review all my changes and discuss with me the current state, issues I have, etc. on monthly meetings we have. The plan is to make this finally enabled by default, hopefully in the first half of this year. There are still some things that need to be solved before this is enabled and there is lot of work ahead, but things look promising.

Plans for the future:

  • Implement stream restoration
    • this will allow us to skip the second portal dialog and I already have plan in my head how to do this in WebRTC. This is currently only supported by xdg-desktop-portal-gnome and xdg-desktop-portal-kde lacks this functionality.
  • Improve UX of the Chromium screen sharing dialog
  • Write tests for all PipeWire/portal code in WebRTC

Even though WebRTC is used in Firefox, I mostly talk about Chromium, because Firefox doesn’t use most recent WebRTC and will need to pick all the changes I did or rebase to newer WebRTC in order to have them. Firefox also has PipeWire/Wayland screen sharing enabled by default and doesn’t have UX issues as there is no internal screen sharing dialog like in Chromium.

I hope all these changes will make your experience better and next time when you read a new blog post I will be informing you about end of this journey.

DMA-BUF support in WebRTC

- jgrulich 1 Comment

It will be almost three years since we landed initial support for screensharing on Wayland with the use of PipeWire in the WebRTC project. This enabled screensharing support in both major Linux browsers. Last year I implemented support for window sharing, added support for PipeWire 0.3 and added support for DMA-BUF and MemFD buffer types. Problem was, as it turned out, the DMA-BUF support was not implemented in a correct way.

The original implementation was using mmap() to get the buffer content. This worked correctly for current Intel GPUs, but was terrifically slow on e.g. AMD GPUs. Proper solution is to use OpenGL context to get the content from buffer. However, there were many implementations using mmap() already, including WebRTC and we needed a way how to properly communicate between the server and the client that when the client advertises DMA-BUF support, it means it doesn’t use mmap() and goes through OpenGL context instead.

Here are some issues if you want to read about the details:

This all resulted into a completely different way how the communication between the consumer and the producer should happen in order to use DMA buffers for way faster and smoother screensharing support. Both sides are now required to query the list of all supported modifiers and add this as a new stream parameter, including flags that the modifiers are mandatory parameter, rest of stream parameters are kept as before so we can keep using other types in case DMA-BUFs are not supported by the producer. Once both sides matches their expectations, we can query whether the stream includes modifiers, based on that we know we can use DMA buffers, which we now properly open using OpenGL context, while we kept mmap() for MemFd buffer types as fallback. This will result into faster screensharing support in your web browsers.

Last but not least, I made screensharing even faster, regardless of buffer type we use. Originally when we received buffer from PipeWire, we copied it to a local variable so we can apply cropping and adjust the position and only after that we copied this adjusted content into a DesktopFrame, which each DesktopCapturer (a class representing screensharing implementation) is supposed to return and let it be displayed by the browser. That means we performed two copy operations for each frame. I improved this implementation and now we copy the PW buffer content directly to a desktop frame which we can return directly so one copy operation less than before. I didn’t do some exact measures, but simply running htop and comparing usage of top 5 processes when sharing a 4k screen I got:

  • Original result: 66%, 64%, 26% 23%, 10%
  • Updated result: 41%, 39%, 19%, 17%, 12%

I also have some other improvements on my TODO list, all of them should bring some additional optimizations and improvements. I will keep you informed once I have news to share with you.

Both changes have been merged into WebRTC, that means it should be in Chrome/Chromium 96 (released during November 2021).

Calendar plasmoid

- jgrulich 19 Comments

I finished my third plasmoid as part of my bachelors thesis. I hope it’s a little bit better than my previous plasmoids and I tried to make it as good equivalent to default KDE calendar plasmoid. Unlike the default KDE calendar plasmoid this plasmoid has separated agenda. You can choose how many weeks you want to display in agenda or choose various colors for various calendars or choose color for all parts of agenda widget.

The second part is “month view” where you can see days with events and display events in selected day. In settings you can choose colors for various cells.

In the end you can add event or edit existing event via default incidence editor. As my previous plasmoids it’s designed to use Akonadi-google resources so if you want to use all collections you will have to add -DALL_COLLECTIONS=true to cmake command.

I hope that someone will like it. If you find some bugs or you have some tips to improvements please let me know.

You can get it from my KDE git repository

Google contacts plasmoid

- jgrulich Leave a comment

I changed some things in my Contacts plasmoid and I don’t know what could I improve on them so I decided to publish it on my blog. It’s my first plasmoid of my bachelor’s thesis and also my first plasmoid experience. I just chose it because it was the simplest part of my bachelor’s thesis and this plasmoid cannot do much. You can simply use it for quickly searching phone numbers or emails of your contacts. You can also edit your contacts or write them email by clicking on their email address. This plasmoid is designed to use Akonadi-google for your Google contacts but it could work with all Akonadi contacts collections. If you want to use all collections you will have to add -DALL_COLLECTIONS=true to cmake command.

Screenshot

I hope that someone will find this useful. If you find some bugs or you have some tips to improvements please let me know.

You can get it from my KDE git repository.

Bye.

Google tasks plasmoid

- jgrulich 2 Comments

I just finished my second google plasmoid which is part of my bachelor’s thesis. It’s a plasmoid for displaying your tasks from your google account and you need for that Akonadi-google resources. I tried to make this plasmoid useful. It is not only for displaying your task but you can mark tasks as completed or simply add new tasks and you also can edit your existing tasks. If you don’t like default colors for tasks or tasks order, you can simply change it from appearance properties of this plasmoid.

Some screenshots:

You can get it from my KDE git repository where you can find my first google contact plasmoid.

If you like this plasmoid I will be happy and I will be also happy if you tell me about some bugs.

Bye 🙂

Scroll To Top