Recording + Streaming on Linux Wayland

versions = gnome:44.1-1.fc38, libwayland:1.22.0-1.fc38, kernel:6.2.13-300.fc38, cpu:amd-r5-5600u-with-built-in-gpu

The distro I'm using is Fedora 38, with the default GNOME + Wayland pair. Although the Wayland was used widely many years ago, it's still weak in support in some edge fields like IME (for CJK users) and maybe screen casting / streaming / recording.

FFMpeg kmsgrab

The main idea here, use FFMpeg's kmsgrab. First, ensure your environment has VAAPI support. For AMD GPU running Fedora Linux, follow this guide.

The awesome BtbN/FFmpeg-Builds is statically built and many useful features are enabled. Choose ffmpeg-master-latest-linux64-gpl.tar.xz, extract the tarball and keep only ffmpeg binary.

Then, use this command:

~/misc/apps/ffmpeg \
  -f kmsgrab -device /dev/dri/card1 -framerate 60 -i - -vf 'hwmap=derive_device=vaapi,scale_vaapi=format=nv12' -c:v h264_vaapi -profile:v high -qp 20 \
  -y rec`date +%s`.mp4

Details (click to show)

watch -n 0.1 \
~/misc/apps/ffmpeg \
  -f kmsgrab -device /dev/dri/card1 -framerate 30 -i - -vf 'hwmap=derive_device=vaapi,scale_vaapi=format=nv12' -c:v h264_vaapi -profile:v high \
  -f rtsp -rtsp_transport tcp rtsp://127.0.0.1:8554/mystream

~/misc/apps/ffmpeg -re -stream_loop -1 -i ../dxchannel-callducks.mp4 -c copy -f rtsp -rtsp_transport tcp rtsp://127.0.0.1:8554/mystream
pactl list short sources

~/misc/apps/ffmpeg -i rec1683730234.mp4 -c:v libx264 -crf 20 -preset slower -y out.mp4
-b:v 2500k -maxrate 4000k
-f kmsgrab -device /dev/dri/card1 -framerate 20 -i - -vf 'hwmap=derive_device=vaapi,hwdownload,format=bgr0' -c:v vp8
-filter:v fps=30
~/misc/apps/ffmpeg -f pulse -ac 2 -i - -y ../o.mp3
-h encoder=h264_vaapi
-y rec`date +%s`.mp4
-qp 40
-vaapi_device /dev/dri/renderD128
-vf 'hwmap=derive_device=vaapi,scale_vaapi=format=nv12'
-vf 'hwmap=derive_device=vaapi,scale_vaapi=w=1920:h=1080:format=nv12'
-f flv rtmp://127.0.0.1:9658/live/home
crop=960:540:480:270 -vf 'hwdownload,format=bgr0'
-maxrate 9M
ffmpeg -format bgra -framerate 60 -f kmsgrab -thread_queue_size 1024 -i - \
  -f alsa -ac 2 -thread_queue_size 1024 -i hw:0 \
  -vf 'hwmap=derive_device=vaapi,scale_vaapi=w=1920:h=1080:format=nv12' \
  -c:v h264_vaapi -g 120 -b:v 3M -maxrate 3M -pix_fmt vaapi_vld -c:a aac -ab 96k -threads $(nproc) \
  output.mkv

In my machine -device /dev/dri/card1 is needed, other environments may use card0 or card2.
If you want -c:v libx264, change format=nv12 to format=bgr0.
Something like Chrome and QEMU-KVM will break recording after entering or exiting full-screen mode, try to delay recording command or avoid full-screen.
For more help with h264_vaapi, try -h encoder=h264_vaapi.

Other Choices

GNOME's built-in recorder. Certainly the simplest but: 1. Recording only, not streaming. 3. Force WebM (libvp8 soft-encoding), terrible for HDPI monitor and needed to be re-encoded before sending file to other people using various machines. So I drop it finally.
OBS Studio. Maybe the most popular full-featured solution. It provides a PipeWire capture source for Wayland. But it's too heavy and complex.

Thanks

Thanks for these article:

https://github.com/BtbN/FFmpeg-Builds

https://github.com/aler9/mediamtx/

https://github.com/aler9/mediamtx/issues/1329#issuecomment-1364853707

https://trac.ffmpeg.org/wiki/Capture/PulseAudio

https://wiki.tonytascioglu.com/scripts/ffmpeg/kmsgrab_screen_capture

https://stackoverflow.com/questions/58754385/record-linux-wayland-drm-screen-using-ffmpegs-kmsgrab-device-with-superimposed

http://www.ffmpeg.org/ffmpeg-codecs.html#VAAPI-encoders

https://trac.ffmpeg.org/wiki/Hardware/VAAPI

https://write.corbpie.com/ffmpeg-list-all-codecs-encoders-decoders-and-formats/