Jellyfin Hardware Transcoding in Docker: QSV, NVENC, VA-API

Setting up jellyfin docker compose hardware transcoding is what separates a media server that idles quietly from one that saturates your CPU every time someone starts a 4K stream. Without GPU acceleration, a single 4K H.265 transcode can push a modern desktop CPU to near-saturation. With hardware transcoding, the GPU handles the decode and encode pipeline and the CPU barely moves.

This guide covers three paths: Intel Quick Sync (QSV) and VA-API, NVIDIA NVENC/NVDEC, and AMD VA-API. All three work well in Docker. The setup complexity is different for each. If Jellyfin is not running in a container yet, start with the Jellyfin Docker Compose setup guide and come back here for the GPU half.

Who This Is For

Good fit if you already have Jellyfin running in Docker and want to add hardware transcoding, or if you’re setting it up fresh and want to do it right the first time. You should be comfortable editing a docker-compose.yml and running a few commands on a Linux host.

Skip this if you’re on Windows with Docker Desktop. Hardware device passthrough in Docker Desktop has extra friction; the Windows installer for Jellyfin is the easier path there.

Time to working hardware transcoding: Intel or AMD takes about 20 minutes. NVIDIA is 45–60 minutes because of the Container Toolkit install step.

Intel Quick Sync and VA-API

Intel integrated GPUs from Skylake (6th gen) onward support H.264 and H.265 hardware decode and encode via Quick Sync. Intel Arc discrete GPUs add AV1 hardware encode. Both expose a DRI render node at /dev/dri/renderD128 on Linux. Passing that device into the container, along with the correct group ID, is the entire setup.

Find your render group ID first:

getent group render | cut -d: -f3

On some distributions this group is named video instead of render. Run ls -la /dev/dri/ to check ownership if you’re unsure.

services:
  jellyfin:
    image: jellyfin/jellyfin
    container_name: jellyfin
    user: 1000:1000
    group_add:
      - "993"  # replace with your actual render GID from getent above
    network_mode: host
    restart: unless-stopped
    volumes:
      - /opt/jellyfin/config:/config
      - /opt/jellyfin/cache:/cache
      - /mnt/media:/media:ro
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128

If you have both an Arc discrete GPU and an Intel iGPU, renderD128 might be either one depending on driver load order. Run vainfo on the host against each device to confirm which is which before picking one.

Verify codec support inside the running container:

docker exec -it jellyfin /usr/lib/jellyfin-ffmpeg/vainfo

A working setup returns a list of VAEntrypoint profiles. If vainfo errors or returns nothing, the device isn’t accessible or the group ID is wrong.

NVIDIA NVENC and NVDEC

NVIDIA requires one extra piece of host software: the NVIDIA Container Toolkit, which allows Docker to expose the GPU inside containers. Install it before touching compose.

On Debian or Ubuntu:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then the compose block:

services:
  jellyfin:
    image: jellyfin/jellyfin
    container_name: jellyfin
    user: 1000:1000
    network_mode: host
    restart: unless-stopped
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - /opt/jellyfin/config:/config
      - /opt/jellyfin/cache:/cache
      - /mnt/media:/media:ro

The official Jellyfin image sets NVIDIA_DRIVER_CAPABILITIES=all and NVIDIA_VISIBLE_DEVICES=all automatically, so you don’t need those in an environment block.

Verify GPU access inside the container:

docker exec -it jellyfin nvidia-smi

One practical note on consumer NVIDIA cards: the cap on concurrent NVENC sessions is a GeForce-wide driver restriction, not a GTX-only one. NVIDIA’s Video Encode and Decode GPU Support Matrix lists a maximum of 12 concurrent sessions for GeForce cards, GTX and RTX alike, while professional and datacenter GPUs are listed as unrestricted. For a single-household Jellyfin server with one or two concurrent streams, that cap is not a constraint.

AMD VA-API

AMD on Linux uses VA-API and the same /dev/dri/renderD128 approach as Intel. The compose block is identical to the Intel block above; only the hardware changes. Driver maturity is the variable: the open-source amdgpu driver handles RDNA and RDNA2 generation cards well. Older GCN hardware is less consistent for hardware decode in practice.

Check what codecs are actually available:

docker exec -it jellyfin /usr/lib/jellyfin-ffmpeg/vainfo --display drm --device /dev/dri/renderD128

If VAProfileH264Main and VAProfileHEVCMain show as supported, you have what you need for the common transcoding cases.

Enabling Transcoding in the Admin Dashboard

Getting the device into the container is half the job. You also need to tell Jellyfin to use it.

1. Open the Admin Dashboard, navigate to Playback, then Transcoding.
2. Under Hardware acceleration, select the appropriate API: VA-API for Intel or AMD on Linux, NVENC for NVIDIA.
3. Enable Allow hardware encoding and Allow hardware decoding.
4. Save and restart the container if Jellyfin prompts for it.

Jellyfin’s hardware acceleration docs include a full matrix of which codecs each GPU generation supports for encode and decode. AV1 is the case worth getting right: hardware decode has been available since NVIDIA’s Ampere generation (RTX 30-series), Intel’s Gen 12 Tiger Lake iGPUs (11th-gen Core) and AMD’s RX 6000 series, while hardware encode is much newer and needs NVIDIA Ada Lovelace (RTX 40-series), an Intel Arc A-series card, or AMD RX 7000.

Troubleshooting

Check the active streams panel. Under Admin > Dashboard > Playback, you can see whether each active stream is direct play, direct stream, or transcoding, and which codec it’s using. This is the fastest way to confirm hardware transcoding is actually running.

Read the ffmpeg log. Set log level to Debug in Playback settings temporarily. The full ffmpeg command Jellyfin built is logged — look for vaapi_device or h264_nvenc in it to confirm the GPU path is active.

Common failure modes:

• Wrong render GID in group_add — vainfo works on the host but fails inside the container
• NVIDIA Container Toolkit not installed — container starts, but nvidia-smi fails inside it
• Hardware acceleration not enabled in the admin UI — the GPU is available but Jellyfin never uses it

If you’re exposing Jellyfin beyond your local network, put it behind a reverse proxy with authentication rather than forwarding the port directly. Jellyfin’s API doesn’t rate-limit by default. Traefik and Nginx Proxy Manager both cover that layer, and either one belongs in front of Jellyfin before you open a port to the internet.

Related across the network

• Proxmox iGPU Passthrough for Plex in an LXC — minilabhq.com
• N100 Mini PC for Plex Transcoding: Quick Sync Limits — minilabhq.com
• Jellyfin vs Plex for Home Server: Which to Run in 2026? — selfhostrealm.com
• Rebuff: Open-Source Prompt Injection Defense, Layer by Layer — aisecreviews.com
• Best Homelab Firewall in 2026: OPNsense, pfSense, UniFi, MikroTik — firewallcompare.com