Create a Selenium Profile with name FirefoxHeadless.

In the profile set args in Firefox Specific section to ["--headless"]

Now if you will run a test with Selenium - FirefoxHeadless profile - Firefox won't show UI and will operate in headless mode.

Legacy

If the modern method described above does not work you may try alternatives.

Place the following snippet to User.js file.

function GetWebDriverNonProfileCapabilities(profile)
{
    var caps = {};
    
    // set capabilities based on profile name
    if (profile == "FirefoxHeadless")
    {
        caps["args:--headless"] = true;
    }
    
    return caps;
}

There is an alternative way. You can set an environment variable MOZ_HEADLESS=1.

function GetWebDriverNonProfileCapabilities(profile)
{
    var caps = {};
    
    // set capabilities based on profile name
    if (profile == "FirefoxHeadless")
    {
        var WshShell = new ActiveXObject("WScript.Shell");
        var _processEnv = WshShell.Environment("PROCESS");
        _processEnv("MOZ_HEADLESS") = "1";
    }

    return caps;
}