Optimize "nearest" Element query prototype
Question

&quot;Is there a more efficient check for siblings that I am missing?&quot;

As done in the example code you gave. No.
You can rewrite the code to remove some of the unneeded complexity. See rewrites at bottom of answer.
Bug
The line...
    const relative = parent.querySelector(selector);

...will throw if parent is null
Unsafe prototype extension
It is a bad idea to add to an existing prototype without precautions. Element.prototype.nearest may not be used at the moment, but at any time it could be added to the DOM, breaking any code that you use it in.
There are three ways to avoid conflicts/future proof prototype extensions.
.1 100% safe. Don't!!! extending the prototype is an open trap ready to bite at any moment.
.2 Almost safe. Use a non standard naming convention eg Element.prototype.select_nearest The standard will not use this name, but this does not mean it is 100% safe as anyone else may use the name.
.3 Safe but awkward. Use a Symbol to define the prototype name. This will guarantee the uniqueness of the name. Unfortunately it does mean you will need to use bracket[notation] to access the property and share that property with code that needs access. For example
// define
const nearest = Symbol(&quot;nearest&quot;);
Element.prototype[nearest] = function() { /* ... code ... */ }   // prototype is global

// call
btn[nearest](&quot;.child&quot;);

To give access to the symbol in global scope use Symbol.for

{   // scope block
    const nearest = Symbol.for(&quot;nearest&quot;);  // find or create
    Element.prototype[nearest] = function() { log(&quot;Nearest&quot;) }  // prototype is global
}

// Access in unconnected scope
{   
    Symbol.for(&quot;nearest&quot;)
    btn[nearest](&quot;.child&quot;);

    // or 
    btn[Symbol.for(&quot;nearest&quot;)](&quot;.child&quot;);
}

Style
A note on style.
Its a bad habit not to delimit code blocks with {} eg if (prev?.matches(selector)) return prev; is safer as if (prev?.matches(selector)) { return prev; }
Unbound function?
Why add to the element prototype a function that calls an unbound arrow function. This just makes the code unduly complex.
You could either call the function nearest directly eg nearest(btn, '.child'); or keep it on the prototype and avoiding the secondary call and the need to pass this See rewrites.
Rewrite
Several rewrites as examples, though apart from the bugs and the points outlined above I don't see any real need for a rewrite.
As prototype
const nearest = Symbol.for(&quot;nearest&quot;);
Element.prototype[nearest] = function(selector) {  
    const prev = this.previousElementSibling;
    if (prev?.matches(selector)) { return prev }
    const next = this.nextElementSibling;
    if (next?.matches(selector)) { return next }
    const parent = this.parentElement;
    if (parent) {
        if (parent.matches(selector)) { return parent }
        return  parent.querySelector(selector) ?? parent[nearest](selector);
    }
}


As function

function nearest(self, sel) {  
    const prev = self.previousElementSibling;
    if (prev?.matches(sel)) { return prev }
    const next = self.nextElementSibling;
    if (next?.matches(sel)) { return next }
    const parent = self.parentElement;
    if (parent) {
        if (parent.matches(sel)) { return parent }
        return  parent.querySelector(sel) ?? parent.nearest(parent, sel);
    }
}


or
const matches = (el, sel) =&gt; el &amp;&amp; el.matches(sel) ? el : undefined;
function nearest(self, sel) {  
    const parent = self.parentElement;
    const sibling = matches(self.previousElementSibling, sel) ?? 
        matches(self.nextElementSibling, sel) ??
        matches(parent, sel) ?? 
        parent?.querySelector(selector) ?? 
        parent?.nearest(parent, selector);
}