How do I remove an unused patch?
Once you removed the patch from composer.json and run updates, core directory will be overridden by the new release and here the "b" directory will be removed. 

For best practices reasons, I invite you to manage your project patches using composer patches plugin, and to be more specific use external patch file method.