Are USB protocol and connector free or are they patented?
You do not need to pay anything if USB logo and the word "USB" is not important to you and you are happy using a non-unique Vendor ID code. 

The logo is protected by trademark and copyright and you receive a license  to use it with conditions. One of those conditions is that you conduct certification testing. The Vendor ID is how they make sure you pay to support the spec. 

There are certainly patents associated with USB, however, this will only become an issue if you attempt to reimplement (fork) USB itself, simply designing a device, but not testing it or paying for the logo, will not likely run into patent issues. 

There is nothing wrong with using a "USB" connector for other purposes, e.g. USB as a device charging port started out as this kind of "off-label" use, or USB reusing interface for a different signal scheme of your design (custom UART).  As long as you do not try to make this into it's own spec or patent that starts to look like USB... which is when you would need to understand the patents. 

Most companies selling USB products pay to join the USB organization to obtain a vendor ID and may conduct some conformance testing depending on what they make. 


USB Implementers forum (USB-If) owns the USB Specification, logo, and associated trademarks. (https://www.usb.org)


Specifications, for the most part, are free to read. 
USB-IF maintains the list of vendor id's
USB-IF sets the rules for testing and licenses the third party labs to do testing

Every organization that wishes to obtain a Vendor ID joins USB-IF and pays a $5000 annual membership fee or a one time $6000 fee for ID without membership. (https://www.usb.org/getting-vendor-id)


Some microcontroller companies let you sublicense use their Vendor ID for prototypes and testing .
There is an "unnoficial" vendor ID  0xF055 used by open source projects.
Joining also gives you the right to influence the future of the specifications. 
All assigned Vendor ID's a listed here, every one of those companies has paid to obtain this number. Note that many of companies that make processors and microcontrollers are on there. 

In order to use the logo and call yourself a USB device you must pay a license fee of $3500 for the logo. (https://www.usb.org/logo-license)


As a condition of the license, you must pass a compliance test. (https://www.usb.org/compliance)
In order to pass a compliance test you must pay an outside lab and provide test samples that may cost you money. 
Some devices may not require testing, usually applicable if you make many similar devices and do not want to test all of them. 
If you are a member paying annual dues, the $3500 fee is waived.





As an aside, conformance testing in and of itself is a great engineering challenge, while you may wish to avoid the fees associated with USB and conformance testing  with independent lab and this is reasonable for  a hobbyist it is still a good exercise to go through the certification tests on your own.

It serves as a design checklist and implementation guide, and is also there to ensure that your device is inter-operable with other USB devices. The last thing you would want as a designer, even of a hobby product,  is a USB widget that damages or is not interoperable with other USB devices.