USB Camera to GPU Memory Directly
Haven't heard of this, but looking in GPUdirect description and developer site, it seems like it is needed that the device driver for the 3rd party device, should be capable of assign GPU address to the device, in order to perform DMA operations (sections 1.2 and 1.3).

I am not used to PCIe, but I guess that as you plan to use USB cameras, they will be communicating with the USB driver in your system. So 2 options I see are:


The one you stated, build your own USB host with RDMA capabilities, or 
modify/create a USB driver for your cameras including Nvidia Kernel driver features, to allow these features to be used from a CUDA program.