Is there a way for me to use my phone as a Xbox One or 360 controller I let my friend borrow my Xbox 360 controller which was the way for me to stream Xbox One onto my computer. Since he now has my controller I can't really do it, so I was wondering if there's a way to use my phone to control my Xbox.
You should be able to do it with the Xbox One SmartGlass app. 

Navigate the Xbox dashboard, pause and rewind your favorite movie, swipe, pinch, and tap to surf the Web on your TV, and use your device's keyboard for easy text input and search on your console.

Browse the entire Xbox catalog from your device.

Get real-time game strategy within the app.

Compatible with the following devices:
    - Windows 8
    - Windows RT Tablet 8+
    - IOS Devices up to 5S and Ipad Air
    - Android devices that run 4.x and higher

It also provide exclusive and interactive content for movies and TV shows.

More details on its website.

There is also an Xbox 360 SmartGlass app.

Both apps have positive reviews and perform well.

Alternatively, if you just want to play the games streamed to your computer, set up the Xbox App like so:


On your PC, launch the Xbox app.
Select Connect from the panel on the left side.
The Xbox app will scan your home network for available Xbox One consoles. Select the name of the console you want to connect to.


More details on their website

Then download a controller app for your phone like DroidJoy app.


Allows you to use your Android device as an Xbox like controller for playing games on your pc
Requires running a "server" app on your computer to communicate with the controller app which you can download on their website