What material should I use to hold a live lamp piece in place?
It looks like that pin is the internal mechanism of what used to be a switch that was integral to the socket.  There is no a way to safely hold that in place.  You are likely to get a high resistance connection or arcing, either of which will eventually destroy the lamp, and could cause a fire in the process.  It generally isn't worth the effort and risk to attempt a home repair/redesign.

Replace the socket.  It looks a lot like this one:



You can find replacement sockets of pretty much every type, including ones with integral switches of different configurations.  This one I linked to at Amazon ($3.28 with free shipping as part of a larger order), but it may also be at your local Home Depot, Lowes, or electrical or lighting supply store.  Or you can redesign the lamp using components that are certified by UL or another organization.  

But repair it by replacing whole components that are designed to be used intact.  The components aren't generally repairable.  If you can't find the exact replacement, modify the brackets or other mechanical parts that hold the socket in place rather than trying to modify the electrical components, themselves.  Hardware store have a big selection of brackets and parts you can use to make things fit an existing fixture.  

The black and white wires in the lamp have crimped terminals that look like they may be salvageable to simplify connecting to the screw terminals on the replacement socket.  You might need a Dremel tool to free them.  If it's too much effort, just snip the wire behind the crimps, pull a little more into the fixture, and make new connections.