Can a transformer work if the core is not circular?
The core doesn't need to be circular, but it must be closed, otherwise the linked flux will be very low.

Moreover, the fact that the pipe is empty doesn't improve the situation, since the flux is concentrated where there is higher permeability, i.e. in the core, but the net section of the core in your case is small. In fact most of the section of the coil is filled with air, which has poor permeability.

You cannot close the core with a simple piece of iron wire. It won't be effective, since the flux will be constrained in the smaller section of the wire. Keep in mind that flux obeys a sort of "Ohm's law for magnetic circuits", called Hopkinson's law.

The role of resistance is taken by a quantity known as reluctance, which is proportional to the net section of the core where the flux flows. The flux is analogous to current. Therefore a tiny section will limit the flux greatly. Since the role of the voltage is taken by the magnetomotive force (MMF) which depends on the current in the coil, you can understand that with the same current in the primary and a high reluctance due to a flux constrained in a little section of wire, the flux will be small, and hence the induced current in the secondary will be small.

If you try to pump more current in the primary, the result will be that the core will saturate (a strongly non-linear effect), with the consequence that its permeability will drop drastically, voiding your attempt.

To have enough coupling between the two coils you need a closed magnetic circuit with substantially low reluctance. Therefore you need a closed path made of ferromagnetic material with a more or less constant section, since any narrowing in the section will increase the reluctance. 

EDIT (prompted by a useful comment by @Asmyldof)

Although, I explained above why your setup is not efficient for a power transformer, and the explanation still stands, there are a couple of issues to be aware of when dealing with transformer operation. This interesting article on transformers has nice pictures and delves into the subject in more detail. I'll point out briefly two key aspects below.

As I said, to be able to have high coupling between primary and secondary winding you need low reluctance and a closed core. This calls for a solid core with a closed magnetic path. Relative to your setup, this will improve the situation, but be aware that using a ferromagnetic core which is also electrically conducting, as iron is, has its drawbacks.

First (and really important for a power transformer) there are core power losses. If the core is made of a good conducting material, eddy currents will be induced in its cross section and this will cause power loss by Joule heating (as in a resistor). This is not the only source of core losses, but for conductive cores it's the most relevant usually. Therefore using a solid iron bar as transformer core you risk to lose much power heating the core itself (that's why cores made of iron are not solid, they are still "filled", but laminated, i.e. made by many layers of insulated material).

The second key aspect is saturation. If you increase the primary current over a certain limit the core will saturate and the permeability will drop, hence the reluctance will rise. Having a not-completely-closed-loop core is, in this case, beneficial. In fact sometimes cores are built with a small air-gap, i.e. the core forms an almost-closed loop, but not quite. The small air gap has much higher reluctance than the rest of the core, hence it increases the overall reluctance of core+gap, which seems bad, but the advantage is that the gap helps linearizing the core, i.e. limits the effect of saturation. Moreover, the gap is very small (say about the thickness a sheet of paper) and this prevents the flux from dispersing in the space around the core, hence it doesn't worsen the overall coupling too much.

Other interesting links about transformers:


Transformers Basics
Transformers Construction