Is there any way to send serial data over physical ethernet layer with no encapsulation?
Yes, absolutely you can ditch the network protocol layers and send data "directly".  But, you probably don't want to.

What you do is use standard Ethernet Phy's, magnetics, and connectors.  But instead of using an Ethernet MAC (media access controller) you use an FPGA to send/receive data without the network overhead.  This has been done for several "not quite Ethernet compatible" interfaces like Ethersound, and other industrial protocols.  

One thing that you can't ditch is the packet nature.  You must still transmit data in packets of 64 to about 1500 bytes (some Phy's allow packets up to 8192 bytes).  You can't transmit packets smaller than 64 bytes, or larger than 1500.  And you must allow for the proper "gap" between packets.  But you have complete control over what is in the packets, and any header (if any).

I am glossing over lots of details, however.  It's actually not all that easy, and the requirements are different depending on which Ethernet standard you want to use (10/100/1000 mbps).  In some cases there are signal encoding issues to deal with.  

I would advise that you not do this to Ethernet.  It requires a large amount of skill to design the FPGA logic-- skill that most people do not have.  And the benefits of doing this are minimal.  It's much easier to simply use the standard Ethernet controllers and the associated protocol stacks than to dream up your own thing.