Floodtide Serpent and Can't Attack Enchantments
In this situation, the Floodtide Serpent cannot attack.

For this situation, the most important quote (from 508.1) is


  If at any point during the declaration of attackers, the active player is unable to comply with any of the steps listed below, the declaration is illegal; the game returns to the moment before the declaration.


The rest of 508.1c says


  If any restrictions are being disobeyed, the declaration of attackers is illegal.


So once you got to the point that you are disobeying the restriction from Incubation Krasis, the attack is illegal and the game returns to the moment before declaration.

If Incubation Krasis was instead enchanting a different creature and you try to attack with, then when we get to 508.1c, it is not restricting Floodtide Serpent's ability to attack. Then when we get to 508.1g, you determine that the cost of attacking is returning one enchantment you control to its owner's hand. Then when you get to 508.1i, you pay that cost by returning Incubation Krasis to your hand. Then Floodtide Serpent becomes an attacking creature.

Note that the text on Floodtide Serpent is not a restriction, it is a cost. The difference is that restrictions have the form "This creature can't attack" or "This creature can't attack unless [some true/false statement about game state]", while attack costs have the form "This creature can't attack unless [you do something]". It wouldn't make sense to count those costs as restrictions because then every attack with an additional cost would have already failed at 508.1c. That would make cards like Propaganda nonsensical.