Currently proposed ways to distinguish between a really big planet and a really small star
The dividing line between star and brown dwarf is the mass at which hydrogen fusion via the pp-chain occurs at its core. A brown dwarf below this mass limit will never reach a sustainable equilibrium due to this reaction and will continue to cool as it gets older. A star will reach the long-lived main sequence.

The dividing line between planet and brown dwarf is much less well defined. Some want the definition to be mass-based, and there is a mass-based dividing line at about 13 Jupiter masses between brown dwarfs that fuse deuterium in their interiors (which provides only a brief hiatus in their cooling) and "planets" that do not.

Others argue that any definition of a planets should be based on whether they formed around a star, but this is problematic, because such objects may then be disrupted or escape and also it is not clear where a binary brown dwarf becomes a star plus planet.

The reason that some campaign for a different definition is that the composition and structure of a planet formed around a star could be different to that of a planet that collapses from a small gas cloud. According to the "core accretion model" for gas giant formation, the core of such a planet would consist of solid ice or rocky material, whereas it would be gas all the way down in a less massive version of a brown dwarf. However, the waters are further muddied by ideas that large planets can also form by direct collapse  in the disc of a forming planetary system.