Biologists studied a group of Sydney octopuses off Australia's east coast and observed a range of behaviour that may indicate complex social signalling.
Octopuses that stand tall, turn dark and spread their web in a "Nosferatu pose" are likely showing aggression.
Conversely, octopuses may display a pale colour after losing a fight or when trying to avoid conflict.
It was previously believed that octopuses were largely solitary creatures. Changes to body colour and other behaviour were interpreted as tactics to avoid predators.
But Prof Peter Godfrey-Smith said the unique study, based on 53 hours of footage and published on Friday in the journal Current Biology, provided a novel perspective on octopus behaviour.
"[An aggressive] octopus will turn very dark, stand in a way that accentuates its size and it will often seek to stand on a higher spot," Prof Godfrey-Smith, who co-authored the report, said.
"Clearly the unusual stance is not a physiological response. It makes it look as big as it can possibly be, with its arms spread out below and the mantle, the back part of the animal, raised over the head.
"The dark colour is produced in concert with those size-accentuating behaviours. There's no particular physiological reason why darkness should be associated with aggression, but it does give the impression of a larger object."
The researchers, based in Australia and the US, dubbed the stance the "Nosferatu pose", referring to the classic 1920s horror film, because the spread of the octopus's web was reminiscent of a vampire's cape.
Octopuses frequently turned pale while retreating from aggressors and also produced high-contrast patterns known as deimatic displays.
The contrasting patterns were most frequently observed when octopuses were attempting to return to their den after they had been forced out, or in the presence of an aggressive individual.
"Suppose there's a large, aggressive guy there and you want to get back into our den, if you approach with a pale colour it could be interpreted as a non-confrontational behaviour," Prof Godfrey-Smith said.
The study also found that two octopuses displaying dark colours were likely to fight if in close proximity, while a darker-coloured octopus was likely to stand its ground against a lighter-coloured octopus.
Another cephalopod, the cuttlefish, shows a similar set of features, with aggressive males darkening their faces and paler males withdrawing from fights.
The unusually high number of octopuses living together in the colony off Jervis Bay, south of Sydney, allowed a more in-depth observation of interactions than had previously been possible.
Prof Godfrey-Smith's co-author David Scheel and his students combed through 53 hours of footage and observed 186 interactions between the octopuses.
This included the Nosferatu pose, along with reaching, grappling and mating.
Watching these long videos was "much more exhausting than I expected", Prof Godfrey Smith said.
"Identifying an individual from minute to minute is very difficult, because they change colour and shape.
"If the octopus wanders out of the frame, there's not much you can do."
Octopuses may have more complex social interactions than previously believed, a new study has found.