The relationship drama, written and directed by Jolie, opened this year's American Film Institute Film Festival in LA.
Hollywood Reporter said it was "far too long" and suffered from "stasis and dramatic flatness".
Variety added that it "it leaves the heart and mind coolly unstirred".
It summed up the film as "a gorgeously unhappy 1970s American couple seeking to escape their demons during an extended stay on the Maltese coast".
Screen Daily said it was "an intimate art-house film for a major studio" and that it was "sure to garner attention because of the marquee attraction" of its stars.
It is the first film in which they have starred together since 2005's Mr & Mrs Smith and is Jolie's third film as director, after Unbroken (2014) and In The Land of Blood and Honey (2011).
Screen Daily said Jolie's "commitment to serious themes is undercut by a rigid, overblown style".
"No matter the type of story she's trying to tell, she often bears down on it so hard that the film doesn't have much room to breathe," it added.
The Wrap called the film "a soporific drama that teeters on parody", adding: "Angelina Jolie Pitt's reputation as a competent filmmaker won't be entirely undone by her third directorial effort but neither will it be enhanced."
It said it was "hard enough watching these two talented actors play blanks who have no chemistry with each other, but the effort seems especially pointless when we learn the root of their problems".
The film is released in the US on 13 November and in the UK on 11 December.
Critics have given a lukewarm reception to Angelina Jolie's By The Sea, in which she stars with husband Brad Pitt, calling it a "vanity project".