After two hours of talks with President Vladimir Putin and Foreign Secretary Sergei Lavrov in Moscow, he said that the "two foremost nuclear powers cannot have this relationship".
Although there was some "common ground" on Syria, "broad differences" remained.
Mr Lavrov said the two sides needed to overcome "time-bomb issues" inherited from Barack Obama's administration.
Tensions have risen since a suspected chemical attack on the rebel-held Syrian town of Khan Sheikhoun last week that left 89 people dead.
The US and its allies blamed the Syrian government, and the US fired 59 cruise missiles at Syria's Shayrat airbase in response.
Syria denied carrying out the chemical attack and has the support of its main ally, Russia, which on Wednesday vetoed a motion brought to the UN Security Council which would have required Bashar al-Assad's government to co-operate with investigators.
It is the eighth time Moscow has used its veto to protect Syria.
Both Mr Tillerson and Mr Lavrov said the discussions were frank.
Mr Tillerson said: "I expressed the view that the current state of US-Russia relations is at a low point. There is a low level of trust between our two countries."
He added: "We need to attempt to put an end to this steady degradation, which is doing nothing to restore the trust between our two countries or to make progress on the issues of greatest importance to both of us."
It was a view echoed by President Donald Trump later, who said at a press briefing with Nato Secretary General Jens Stoltenberg: "We may be at an all-time low in terms of our relationship with Russia."
Mr Trump said "it would be a fantastic thing" if the nations improved ties, but warned "it may be just the opposite".
Mr Lavrov said a pragmatic conversation was needed to remove the "irritants that piled up under the Obama administration".
Both ministers said there was agreement on working towards a stable and unified Syria, and in defeating so-called Islamic State and other terrorist groups.
Both agreed to work towards an international investigation of the chemical attack.
But it was obvious there were many differences.
Mr Tillerson said there was clear evidence that the recent chemical attack "was planned, directed and executed by Syrian regime forces".
He said the reign of Bashar al-Assad must end and that Mr Trump's description of the Syrian president as "an animal" was "a characterisation President Assad has brought on himself".
Mr Lavrov said there was no evidence Syria was behind the chemical attack and pointed to the turmoil that had been brought to other nations when leaders had been removed.
There was little concrete to come from the meeting, although the fact Mr Putin granted Mr Tillerson an audience - which had been in doubt - was a measure of success.
Mr Putin did say he was ready to restore the US-Russia air safety agreement over Syria, and Mr Tillerson said there would be working groups set up on smaller issues.
Ahead of the meeting, Mr Lavrov said Russia had "a lot of questions regarding very ambiguous and contradictory ideas... coming from Washington".
Afterwards he said "we understand each other better" and there were "many prospects for co-operation with the US".
He said there were "certain time-bomb issues inherited from the previous administration - we know to overcome these obstacles we have to make efforts and that the Americans must move towards us as well".
Earlier, the US and Russian presidents revealed the wide gulf between them over Syria.
Referring to Mr Assad in a Fox Business Network interview, Mr Trump said: "Frankly, Putin is backing a person that's truly an evil person... if Russia didn't go in and back this animal, you wouldn't have a problem right now." He later called Mr Assad a "butcher".
Speaking on Mir television, Mr Putin rejected allegations that Syria was behind the chemical attack, saying Syria had given up its chemical stockpile.
He said: "Where is the proof that Syrian troops used chemical weapons? There isn't any."
Referring to the US air strike, he said: "But there was a violation of international law. That is an obvious fact."
And despite hopes that US-Russia ties would improve under a Trump presidency, Mr Putin said: "One could say that the level of trust on a working level, especially on the military level, has not improved, but rather has deteriorated."
US Secretary of State Rex Tillerson says ties with Russia are at a low point and must improve.