Media playback is not supported on this device
Hartley's team won a thrilling third Test 44-40 in Sydney on Saturday to complete a series whitewash.
"We have scored 44 points and not played perfectly so there is still a lot to work on," said Hartley.
"We weren't at our best and we had a number of players sub-par," added England coach Eddie Jones.
England had never won a series in Australia, but they followed a 39-28 victory in the first Test in Brisbane with a 23-7 triumph in Melbourne before completing the job in Sydney.
Jones has guided them to nine successive wins, including the Grand Slam in the Six Nations, since replacing Stuart Lancaster in November after hosts England failed to get past the group phase of the 2015 Rugby World Cup.
Australia reached the final of that tournament, but England have since moved above them to second in the world rankings, behind only World Cup winners New Zealand.
Find out how to get into rugby union with our special guide.
However, Hartley thinks his side can improve further.
"It's been a long season - we didn't want to use this as an excuse but it was evident out there," he added. "Australia threw everything at us, but it was great character by the guys to hang in there and close it out.
"It was our last 80 minutes of the season and I'm proud of how the guys conducted themselves. We have made some history. We put everything into it, can be proud of what we have achieved and will enjoy it.
"We put a lot of pressure on ourselves to play the perfect game and we can all agree it wasn't perfect."
Australian Jones agreed with his captain that his side would need to improve if they were to move above New Zealand.
"That was a brave effort to beat a very good and determined Australian side,'' said Jones.
"We're pleased with the 3-0 result, but realise we've got a lot of work to do.
"We're inconsistent in our defence, we've had two poor games in defence and one very good game and if we want to be the number one team in the world then we need greater consistency.''
England fly-half Owen Farrell kicked 24 points in the third Test and Jones said: "Owen has been absolutely terrific on tour. He's kicked well and played well - his kicking is solar-system class.
"The players have been training since June last year. They haven't had a weekend off. That is a fantastic effort.
"To win that game was a superb effort. They showed fitness, toughness and were brave."
Australia coach Michael Cheika felt his side had wasted a good chance to claim a victory in Sydney.
"There were too many little inaccuracies after really good build-ups," he said. "In two matches we scored more tries and you can't say: 'Oh well, we should have won because we scored more tries.'
"We lost because we made too many little inaccuracies. we lost too many moments in the game and that cut off our flow."
For the latest rugby union news follow @bbcrugbyunion on Twitter.
Subscribe to the BBC Sport newsletter to get our pick of news, features and video sent to your inbox.
England are still searching for their "perfect game" despite clinching a 3-0 Test series victory over Australia, captain Dylan Hartley says.