His comments come amid the expenses scandal that has embroiled the government for the past three weeks.
Mr Abbott said on Friday the woman at the centre of the scandal, Speaker of the House of Representatives Bronwyn Bishop, was "deeply remorseful" about her use of taxpayers' funds to travel to friends' weddings and Liberal party functions.
The prime minister has himself had to pay back money spent on travel in the past.
The scandal began when it became public that Ms Bishop had spent A$5,000 ($3,647; Â£2,339) of public funds on a helicopter flight to a Liberal Party function that could have easily been reached by road.
She has also claimed expenses for travel to several weddings of her Liberal Party colleagues.
Australia's rules about travel entitlements for politicians are hazy but Ms Bishop's spending set social media on fire and she soon became the speaker who sparked a thousand memes.
The Facebook page Bronwyn Bishop Memes gained more than 13,000 followers in just a week, as users poked fun at her chopper ride.
Memes have adopted the helicopter as their main tool to mock the speaker, with Ms Bishop using chopper flights for everything from taking extreme holidays and dropping the kids at the local swimming pool to going for a jog - via chopper.
The public have also had fun at her expense with cultural references ranging from Australian TV soap operas to movies such as 101 Dalmatians and Francis Ford Coppola's Apocalypse Now.
Social media taunts even included sly hints towards other expenses scandals, such as the misuse of living away from home allowances for politicians.
Ms Bishop has apologised and is paying back the helicopter costs but she is resisting calls, from independent senators and reportedly even from some members of the government, to resign.
"I love this country very much and it does sadden me that I have let [the public] down," Ms Bishop told reporters.
"I won't resign from the position but I will be working very hard," she said.
Prime Minister Tony Abbott has reminded Australian politicians they cannot "get away with exploiting the rules" on expenses.