The surprise move, announced on Tuesday evening, is part of a crackdown on corruption and illegal cash holdings.
Banks will be closed on Wednesday and ATM machines will not be working.
India is overwhelmingly a cash economy. New 500 and 2,000 rupee denomination notes will be issued to replace those removed from circulation.
"Black money and corruption are the biggest obstacles in eradicating poverty," Mr Modi said.
People will be able to exchange their old notes for new ones at banks over the next 50 days but they will no longer be legal tender.
The announcement prompted people across the country to rush to ATMs that offer 100 rupee notes in an attempt not to be left without cash over the next few days.
The move is designed to lock out money that is unaccounted for - known as "black money " - which may have been acquired corruptly, or be being withheld from the tax authorities.
Finance Secretary Shaktikant Das warned people with large stashes of hidden cash that banks would closely monitor the exchange of old notes for new ones.
Mr Modi has set his stall out as a modernising, anti-corruption crusade.
Scrapping notes that are very, very common is his biggest offensive yet. Most transactions in daily life are in cash and 45% of those are in notes in denominations of 500 rupees and over.
Not a single news organisation seemed to know this was coming. I saw one news anchor produce a wad of 500s from his own pocket on air wondering whether these were now just pieces of paper - and also wondering if the bars of Delhi would see a sudden surge of business.
It has caught the country completely off guard. There will also be limits on cash point withdrawals over the next couple of weeks.
Read more from Simon here
The 500 and 1,000 rupee notes are the highest denomination notes in the country and are extremely common in India. Airports, railway stations and hospitals will only accept them until 11 November.
People will be able to exchange their money at banks between 10 November and 30 December.
Mr Modi's ruling Bharatiya Janata Party came into power in 2014 promising to bring billions of dollars of black market money into the country's financial system. His government is half way through its term of office.
The announcement comes just over a month after the government raised nearly $10bn through a tax amnesty for Indians to declare hidden income and assets.
The BBC's Justin Rowlatt in Delhi says the issue of "black money" is a huge problem in India and the latest move is the prime minister's big demonstration that he is taking it seriously.
The idea is to lock out money that is unaccounted for and make it visible for tax purposes - banks will be happy to exchange a few thousand rupees, but will be asking questions of those who turn up with hundreds of thousands or millions in currency.
There are no precise figures available but experts say the government's move could be "a very powerful measure" to curb "black money".  IIFL Holdings Ltd Chairman Nirmal Jain told Bloomberg that it will have "a deflationary impact in general and more specifically on real estate prices - making homes affordable".
It seems not. An individual can put as much as he or she likes into the bank - but withdrawals are limited so the banking system may end up being flooded with cash.
Government guidelines say it is possible to exchange 4,000 rupees - but it is not clear if this is per day or in total.
Critics say the new rules may make it especially difficult for people who choose to keep their cash at home rather than in a bank account and for people with large rupee cash reserves who live abroad.
If there is a legitimate explanation for the cash, the authorities say, it will be possible to exchange it.
Cash points will close on Wednesday and in some places also on Thursday - a development that it seems may cause cash blockages or queues at ATMs.
It's a bold step because many people who voted for Mr Modi were small traders who overwhelmingly did their business in cash.
Our correspondent says these are people who probably do have a few hundred thousand rupees - a few thousand dollars - stored under their beds and will have problems when they turn up in the bank on Thursday trying to change their money.
The move leaves a lot of uncertainty about the Indian economy at least in the short term.
Indian Prime Minister Narendra Modi has announced that the 500 ($7.60) and 1,000 rupee banknotes will be withdrawn from the financial system overnight.