The hotel is considered a modernist masterpiece and a perfect representation of 1960s Japanese style.
A pair of glass towers designed by Yoshio Taniguchi, son of the Okura's original architect, will replace it.
Built in 1962, the hotel has been favoured by world leaders, celebrities and was even James Bond's preferred abode in You Only Live Twice.
US President Barack Obama joined a roll call of other US leaders when he stayed there in 2009.
A farewell concert was held in the hotel's lobby on Monday. The main building will be torn down next month but an annex will remain open.
The hotel's management say that the redevelopment is necessary for the hotel to keep its five-star rating.
"It is difficult for us now to provide a top Japanese hotel due to the building's age," an Okura spokeswoman told the AFP news agency.
The new hotel will have 510 rooms housed in towers reaching 41 storeys. It is due to open in 2019.
However, the decision to demolish the hotel has been controversial, with a social media campaign and petitions launched to save it.
The British design magazine Monocle described the closure as "a heartbreaking and irreparable loss".
Academics have also said that the Okura's design is unique and irreplaceable.
"It is a destruction of culture. This building is a one and only, which truly integrates Japanese aesthetics with a Western-style hotel," Tadashi Yamane a professor of urban design at Tokyo City University told AFP.
Tokyo's iconic Okura hotel has closed ahead of its demolition and redevelopment for the 2020 Olympics.