The UK financial regulator will hand out the fine to the bank over the issue of inappropriate mortgage advice to customers.
The FCA is expected to release further details of its ruling on Wednesday.
RBS, which is 81% owned by the UK taxpayer, declined to comment.
This latest fine will be another setback for chief executive Ross McEwan, whose troubled bank made a £8.2bn loss last year.
RBS has already been fined £390m for its part in the rate fixing scandal involving the London interbank offered rate (Libor) and has allocated £3.2bn to compensate customers mis-sold loan insurance.
Royal Bank of Scotland is to be fined about £15m by the city watchdog the Financial Conduct Authority (FCA), the BBC has learned.