Drivers who get too close to cyclists could end up in court on a charge of dangerous or careless driving.
Those caught too near will be advised that bikes should be given as much room as cars. Evidence will also be captured via a bike-mounted camera.
The scheme could be repeated across the country.
Police officers believe many people are put off cycling on Edinburgh's streets because of the fear of other vehicles coming too close when they overtake.
PC Dominic Doyle, of Police Scotland, said: "A lot of drivers see a person on a bicycle riding on the road in a cycle lane and don't consider that passing them closely puts all road users in danger.
"It could constitute careless or even dangerous driving and you would fail your driving test for that.
"Those drivers that we stop during this operation will be shown how closely they passed the unmarked officer by standing on our specially-designed mat.
"You should allow at least as much room as you would when overtaking a car when passing a person on a bicycle, giving consideration to the maximum speed limit and time and distance available to you."
A total of 398 cyclists have been injured in collisions in the capital over the past two years, including 62 seriously. One has died.
Keith Irving, Cycling Scotland chief executive, said: "Operation Close Pass is a welcome new initiative by Police Scotland to improve road safety.
"Education and enforcement are essential to making safer roads for everyone, whether we are cycling, driving or walking.
"This initiative is one of the many we need to take to tackle the increase in serious injuries amongst people cycling."
Ch Supt Andy Edmonston, head of road policing for Police Scotland, said: "We are constantly working to make Scotland's roads safer for everyone, and I am very pleased to see this innovative approach being piloted in Edinburgh where a large number of people cycle to commute and for pleasure."
An unmarked police cyclist is taking to Edinburgh's streets to highlight the space cars should give bikes on the road.