How to change icon shape in Android Oreo 8.0
Since you did not mention the developer preview version. Assuming you are using dev preview 1.

The newly added menu that’s available on the second developer preview of Android O contains a “Change icon shape” option on the main Settings screen that’s located near the bottom of the interface, immediately above the “About” section.
Tapping that option prompts a pop-up window from which you can choose the shape.

Source: https://www.androidheadlines.com/2017/05/android-o-pixel-launcher-supports-five-app-icon-shapes.html