Upgrading to SQL Server 2017 from SQL Server 2012 Upgrade or New Build? Any suggestions?
Does anyone have any gotchas or war stories to share about this process?

We have a 2012 instance in Prod that has both OLTP and OLAP databases. We are creating two instances, one to be dedicated to each of those tasks. The OLAP instance is a new build, however to minimize end-user down time we are considering an upgrade for the instance that will house the OLAP databases. 

I'm looking for general recommendations that go beyond what I'm reading in the Microsoft docs.
Unless it's a CU or SP, I'd always stick to a side-by-side upgrade over an in-place one.  It allows you to rigorously test and UAT the new version and ensure that all your queries/processes still run efficiently and aren't plagued by new Execution Plans or query engine "features" that can turn out to be a serious hindrance.

If you can't spout the license, try upgrading to 2017 Developer Edition, which is still a full version.  As long as you don't use the Dev edition for production, you can do all the dev/testing/staging you want on it.

Here's my favorite post on the topic. 
The e-book is available in the FRK. (I seem to remember this post being longer but I guess not).
https://www.brentozar.com/archive/2015/10/sql-server-migration-and-upgrade-checklist-video-and-ebook/