Design views display and views filters separately? Looking of the exported views code it looks like the code for the filters is usually in a separate section from the fields and their layout.

With that in mind I want to separate out the aspect of views which cover the selection of fields and their display from the design of the filters, so I can continue to improve the filters on one page, whilst designing the data selection and presentation on another. I would then merge them into a new view later.

Is something like this feasible? Of course that means when designing a view I should not remove any of the fields in the display view which are used in the filters.
Views results (sometimes in Fields Format), Exposed Filters, Attachments are all output into their own HTML containers. But you can override each View output in templates (look under Theme: Information under Advanced View settings). 

For whatever purpose you can create two different Views, or two Displays in the same View, define Fields in one, and define Filters in the other. Then later you can export both, combine and replace the relevant parts of the export into one and Import with an override of the existing View so you end up with a combined configuration. 

But why would you want to separate them in the first place? You can add exposed Filters to a View, work on CSS to style them, then later add Fields and style them. 

It isn't necessary to have the exact same Fields a in View that you want to use as Filters. Fields and Filters are independent. Maybe this is what has been tripping you up? 

You can have Filters that filter out one Content type and also filter by creation date. While in Fields show Titles, Author and Taxonomy terms. A completely different set of information in those sections. 

A lot of confusion may be coming from nomenclature. Views Displays are the different outputs you create according to your needs (Page Display, Block Display). Fields are bits of information defined in Content types. Views Results can be displayed as Fields. Filters control qualitatively what is shown in Views, if you expose them then users can change that themselves (Exposed Filters you usually see above Results that let you control the filters on the fly).