How do chefs come up with recipes for good food?
I am not a professional chef, but I have worked in restaurant kitchens, and have been lucky enough to befriend some enormously talented people - line cooks, pastry chefs, culinary school graduates, and executive chefs.  I'm basing this mostly on my observations of the way they work, and the conversations I've had with them.  Although I never sat down and asked them directly how they came up with dishes, I have sometimes served as a sounding board for their ideas, offered suggestions, and seen menus-in-progress, which I think provide some valuable insight.

Keep in mind that every chef is a bit different.  There are many sources of inspiration that a chef can use when creating a dish.  They can think back to their childhood, look at trends among their peers and in the industry as a whole, look to many different culinary traditions, and so on.  The degree to which each person does this is unique, and it informs a large part of what we'd call their "style".  But, in general, I think there are some fairly universal processes.

There are really two distinct things happening when coming up with a dish: ideation and execution.

By ideation, I mean the process of coming up with an initial idea for a dish.  This is when the chef has pretty much a blank slate (though often times, it's in the context of coming up with a specific type of dish; an appetizer, or a meat course) and they're thinking about possibilities for what they want to do.  This can take a lot of different forms, and I've heard chefs describe how their ideas first came to them in a lot of different ways.  It can be:


A "lightbulb moment", where the concept spontaneously and suddenly appears in their mind, oftentimes when they're working on something else.
Thinking about multiple dishes and trying to re-match "elements" or components of each dish with one another.  For instance, taking the flavors of a ceasar salad (anchovy, egg, parmesan) and combining them with the presentation of a deviled egg.  [I should note that this is my own example, which is why it isn't a great one.  Like I said: I'm not a professional.]
Being inspired by a single ingredient, and finding a few other flavors that complement or contrast that core flavor.  For example, complementing a sweet corn soup with snow crab meat and a melted paprika butter.  [A much better example, from an actual chef!]
Taking a dish (from any number of sources) and deliberately trying to do something different with a single element, component, or flavor.  For example, taking tacos (which often use Latin American flavors) and instead using a southeastern Asian flavor palette (fish sauce, cilantro, ginger, garlic).  


Notice that one of these approaches is not sitting down with a dictionary of flavors and trying to assemble combinations that "scientifically" go together somehow.  Good chefs have an intuitive feel for what flavors go well together, a kind of sensory memory, that is not just informed by flavor pairings but also by culinary traditions.  The list of potential flavor combinations is close to infinite, but people tend to gravitate towards the familiar in their food.  To some degree, diners want comfort and foods or flavors they recognize.  I'd argue that this is true even in molecular gastronomy, where the point is often to present unfamiliar flavors in a very mundane, familiar presentation, or the reverse - take known and loved flavors, and present them in a seemingly bizarre way.  

Chefs know this from their training and experience, and they don't try to draw out flavor combinations from a blank starting point.  Instead, they utilize pairings that they just "know" work well, or that they love.  Sometimes they'll swap out a single component from that pairing for something similar, but in general they're not trying to invent totally new, never-done-before combinations of flavor.

For this reason, a lot of chefs I've known have truly absurd collections of cookbooks and recipes.  One head chef that I worked with for years estimated that he had over 1,000 books (and he mentioned this as he was browsing online, making a wishlist of more that he wanted).  The reason is not because they're cooking these dishes as written - instead, they're drawing on them for initial inspiration.  They're reading the recipe as written and thinking about how they want to utilize the particular method, what different components they can swap out, or how they could adapt the same dish to a totally different method of cooking.

Once the initial idea takes shape, it's time for execution.  This is typically a very iterative process, meaning that it involves cooking the dish, tasting it, and seeking feedback.  That feedback could come from themselves (by thinking about what they'd change or do better) but it often comes from others.  This can happen with specific elements of the dish cooked separately, or the entire assemblage of them.  Chefs usually seem to taste individual pieces as they go until they're happy; then they put everything together, taste the combination, and then try it out on others, but that varies depending on their style and how comfortable they are with what they're creating.

In a healthy team environment, there's usually a lot of collaboration.  Chefs will talk through their partly-completed ideas out loud and share flashes of inspiration they've had for one core element or a dish they want to do.  They might cook a small batch of something and ask everyone to taste.  One of my favorite parts of creating a new menu were workshop sessions with multiple chefs all bouncing ideas off one another, cooking their own dishes, then tasting each as a crew and offering suggestions (or sometimes coming up with totally new ideas by saying "why couldn't we do that like this instead?").  Feedback can also come from front-of-house staff or from guests; the first week or so is a very important time for a new menu because people are seeing things for the first time, and you can get their reactions (if you're humble enough to ask).

This is often the part where chefs will sit down with a "flavor bible" and look for flavors that go well with what's already in the dish.  They may feel that something is missing, and try to find something that will complement other components.  Or perhaps they want to use a different flavor from what's already present, and are looking for a replacement that will work well with other ingredients.  Or, they have one particular ingredient that they're happy with, but it's out of season, or too expensive, and they're trying to find a reasonable substitute.

Good chefs will also typically be thinking about plating at this point, and how the dish should be arranged and presented.  I often hear that "we eat with our eyes", although I think it's more accurate to say that plating is sort of an introduction to the dish.  In order for something to be really, really good, it can't just taste excellent.  It has to sound enticing on the menu, and it has to look (and smell) beautiful when it arrives - all of that contributes to anticipation of the flavor.

There's one more thing that really good, experienced chefs do well which goes beyond coming up with any single dish.  Assembling a menu is really an impressive skill that I think goes under-appreciated.  It involves repeating all of the above, not just for a single dish but for an entire array of them, all of which have to individually sound enticing and balance one another.  Creating an entire menu involves balancing a ridiculous number of factors, like:


The style of the restaurant (or venue, for special events)
The season (what ingredients are fresh, readily available; and just as important, what isn't)
The balance they want to strike between different types of dishes (appetizers, main courses, pastas, sandwiches, desserts, sides, etc.)
The balance they want to strike between different offerings (do they have something vegetarian-friendly?  A fish option?  A meat option?  Items for adventurous diners?  Items for the non-adventurous?)
The general price point they need to consider for their target audience
The variety of specific ingredients (avoiding the same flavor combinations being used in everything - or perhaps, being able to re-use the same components in multiple places for efficiency)


It's really pretty insane, and this in my mind is the defining line between a "chef" and an "executive chef".  I'm always impressed by the skill involved in not just coming up with a list of ideas, but bashing them into something that feels balanced, sounds consistently delicious, and expresses their unique style - and then in turn, having each one of those dishes executed well!  Home cooks can often come up with great dishes, and fantastic, original takes on classics, but you have to be a real pro to come up with 20-30 of them that fit into a cohesive whole.  So, the next time you sit down at a restaurant, scan the menu, and think "this all sounds really amazing", stop and give the chef a silent cheer.

In short?  It's really, really complicated.  There are no hard and fast rules.  It's not a procedure or a science; it's very much an art.  Chefs spend their entire careers learning how to do this well, and there is always the opportunity to get better.