How much travel is too much travel?
Different people set their thresholds differently, but it all comes down to balance and cost/benefit analysis.

I don't buy into the "don't say no until tenure" argument, because a) until tenure is a pretty big chunk of your life, especially when you add in postdoc, and b) the patterns and expectations you establish before tenure will likely remain with you afterwards.  You need to make a decision which is sustainable for you as your career is now and reassess every couple of years to make sure it is still working.

Myself, I travel 1-2 times per month.  Some academics I know travel nearly once a week; others travel only once or twice a year.  In my observation, this does not really correlate with career stage, but more with desire for public recognition vs. the tradeoffs the person is comfortable with in their life.

So, how to make that judgement for yourself?
Personally, I find Latour's model of science as "credibility investment" a useful analysis tool.  Under this model, you can view scientific life as manipulation of three currencies:


Credibility is the main and most unusual currency of academia, which is generated by your publications, your position and appointments, your visibility in the community, service, etc, and which is invested in order to obtain funding.
Results are the data, theorems, etc. produced by your work, which are the raw material required for generation of credibility, and which generally requires money to produce.
Funding is money or other forms of support, which can be obtained through investment of credibility and is itself invested in order to produce results.


Success as an academic requires management of the flow of these three quantities: your relative levels in any may rise and fall with time, but if you go broke in any of the three areas, your career is in deep trouble.  They are hard to quantify, but with self-reflection and comparison to peers you can often have a sense of whether you are "doing well" or "doing poorly" in each area.

Bringing it back to travel, then: most travel can be viewed as part of obtaining credibility (e.g., conferences, invited talks) or obtaining funding (e.g., visits to program managers).  Their cost, as you have identified, is primarily opportunity to produce results and to write your results up in papers (yes, they cost money too, but your time is usually more valuable).  For each individual conference, you can ask how much this conference is likely to serve those goals, relative to the time that you will lose as a result.  If you become good at working on the road (I love airplanes as internet-free time), then the effective cost may decrease, changing your relative weightings, but the principles remain the same.