Speed up Fluid Simulation for Fluidless Frames
The speedup is possible:

Update based on comment for Blender 2.81a:
(To my best knowledge, as it is a good bit more complex.)

This older version does not use Mantaflow, means it's still using the older fluid solver, which takes it's Time Start and End values as Seconds and not frames.

There is a bit more to the simulation with the old solver:


Time Start and End are values expected in seconds
Using any value higher then 0 as start essentially means the simulation skips the time til then and the result will be like starting in the middle of a movie.
The time between start and end (default 4 seconds) is stretched/squished according to the timeline given Start and End. Lets say start frame 1 and end-frame 200 with 25 fps, would mean the 4 seconds would be simulated over 200 frames.Essentially stretching it as 200/4 is 50 and not 25 fps. So to have the speed correct we would need to set End to 100. If the simulation is set to end at 8 seconds, 200 frames would be right, different fps though can make this equation a bit tougher.
After the simulation is baked, it can be played with an offset, means then you can use a positive offset to start the simulation again somewhere(depending on the frames given) from the middle, or use a negative value to delay the simulation and start it later then it was calculated for. "-25" would make it then start 1 second later at frame 25. (Why at one hand seconds and on the other side frames are used, is beyond me.)
Last but not least, there are 2 other places you may need to visit and adjust:

Making the timeline show seconds as to have a better indicator for how long the simulation may go and

the Output Properties to adjust the "Frame Rate" with it's "fps", as that influences the timeline as well as the calculation you may need to do to figure out what to set the simulation end and later offset.


Friendly suggestion, I'd update to the new stable Blender version 2.82a with Mantaflow if possible, as below shown the setting up became a lot easier:
You may not only have to time the inflow(s), but also set the Frame Start in the domain. See picture below, the frames that will be calculated, be there fluid or not, are set in the Cache Section. All in frames.
Version 2.82a+

While you can always cut off the inflow, the simulation is stubborn and goes through all frames between Frame Start and End when you bake.

Happy Blending