Roofline model for an FPGA
This is the model, assuming two intersecting lines, and you can apply it to anything.

For example, automatic sawing machine:


speed of the circular saw (performance);
speed of the transport feeding machine with the wood (bandwidth).


If rotary force of the saw is not enough, whatever speed transport would have, device (application) will be bound by the circular saw. If transport is very slow, whatever speed saw would have, the device (application) will be bound by the transport.


  This would assume that the remaining resources are reserved for other applications and are unavailable for maximizing the performance


This model assumes that you use this rotary saw device only for sawing the wood. If something else is interferring, you will have to account for it.


  Also, does it make sense to characterize a hardware module in this matter, or the roofline model only applies to an FPGA device as a whole?


It applies to the design. The drill nearby the sawing machine does not participate in the sawing, thus it will not appear on the graph.

As with the example of sawing machine, the goal is to find out the intersection point on the graph, and either optimize both performance and bandwidth for maximal ratio of throughput/cost (point of intersection), or have a good idea what (performance or bandwidth) must be looked at in order to increase that ratio (move lines on the graph/ceilings - e.g. use other type of sawing mechanism if it appears that transport is fast).

Edit: maybe the example of the sawing machine is not that good - if tranport is too fast then machine will just break because disc will stall, motor will overheat and will fry (however it is also typical situation with computing systems when processor is unable to capture data from the bus properly because it lacks the speed for the task given = operation of the software breaks). Other way round looks better - if transport is slow, the disc will be applied with less force, and the output will be limited by the transport.