What is normally not acceptable to budget in a grant
First of, Marc has it right - these things need to be defined in the grant rules / grant agreement. There is no universal truth in these matters. For instance,  European Union funded projects almost always contain travel budgets, which can be used for project meetings or research-related trips (e.g., conferences). The  Austrian Science Fundation (FWF), on the other hand, explicitly disallow travel budgets for conference trips and claims that these expenses need to be covered by overheads. Contrary, the Swiss National Science Fundation (SNF) works very similar to FWF, but allows for travel budgets.

That being said, typically only costs that are specific to running a project and which would not have been accrued without the project are covered. Besides personnell costs, that typically includes buying special hardware that is essential to the project, but which is not usually used in the rest of the PI's research. What is usually not covered are indirect costs that stem from hiring any researcher. This includes mundane things, such as the office space for the researcher, desks, telephony costs, coffee (if you work in a place that buys such things for employees), and IT equipment.

As for the "why are CPUs covered, but not entire computers?" case - I presume that your grant does not allow you to buy just any CPU from your grant, but it should allow hardware specific to your project (e.g., buying or renting a powerful server or cluster for number crunching in relation to a computationally intense project). It will likely not allow you to buy just a regular desktop CPU, just as it would not allow you to buy the entire desktop computer.