Why can't I change anything in hook_node_view_alter? I'm trying to change one of the side panels, no clue what is rendering it but it displays a list of OG members.

Anyways, in template_block_view and hook_page_alter, the widget has already been rendered into HTML. The only place I've been able to find where its still in pre-render form is hook_node_view_alter but no matter what I do to the $build variable, it has precisely 0 effect on anything (yes, I made sure the function is being called). 

So if its not to change the view, what is hook_node_view_alter for? This is on drupal 7.
hook_node_view and hook_node_view_alter are only going to give you control over the inner content of a particular $node that you may be viewing. Such as the inner content when going to www.mysite/node/2

Now if you have additional "blocks" or maybe as you called them "side panels", these are not a part of the $node array and thus cannot be changed in hook_node_view or alter. If the side panel is indeed a block, then you can change it's output with hook_block_view and hook_block_view_alter. Among other things depending on what you are trying to do. The process is to find the block ID, and target it inside your hook. However I may be wrong, because it really depends on what the block is.

You will need to provide a printscreen showing exactly what you are talking about if we are going to figure out the right direction you should take on altering it.

I would suggest downloading the "devel" module, enabling it, then look at the ouptut from hook_block_view_alter and see if it's what you're looking for.

function mymodule_block_view_alter(&amp;$data, $block) {
  dpm($block); //make sure the module devel is installed and enabled for this function
}