Restrict what content can be added to a panel page (using panels & organic groups)
I guess that by:

when admin of group A edits the panel page

you mean edit the group's node that is being viewed through a panel,and not that he can edit the panel's settings,
and when you're saying

selects Existing node when adding content to a panel pane

you mean that selects content through field (entity-reference). That said, you don't need to make panels respect OG permissions -I am not sure if that's possible- but to filter out the content showing in that field based on Group A or B. To achieve that you may use field_get_items, see the contents of the field and check which node belongs to which group (every node belonging to a group has a Group id (gid) by that gid you can node_load the group node and access its fields) and configure set/unset the field's values accordingly or even you may use hook_form_alter to do so.