plot_combo.py
def _get_episode_results(args, data, name_task, pkl_file):
    """Given the pickle file which has results (`data`) determine what to plot.

    We could use total_rewards but better to use `task.done`, because total_rewards
    shows the overall sum of delta rewards, but for convex hull tasks, that is not
    interpretable, for cloth and bag-items tasks, we'd rather use coverage or our
    defined success criterion, for cables we probably want percentage of beads in a
    zone, and also total_rewards won't take into account how some initial states
    may start with some reward (especially for cable/cloth stuff), etc. For some
    bag tasks we have to do a little extra filtering. Research code. :-(

    So, to be clear, we use task.done as our metric, EXCEPT for: (a) cable tasks
    (not cable-ring) where we report percentage of beads in a zone, and (b) cloth
    flat stuff, where we report coverage. Otherwise, use `task.done`.

    (13 Oct 2020) realized there is a slight issue, len(episode_list) CAN be 0
    for gt_state agents because before today, I was not saving the initial states.
    AH! This only matters if we wanted to report the gain in reward over the
    starting state. The final state still is OK, and the length is OK since we
    can take the value in the last_info and that is correct. It's just that before
    this, the length of the `episode_list` is one minus that.

    Args:
        data: A list of length 20, since we did 20 test-time episodes per snapshot.
            Each item will contain information about the episode.
        name_task: The task name.
        pkl_file: Pickle file that we are loading from now, useful in case something
            wrong happens.

    Returns:
        Values to report for plotting. The xs is just a scalar representing the
        iteration, which should be the same for all stuff in this `data` file.
        The ys must be whatever we designate for showing results, whether it is
        success rate, coverage, percentage of beads in zone, etc. This is where
        task specific logic occurs.
    """
    total_rewards = []
    lengths = []
    dones = []

    # These are what we actually report! Others above are helpful for debugging.
    iters = []
    metrics = []

    for item in data:
        assert len(item) == 3, len(item)

        # TODO(daniel): argh, see hack above.
        # Use `first_info` for `info` from the first time step in this episode.
        # episode_list = [(act_1,info_1), ..., (act_T,info_T)]
        itr, episode_list, last_info = item
        if len(episode_list) == 0:
            print(f'Note, zero length episode list in {pkl_file}')
            assert 'gt_state' in pkl_file, \
                'We expect this to be a gt_state agent on or before October 13.'
        elif 'gt_state' in pkl_file:
            # AH, if we did gt_state then we don't have 'extras' in the first info.
            # The reason is that first info is directly the env.info, normally we
            # don't add this for image-based policies (since we skip the first
            # addition to the episode_list) but with gt_state we do add it.
            pass
        else:
            # Please fix this!
            first_info = episode_list[0][1]
            assert 'extras' in first_info, \
                f'Issue with {first_info.keys()}\nfor {pkl_file}, length: {len(episode_list)}'
        assert 'final_info' in last_info, last_info.keys()

        # Note: we don't need to report reward or length, leaving here just in case.
        if 'total_rewards' not in last_info['final_info'].keys():
            assert name_task == 'insertion-goal', \
                f'Something went wrong: look at: {last_info} in {pkl_file}'
            total_rewards.append(last_info['reward'])
        else:
            total_rewards.append(last_info['final_info']['total_rewards'])
        lengths.append(last_info['length'])
        iters.append(itr)

        # ------------------------------------------------------------------------------ #
        # We should really be using `task.done`. Actually this over-counts successes for
        # bag-items tasks, since we have 'done' whenever we reach task stage 3 and get any
        # reward, but we should be filtering based on if reward > 0.5. See `main.py` and
        # its `ignore_this_demo()`.
        # Note: we can try putting fraction of item into target for bag-color-goal?
        # ------------------------------------------------------------------------------ #
        done = last_info['final_info']['task.done']
        if name_task not in ['bag-items-easy', 'bag-items-hard']:
            dones.append(done)
        elif name_task == 'bag-items-easy':
            episode_total_reward = last_info['final_info']['total_rewards']
            truly_done = done and episode_total_reward > 0.5
            dones.append(truly_done)
        elif name_task == 'bag-items-hard':
            truly_done = (done and
                         (last_info['final_info']['zone_items_rew'] == 0.5) and
                         (last_info['final_info']['zone_beads_rew'] > 0))
            dones.append(truly_done)

        # Now for the actual metric. Put this in a list (not np.array). Don't forget to
        # adjust the plot labels if necessary (outside this code).
        if args.binary:
            metrics.append(dones[-1])
        else:
            if name_task in ['cloth-flat', 'cloth-flat-notarget']:
                coverage = last_info['final_info']['cloth_coverage']
                metrics.append(coverage)
            elif name_task in ['cable-shape', 'cable-shape-notarget', 'cable-line-notarget']:
                nb_zone = last_info['final_info']['nb_zone']
                nb_beads = last_info['final_info']['nb_beads']
                frac = nb_zone / nb_beads
                metrics.append(frac)
            elif name_task in ['bag-color-goal']:
                # I think this is fair, gives partial credit to stuff inside the bag.
                frac = last_info['final_info']['frac_in_target_bag']
                metrics.append(frac)
            else:
                metrics.append(dones[-1])

    # The iteration should be the same for all in `data`.
    assert np.std(iters) == 0, iters
    xs = int(np.mean(iters))

    # Note: do not make `ys` an np.array since we use `extend` later.
    ys = metrics
    return {'x': xs, 'y': ys, 'total_rewards': total_rewards, 'lengths': lengths, 'dones': dones}
Given the pickle file which has results (`data`) determine what to plot.

    We could use total_rewards but better to use `task.done`, because total_rewards
    shows the overall sum of delta rewards, but for convex hull tasks, that is not
    interpretable, for cloth and bag-items tasks, we'd rather use coverage or our
    defined success criterion, for cables we probably want percentage of beads in a
    zone, and also total_rewards won't take into account how some initial states
    may start with some reward (especially for cable/cloth stuff), etc. For some
    bag tasks we have to do a little extra filtering. Research code. :-(

    So, to be clear, we use task.done as our metric, EXCEPT for: (a) cable tasks
    (not cable-ring) where we report percentage of beads in a zone, and (b) cloth
    flat stuff, where we report coverage. Otherwise, use `task.done`.

    (13 Oct 2020) realized there is a slight issue, len(episode_list) CAN be 0
    for gt_state agents because before today, I was not saving the initial states.
    AH! This only matters if we wanted to report the gain in reward over the
    starting state. The final state still is OK, and the length is OK since we
    can take the value in the last_info and that is correct. It's just that before
    this, the length of the `episode_list` is one minus that.

    Args:
        data: A list of length 20, since we did 20 test-time episodes per snapshot.
            Each item will contain information about the episode.
        name_task: The task name.
        pkl_file: Pickle file that we are loading from now, useful in case something
            wrong happens.

    Returns:
        Values to report for plotting. The xs is just a scalar representing the
        iteration, which should be the same for all stuff in this `data` file.
        The ys must be whatever we designate for showing results, whether it is
        success rate, coverage, percentage of beads in zone, etc. This is where
        task specific logic occurs.
    
Given the pickle file which has results (`data`) determine what to plot.
We could use total_rewards but better to use `task.done`, because total_rewards
shows the overall sum of delta rewards, but for convex hull tasks, that is not
interpretable, for cloth and bag-items tasks, we'd rather use coverage or our
defined success criterion, for cables we probably want percentage of beads in a
zone, and also total_rewards won't take into account how some initial states
may start with some reward (especially for cable/cloth stuff), etc. For some
bag tasks we have to do a little extra filtering. Research code.

So, to be clear, we use task.done as our metric, EXCEPT for: (a) cable tasks
(not cable-ring) where we report percentage of beads in a zone, and (b) cloth
flat stuff, where we report coverage.

(13 Oct 2020) realized there is a slight issue, len(episode_list) CAN be 0
for gt_state agents because before today, I was not saving the initial states.
AH. This only matters if we wanted to report the gain in reward over the
starting state. The final state still is OK, and the length is OK since we
can take the value in the last_info and that is correct. It's just that before
this, the length of the `episode_list` is one minus that.
Given the pickle file which has results ( ` data ` ) determine what to plot . We could use total_rewards but better to use ` task . done ` because total_rewards shows the overall sum of delta rewards but for convex hull tasks that is not interpretable for cloth and bag - items tasks we ' d rather use coverage or our defined success criterion for cables we probably want percentage of beads in a zone and also total_rewards won ' t take into account how some initial states may start with some reward ( especially for cable / cloth stuff ) etc . For some bag tasks we have to do a little extra filtering . Research code . So to be clear we use task . done as our metric EXCEPT for : ( a ) cable tasks ( not cable - ring ) where we report percentage of beads in a zone and ( b ) cloth flat stuff where we report coverage . ( 13 Oct 2020 ) realized there is a slight issue len ( episode_list ) CAN be 0 for gt_state agents because before today I was not saving the initial states . AH . This only matters if we wanted to report the gain in reward over the starting state . The final state still is OK and the length is OK since we can take the value in the last_info and that is correct . It ' s just that before this the length of the ` episode_list ` is one minus that .
def _get_episode_results(args, data, name_task, pkl_file):
    total_rewards = []
    lengths = []
    dones = []
    iters = []
    metrics = []
    for item in data:
        assert len(item) == 3, len(item)
        itr, episode_list, last_info = item
        if len(episode_list) == 0:
            print(f'Note, zero length episode list in {pkl_file}')
            assert 'gt_state' in pkl_file, \
                'We expect this to be a gt_state agent on or before October 13.'
        elif 'gt_state' in pkl_file:
            pass
        else:
            first_info = episode_list[0][1]
            assert 'extras' in first_info, \
                f'Issue with {first_info.keys()}\nfor {pkl_file}, length: {len(episode_list)}'
        assert 'final_info' in last_info, last_info.keys()
        if 'total_rewards' not in last_info['final_info'].keys():
            assert name_task == 'insertion-goal', \
                f'Something went wrong: look at: {last_info} in {pkl_file}'
            total_rewards.append(last_info['reward'])
        else:
            total_rewards.append(last_info['final_info']['total_rewards'])
        lengths.append(last_info['length'])
        iters.append(itr)
        done = last_info['final_info']['task.done']
        if name_task not in ['bag-items-easy', 'bag-items-hard']:
            dones.append(done)
        elif name_task == 'bag-items-easy':
            episode_total_reward = last_info['final_info']['total_rewards']
            truly_done = done and episode_total_reward > 0.5
            dones.append(truly_done)
        elif name_task == 'bag-items-hard':
            truly_done = (done and
                         (last_info['final_info']['zone_items_rew'] == 0.5) and
                         (last_info['final_info']['zone_beads_rew'] > 0))
            dones.append(truly_done)
        if args.binary:
            metrics.append(dones[-1])
        else:
            if name_task in ['cloth-flat', 'cloth-flat-notarget']:
                coverage = last_info['final_info']['cloth_coverage']
                metrics.append(coverage)
            elif name_task in ['cable-shape', 'cable-shape-notarget', 'cable-line-notarget']:
                nb_zone = last_info['final_info']['nb_zone']
                nb_beads = last_info['final_info']['nb_beads']
                frac = nb_zone / nb_beads
                metrics.append(frac)
            elif name_task in ['bag-color-goal']:
                frac = last_info['final_info']['frac_in_target_bag']
                metrics.append(frac)
            else:
                metrics.append(dones[-1])
    assert np.std(iters) == 0, iters
    xs = int(np.mean(iters))
    ys = metrics
    return {'x': xs, 'y': ys, 'total_rewards': total_rewards, 'lengths': lengths, 'dones': dones}
def _get_episode_results ( args , data , name_task , pkl_file ) : total_rewards = [ ] lengths = [ ] dones = [ ] iters = [ ] metrics = [ ] for item in data : assert len ( item ) == 3 , len ( item ) itr , episode_list , last_info = item if len ( episode_list ) == 0 : print ( f'Note, zero length episode list in {pkl_file}' ) assert 'gt_state' in pkl_file , 'We expect this to be a gt_state agent on or before October 13.' elif 'gt_state' in pkl_file : pass else : first_info = episode_list [ 0 ] [ 1 ] assert 'extras' in first_info , f'Issue with {first_info.keys()}\nfor {pkl_file}, length: {len(episode_list)}' assert 'final_info' in last_info , last_info . keys ( ) if 'total_rewards' not in last_info [ 'final_info' ] . keys ( ) : assert name_task == 'insertion-goal' , f'Something went wrong: look at: {last_info} in {pkl_file}' total_rewards . append ( last_info [ 'reward' ] ) else : total_rewards . append ( last_info [ 'final_info' ] [ 'total_rewards' ] ) lengths . append ( last_info [ 'length' ] ) iters . append ( itr ) done = last_info [ 'final_info' ] [ 'task.done' ] if name_task not in [ 'bag-items-easy' , 'bag-items-hard' ] : dones . append ( done ) elif name_task == 'bag-items-easy' : episode_total_reward = last_info [ 'final_info' ] [ 'total_rewards' ] truly_done = done and episode_total_reward > 0.5 dones . append ( truly_done ) elif name_task == 'bag-items-hard' : truly_done = ( done and ( last_info [ 'final_info' ] [ 'zone_items_rew' ] == 0.5 ) and ( last_info [ 'final_info' ] [ 'zone_beads_rew' ] > 0 ) ) dones . append ( truly_done ) if args . binary : metrics . append ( dones [ - 1 ] ) else : if name_task in [ 'cloth-flat' , 'cloth-flat-notarget' ] : coverage = last_info [ 'final_info' ] [ 'cloth_coverage' ] metrics . append ( coverage ) elif name_task in [ 'cable-shape' , 'cable-shape-notarget' , 'cable-line-notarget' ] : nb_zone = last_info [ 'final_info' ] [ 'nb_zone' ] nb_beads = last_info [ 'final_info' ] [ 'nb_beads' ] frac = nb_zone / nb_beads metrics . append ( frac ) elif name_task in [ 'bag-color-goal' ] : frac = last_info [ 'final_info' ] [ 'frac_in_target_bag' ] metrics . append ( frac ) else : metrics . append ( dones [ - 1 ] ) assert np . std ( iters ) == 0 , iters xs = int ( np . mean ( iters ) ) ys = metrics return { 'x' : xs , 'y' : ys , 'total_rewards' : total_rewards , 'lengths' : lengths , 'dones' : dones }
Given the pickle file which has results (`data`) determine what to plot.
Given the pickle file which has results ( ` data ` ) determine what to plot .
"""Given the pickle file which has results (`data`) determine what to plot.

    We could use total_rewards but better to use `task.done`, because total_rewards
    shows the overall sum of delta rewards, but for convex hull tasks, that is not
    interpretable, for cloth and bag-items tasks, we'd rather use coverage or our
    defined success criterion, for cables we probably want percentage of beads in a
    zone, and also total_rewards won't take into account how some initial states
    may start with some reward (especially for cable/cloth stuff), etc. For some
    bag tasks we have to do a little extra filtering. Research code. :-(

    So, to be clear, we use task.done as our metric, EXCEPT for: (a) cable tasks
    (not cable-ring) where we report percentage of beads in a zone, and (b) cloth
    flat stuff, where we report coverage. Otherwise, use `task.done`.

    (13 Oct 2020) realized there is a slight issue, len(episode_list) CAN be 0
    for gt_state agents because before today, I was not saving the initial states.
    AH! This only matters if we wanted to report the gain in reward over the
    starting state. The final state still is OK, and the length is OK since we
    can take the value in the last_info and that is correct. It's just that before
    this, the length of the `episode_list` is one minus that.

    Args:
        data: A list of length 20, since we did 20 test-time episodes per snapshot.
            Each item will contain information about the episode.
        name_task: The task name.
        pkl_file: Pickle file that we are loading from now, useful in case something
            wrong happens.

    Returns:
        Values to report for plotting. The xs is just a scalar representing the
        iteration, which should be the same for all stuff in this `data` file.
        The ys must be whatever we designate for showing results, whether it is
        success rate, coverage, percentage of beads in zone, etc. This is where
        task specific logic occurs.
    """ # These are what we actually report! Others above are helpful for debugging. # TODO(daniel): argh, see hack above. # Use `first_info` for `info` from the first time step in this episode. # episode_list = [(act_1,info_1), ..., (act_T,info_T)] # AH, if we did gt_state then we don't have 'extras' in the first info. # The reason is that first info is directly the env.info, normally we # don't add this for image-based policies (since we skip the first # addition to the episode_list) but with gt_state we do add it. # Please fix this! # Note: we don't need to report reward or length, leaving here just in case. # ------------------------------------------------------------------------------ # # We should really be using `task.done`. Actually this over-counts successes for # bag-items tasks, since we have 'done' whenever we reach task stage 3 and get any # reward, but we should be filtering based on if reward > 0.5. See `main.py` and # its `ignore_this_demo()`. # Note: we can try putting fraction of item into target for bag-color-goal? # ------------------------------------------------------------------------------ # # Now for the actual metric. Put this in a list (not np.array). Don't forget to # adjust the plot labels if necessary (outside this code). # I think this is fair, gives partial credit to stuff inside the bag. # The iteration should be the same for all in `data`. # Note: do not make `ys` an np.array since we use `extend` later.
Values to report for plotting. The xs is just a scalar representing the
iteration, which should be the same for all stuff in this `data` file.
The ys must be whatever we designate for showing results, whether it is
success rate, coverage, percentage of beads in zone, etc. This is where
task specific logic occurs.
Values to report for plotting . The xs is just a scalar representing the iteration which should be the same for all stuff in this ` data ` file . The ys must be whatever we designate for showing results whether it is success rate coverage percentage of beads in zone etc . This is where task specific logic occurs .
args
data
A list of length 20, since we did 20 test-time episodes per snapshot.
Each item will contain information about the episode.
A list of length 20 since we did 20 test - time episodes per snapshot . Each item will contain information about the episode .
name_task
The task name.
The task name .
pkl_file
Pickle file that we are loading from now, useful in case something
wrong happens.
Pickle file that we are loading from now useful in case something wrong happens .