swift/proxy/controllers/obj.py
def _determine_chunk_destinations(self, putters, policy):
        """
        Given a list of putters, return a dict where the key is the putter
        and the value is the frag index to use.

        This is done so that we line up handoffs using the same frag index
        (in the primary part list) as the primary that the handoff is standing
        in for.  This lets erasure-code fragment archives wind up on the
        preferred local primary nodes when possible.

        :param putters: a list of swift.proxy.controllers.obj.MIMEPutter
                        instance
        :param policy: A policy instance which should be one of ECStoragePolicy
        """
        # Give each putter a "frag index": the index of the
        # transformed chunk that we'll send to it.
        #
        # For primary nodes, that's just its index (primary 0 gets
        # chunk 0, primary 1 gets chunk 1, and so on). For handoffs,
        # we assign the chunk index of a missing primary.
        handoff_conns = []
        putter_to_frag_index = {}
        for p in putters:
            if p.node_index is not None:
                putter_to_frag_index[p] = policy.get_backend_index(
                    p.node_index)
            else:
                handoff_conns.append(p)

        # Note: we may have more holes than handoffs. This is okay; it
        # just means that we failed to connect to one or more storage
        # nodes. Holes occur when a storage node is down, in which
        # case the connection is not replaced, and when a storage node
        # returns 507, in which case a handoff is used to replace it.

        # lack_list is a dict of list to keep hole indexes
        # e.g. if we have 2 holes for frag index 0 with ec_duplication_factor=2
        # lack_list is like {0: [0], 1: [0]}, and then, if 1 hole found
        # for frag index 1, lack_list will be {0: [0, 1], 1: [0]}.
        # After that, holes will be filled from bigger key
        # (i.e. 1:[0] at first)

        # Grouping all missing fragment indexes for each frag_index
        available_indexes = putter_to_frag_index.values()
        lack_list = collections.defaultdict(list)
        for frag_index in range(policy.ec_n_unique_fragments):
            # Set the missing index to lack_list
            available_count = available_indexes.count(frag_index)
            # N.B. it should be duplication_factor >= lack >= 0
            lack = policy.ec_duplication_factor - available_count
            # now we are missing one or more nodes to store the frag index
            for lack_tier in range(lack):
                lack_list[lack_tier].append(frag_index)

        # Extract the lack_list to a flat list
        holes = []
        for lack_tier, indexes in sorted(lack_list.items(), reverse=True):
            holes.extend(indexes)

        # Fill putter_to_frag_index list with the hole list
        for hole, p in zip(holes, handoff_conns):
            putter_to_frag_index[p] = hole
        return putter_to_frag_index

        Given a list of putters, return a dict where the key is the putter
        and the value is the frag index to use.

        This is done so that we line up handoffs using the same frag index
        (in the primary part list) as the primary that the handoff is standing
        in for.  This lets erasure-code fragment archives wind up on the
        preferred local primary nodes when possible.

        :param putters: a list of swift.proxy.controllers.obj.MIMEPutter
                        instance
        :param policy: A policy instance which should be one of ECStoragePolicy
        
Given a list of putters, return a dict where the key is the putter
and the value is the frag index to use.

This is done so that we line up handoffs using the same frag index
(in the primary part list) as the primary that the handoff is standing
in for.  This lets erasure-code fragment archives wind up on the
preferred local primary nodes when possible.
Given a list of putters return a dict where the key is the putter and the value is the frag index to use . This is done so that we line up handoffs using the same frag index ( in the primary part list ) as the primary that the handoff is standing in for . This lets erasure - code fragment archives wind up on the preferred local primary nodes when possible .
def _determine_chunk_destinations(self, putters, policy):
         For primary nodes, that's just its index (primary 0 gets
         chunk 0, primary 1 gets chunk 1, and so on). For handoffs,
         we assign the chunk index of a missing primary.
        handoff_conns = []
        putter_to_frag_index = {}
        for p in putters:
            if p.node_index is not None:
                putter_to_frag_index[p] = policy.get_backend_index(
                    p.node_index)
            else:
                handoff_conns.append(p)
         Note: we may have more holes than handoffs. This is okay; it
         just means that we failed to connect to one or more storage
         nodes. Holes occur when a storage node is down, in which
         case the connection is not replaced, and when a storage node
         returns 507, in which case a handoff is used to replace it.
         lack_list is a dict of list to keep hole indexes
         e.g. if we have 2 holes for frag index 0 with ec_duplication_factor=2
         lack_list is like {0: [0], 1: [0]}, and then, if 1 hole found
         for frag index 1, lack_list will be {0: [0, 1], 1: [0]}.
         After that, holes will be filled from bigger key
         (i.e. 1:[0] at first)
         Grouping all missing fragment indexes for each frag_index
        available_indexes = putter_to_frag_index.values()
        lack_list = collections.defaultdict(list)
        for frag_index in range(policy.ec_n_unique_fragments):
             Set the missing index to lack_list
            available_count = available_indexes.count(frag_index)
             N.B. it should be duplication_factor >= lack >= 0
            lack = policy.ec_duplication_factor - available_count
             now we are missing one or more nodes to store the frag index
            for lack_tier in range(lack):
                lack_list[lack_tier].append(frag_index)
         Extract the lack_list to a flat list
        holes = []
        for lack_tier, indexes in sorted(lack_list.items(), reverse=True):
            holes.extend(indexes)
         Fill putter_to_frag_index list with the hole list
        for hole, p in zip(holes, handoff_conns):
            putter_to_frag_index[p] = hole
        return putter_to_frag_index
def _determine_chunk_destinations ( self , putters , policy ) : handoff_conns = [ ] putter_to_frag_index = { } for p in putters : if p . node_index is not None : putter_to_frag_index [ p ] = policy . get_backend_index ( p . node_index ) else : handoff_conns . append ( p ) available_indexes = putter_to_frag_index . values ( ) lack_list = collections . defaultdict ( list ) for frag_index in range ( policy . ec_n_unique_fragments ) : available_count = available_indexes . count ( frag_index ) lack = policy . ec_duplication_factor - available_count for lack_tier in range ( lack ) : lack_list [ lack_tier ] . append ( frag_index ) holes = [ ] for lack_tier , indexes in sorted ( lack_list . items ( ) , reverse = True ) : holes . extend ( indexes ) for hole , p in zip ( holes , handoff_conns ) : putter_to_frag_index [ p ] = hole return putter_to_frag_index
Given a list of putters, return a dict where the key is the putter
and the value is the frag index to use.
Given a list of putters return a dict where the key is the putter and the value is the frag index to use .
"""
        Given a list of putters, return a dict where the key is the putter
        and the value is the frag index to use.

        This is done so that we line up handoffs using the same frag index
        (in the primary part list) as the primary that the handoff is standing
        in for.  This lets erasure-code fragment archives wind up on the
        preferred local primary nodes when possible.

        :param putters: a list of swift.proxy.controllers.obj.MIMEPutter
                        instance
        :param policy: A policy instance which should be one of ECStoragePolicy
        """ # Give each putter a "frag index": the index of the # transformed chunk that we'll send to it. # # For primary nodes, that's just its index (primary 0 gets # chunk 0, primary 1 gets chunk 1, and so on). For handoffs, # we assign the chunk index of a missing primary. # Note: we may have more holes than handoffs. This is okay; it # just means that we failed to connect to one or more storage # nodes. Holes occur when a storage node is down, in which # case the connection is not replaced, and when a storage node # returns 507, in which case a handoff is used to replace it. # lack_list is a dict of list to keep hole indexes # e.g. if we have 2 holes for frag index 0 with ec_duplication_factor=2 # lack_list is like {0: [0], 1: [0]}, and then, if 1 hole found # for frag index 1, lack_list will be {0: [0, 1], 1: [0]}. # After that, holes will be filled from bigger key # (i.e. 1:[0] at first) # Grouping all missing fragment indexes for each frag_index # Set the missing index to lack_list # N.B. it should be duplication_factor >= lack >= 0 # now we are missing one or more nodes to store the frag index # Extract the lack_list to a flat list # Fill putter_to_frag_index list with the hole list
self
putters
a list of swift.proxy.controllers.obj.MIMEPutter
instance
a list of swift . proxy . controllers . obj . MIMEPutter instance
policy
A policy instance which should be one of ECStoragePolicy
A policy instance which should be one of ECStoragePolicy