Title of Invention: Read-Modify-Write Processing of Chunks at the Storage Server Level in a Distributed Object Storage System Abstract: The present disclosure provides systems and methods for sharding objects stored in a distributed storage system. Such sharding may be advantageously utilized for an object that stores a collection of key-value records and for otherwise encoded objects. The object sharding techniques disclosed herein advantageously enable a read-modify-write process at the storage server level. One embodiment disclosed herein provides a method of creating a new chunk by modifying a payload of an existing chunk at the storage server level in a distributed object storage system. The method includes validating the new chunk at the gateway server. Other embodiments, aspects and features are also disclosed. Detailed Description: CROSS-REFERENCE TO RELATED APPLICATION(S) The present application claims the benefit of U.S. Provisional Patent Application No. 62/098,727, filed Dec. 31, 2014, the disclosure of which is hereby incorporated by reference in its entirety. BACKGROUND 1. Technical Field The present disclosure relates generally to data storage systems and data communication systems. 2. Description of the Background Art With the increasing amount of data is being created, there is increasing demand for data storage solutions. Storing data using a cloud storage service is a solution that is growing in popularity. A cloud storage service may be publicly-available or private to a particular enterprise or organization. Popular public cloud storage services include Amazon S3™, the Google File System, and the OpenStack Object Storage (Swift) System™. Cloud storage systems may provide “get” and “put” access to objects, where an object includes a payload of data being stored. The payload of an object may be stored in parts referred to as “chunks”. Using chunks enables the parallel transfer of the payload and allows the payload of a single large object to be spread over multiple storage servers. SUMMARY The present disclosure provides systems and methods for sharding objects stored in a distributed storage system. Such sharding may be advantageously utilized for an object that stores a collection of key-value records and for otherwise encoded objects. In accordance with one embodiment disclosed herein, a block sharding technique is used. In accordance with another embodiment disclosed herein, a key sharding technique is used. The object sharding techniques advantageously enable a read-modify-write process at the storage server level. One embodiment disclosed herein provides a method of creating a new chunk by modifying a payload of an existing chunk at the storage server level in a distributed object storage system. A rendezvous transfer is sent from a gateway server to storage servers in a rendezvous group, wherein the rendezvous transfer specifies a change to the payload of the existing chunk. The payload of the existing chunk is read from a storage device by each of the storage servers in the rendezvous group. The payload of the existing chunks modified by applying the specified change to create a new payload by each of the storage servers in the rendezvous group. A cryptographic hash is applied to the new payload to generate a new content hash identifying token by each of the storage servers in the rendezvous group. The new chunk is written to the storage device by each of the storage servers in the rendezvous group, wherein the new chunk includes the new payload and the new content hash identifying token. Finally, the new chunk is validated at the gateway server. Other embodiments, aspects, and features are also disclosed. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 depicts a non-blocking switch. FIG. 2 depicts where the replicast transport layer may fit into the conventional OSI/ISO Seven Layer Model in accordance with an embodiment of the invention. FIG. 3 is a simplified depiction of chunk transmission in an exemplary distributed storage system in accordance with an embodiment of the invention. FIG. 4 depicts congestion on inbound links of a distributed storage system with serial transmission of chunk copies. FIG. 5 depicts a distributed chunk put operation using relayed unicast transmission in a distributed storage system. FIGS. 6-8 depict steps of a client-consensus-based procedure for proposing a chunk put to a distributed storage system in accordance with an embodiment of the invention. FIGS. 9-11 depict steps of a cluster-consensus-based procedure for proposing a chunk put to a distributed storage system in accordance with an embodiment of the invention. FIGS. 12 and 13 depict steps of a client-consensus-based procedure for proposing a chunk put to a distributed storage system with de-duplication in accordance with an embodiment of the invention. FIGS. 14 and 15 depict steps of a cluster-consensus-based procedure for proposing a chunk put to a distributed storage system with de-duplication in accordance with an embodiment of the invention. FIGS. 16 and 17 depict steps of a client-consensus-based procedure for requesting a chunk get from a distributed storage system in accordance with an embodiment of the invention. FIGS. 18 and 19 depict steps of a cluster-consensus-based procedure for requesting a chunk get from a distributed storage system in accordance with an embodiment of the invention. FIGS. 20 and 21 depict steps of a cluster-consensus-based procedure for requesting a chunk get from a distributed storage system where an additional target is requested in accordance with an embodiment of the invention. FIGS. 22 and 23 depict a possible encoding of a generic “Replicast” message over L2 frames. FIG. 24 depicts exemplary encoding structures for chunk put proposal messages in accordance with an embodiment of the invention. FIG. 25 depicts exemplary encoding structures for chunk accept messages in accordance with an embodiment of the invention. FIG. 26 depicts exemplary encoding structures for rendezvous transfer messages in accordance with an embodiment of the invention. FIG. 27 depicts exemplary encoding structures for payload acknowledgement messages in accordance with an embodiment of the invention. FIG. 28 depicts exemplary encoding structures for get request messages in accordance with an embodiment of the invention. FIG. 29 depicts exemplary encoding structures for get response messages in accordance with an embodiment of the invention. FIG. 30 depicts exemplary encoding structures for an error get response message in accordance with an embodiment of the invention. FIG. 31 depicts a simplified example of a computer apparatus which may be configured as a client or a server in the system in accordance with an embodiment of the invention. FIG. 32 depicts a process for weighting servers in an object storage system in accordance with an embodiment of the invention. FIG. 33 depicts a sample assignment of server IDs to rows of a hash table for an object storage system in accordance with an embodiment of the invention. FIG. 34 depicts processing of a chunk hash ID to determine a set of server IDs in an object storage system in accordance with an embodiment of the invention. FIG. 35 depicts an exemplary implementation of KVT-encoding structure for content stored in persistent storage supporting a single storage device in accordance with an embodiment of the invention. FIG. 36 depicts an exemplary method of relocating content encoded with the KVT-encoding structure in accordance with an embodiment of the invention. FIG. 37 depicts an exemplary KVT-encoding structure to implement persistent storage of information about chunks in accordance with an embodiment of the invention. FIG. 38A illustrates an exemplary KVT-encoding structure of a payload in accordance with an embodiment of the invention. FIG. 38B illustrates an exemplary KVT-encoding structure of a back reference in accordance with an embodiment of the invention. FIG. 39 depicts a system that layers a persistent key-value storage provider server above a conventional persistent storage device in accordance with an embodiment of the invention. FIG. 39B illustrates an exemplary architecture for a storage server 3920 which implements persistent storage of key-value tuples in accordance with an embodiment of the invention. FIG. 40 illustrates an implementation of a storage device transaction log as a memory buffer for a sequence of transaction-log entries in accordance with an embodiment of the invention. FIG. 41 illustrates a method of partitioning an object payload having key-value records using sorting to generate “sorted shard” payloads in accordance with an embodiment of the invention. FIG. 42 illustrates a method of partitioning an object payload having key-value records based on offsets to generate “block-shard” payloads in accordance with an embodiment of the invention. FIG. 43 depicts data fields of a “block-shard” chunk reference in accordance with an embodiment of the invention. FIG. 44 illustrates the mapping of assets to a circle of hash values in accordance with an embodiment of the invention. FIG. 45 illustrates the mapping of servers to the same hash circle as FIG. 44 for use in consistent hashing in accordance with an alternate embodiment of the invention. FIG. 46 illustrates the migration of assets to different servers due to the addition of a new server under consistent hashing in accordance with an alternate embodiment of the invention. FIG. 47 illustrates assigning assets to negotiating groups using a “flexible” hashing procedure in accordance with an embodiment of the invention. FIG. 48 illustrates a consistent hashing procedure to determine servers for storing and accessing a payload chunk of a “block shard” in accordance with an alternate embodiment of the invention. FIG. 49 illustrates a flexible hashing procedure to determine a negotiating group for storing and accessing a payload chunk of a “block shard” in accordance with an embodiment of the invention. FIG. 50 is a flow chart of a method (process) of performing a delta edit of a block-sharded object stored in a distributed storage system in accordance with an embodiment of the invention. FIG. 51 is a flow chart of a method (process) of retrieving block-sharded content in accordance with an embodiment of the invention. FIG. 52 depicts a key-value reference chunk in accordance with an embodiment of the invention. FIG. 53 depicts the partitioning of an object payload into key shards in accordance with an embodiment of the invention. FIG. 54 illustrates a flexible hashing procedure to determine a negotiating group for storing and accessing a payload chunk of a “key shard” in accordance with an embodiment of the invention. FIG. 55 depicts a system with putget brokers and agents in accordance with an embodiment of the invention. FIG. 56 is a flow chart of a put transaction to create a new object version based upon a prior version with enumerated edits in accordance with an embodiment of the invention. FIG. 57 is a flow chart of a method to perform a get transaction to collect and obtain key-value records from selected replicas of relevant key shards in accordance with an embodiment of the invention. FIG. 58 is a diagram depicting a technique that allows for the creation of the new chunk at the storage servers in a way that conserves storage network bandwidth in accordance with an embodiment of the invention. DETAILED DESCRIPTION I. Scalable Transport for Multicast Replication As described herein, multicast communications may provide great value in storage networking. There are several instances where the desired distribution of information is from one sender to multiple receivers. These include: 1) Replication of storage content. Creating more than one replica is generally desired to protect against the loss of individual storage servers. 2) Finding the most suitable set of storage servers to accept new content. Storage servers have dynamically varying capacities and work queue depths. Finding the servers that could store the content most quickly allows transactions to complete in the shortest possible time. If point-to-point protocols performed this survey, the polling process could easily take more time than the potential savings in time. 3) Finding the best storage server to process a specific get request has similar issues as finding the most suitable set of storage servers to accept new content. 4) Network replication to additional servers in order to create extra replicas for caching content, distributing content to archival storage and to remote networks comes with only minimal network traffic increase and no increase in server operations for reading the extra copies for these separate purposes. 5) Enabling parallel updating of distributed content (e.g. a Distributed Hash Allocation Table) on all cluster members. While the value of these potential multicast communications is clear, the conventional wisdom has been that multicast cannot be reliable enough for storage, and therefore the overhead of point-to-point protocols must be tolerated in order to provide reliability. Use of multicast in storage clusters has been conventionally limited to the “keep-alive” protocols, which track the set of storage servers currently connected. Conventionally multicast messaging is not used for the storage content itself. The present disclosure provides an effectively reliable transport layer for multicast communications. The methods disclosed herein are optimized for storage networking within a local area network with short and reasonably uniform round-trip times, but they may be layered on top of any transport services that provides an unreliable datagram service and multicast addressing. Congestion Control in Point-to-Point Network Replication Replicating storage over a shared network typically involves large transfers initiated independently by multiple sources. In such cases, some form of congestion control is required to avoid over-subscribing the buffering capacity of the storage network. Designing a core network to be non-blocking is feasible and desirable; however, it is currently implausible to design the target nodes with enough capacity to receive concurrent transfers from every source that might want to transmit. Moreover, even with a non-blocking core, some form of congestion control is required to allocate the bandwidth on the last link to the target servers. In a simple solution, each content source replicates content to each distinct storage server using Unicast point-to-point reliable connections (most frequently TCP/IP). However, this solution can make the link from the content source a bottleneck that limits the speed of sending the new content into the network. According to this solution, if K is the desired replication count (e.g., three), then the source server sends the content K times (each time over a point-to-point connection). The OpenStack Object Service (“Swift”) has this design characteristic. Another Object Storage system, the Hadoop Distributed File System (HDFS), partially addresses this problem by serializing replication of new content. The content source puts to a first storage server, which then replicates that chunk to a second storage server, which then replicates it to a third storage server. If the server performs replication on a cut-through basis, the impact on an isolated put transaction would appear to be minimal (two times the cut-through latency, which might be as small as a single Ethernet frame for each hop). However, such an analysis ignores the problem that the servers that are receiving chunks or objects must also retrieve and deliver content on demand. Most object storage systems operate in a write-once read-many (WORM) fashion. When a storage server takes responsibility for replicating new content the server has the limitation that the replication bandwidth is in contention with retrieval bandwidth. The only way to avoid congestion drops is to delay either the replication traffic or the retrieval traffic. The latency for puts and/or reads (gets) will suffer because of this contention. To optimize replication of content in distributed storage systems, the presently-disclosed solution multicasts the content so as to optimize both the load on the servers and the shaping of the network traffic. While it is desirable to multicast replication traffic, doing so should not be allowed to cause network congestion and resulting dropped packets. Hence, as further disclosed herein, packet drops may be dealt with by retransmission using unicast replication on a point-to-point basis. Reasons why Multicast was not Used in the Past This section describes some reasons why applicants believe that multicast has not been used in the past for replication of stored content. It would be optimum for storage networking to put the content to the network once and have that content reliably arrive at each of N destinations. One challenge to achieving this optimum result is that each chunk is typically encoded as multiple L2 (layer 2) frames. L2 networking protocols, such as Ethernet or InfiniBand™, have individual frames that are small compared to the size of the typical chunk saved in a storage system. For example in a single user Ubuntu workstation with over 584,843 files, the average file size is 240,736 bytes, (totaling ˜140 GB) although the median file size is only 3,472 bytes and the largest file is 2.9 GB. Hence, a typical put (or get) will need to deliver very many L2 frames without any portion of the message being dropped. Modern wired networks have exceedingly low error rates on raw transmission. Therefore, when a message is comprised of hundreds of L2 frames the most common cause of non-delivery of a message is congestion-induced drops. Message drops caused by a transmission error on even one of hundreds of frames are exceptionally rare. With unicast delivery, the transport protocol acknowledges each packet within the overall message (typically only sending the acknowledgement for every other packet). However, typical point-to-point transport protocols negotiate a packet or byte sequence number during connection establishment. It is not feasible to simply multicast a TCP packet to multiple receivers and simply collect the acknowledgements from each target because each target will have selected a different random starting sequence for itself. With TCP, for example, network elements would have to modify the TCP sequence number for each target. Generally, multicast delivery has been limited to unreliable delivery or has relied on negative acknowledgements to allow limited retransmission requests. The challenges of using multicast distribution for reliable delivery of bulk payload has limited deployment of multicast addressing within storage clusters to control plane functions such as node discovery, health checking and negotiating which server would be assigned to store each object. However, conventional unicast protocols have been used to reliably transfer bulk payload. As desirable as sending once and receiving multiple times would be, the conventional wisdom has been that this cannot be achieved with reliable delivery. Splitting the traffic submitted once within the network to multiple destinations is challenging with TCP-like protocols. Either the splitter must act as a full application layer gateway, complete with providing persistent storage for all payload it has acknowledged until the gateway itself has been acknowledged by each target, or it must spoof flow control responses from the splice point such that no packet is originated until there is a window for delivery to each of the targets, and it acknowledges no packet until it has been acknowledged by all targets. Such a gateway would also have to track which targets had acknowledged a given packet and only forward retransmitted packets to those targets that had not already acknowledged it. Re-transmitting an acknowledged packet will cause the destination to conclude that its acknowledgement had not been received, from which it would infer that the network must be congested. Advantageously, the use of multicast addressing is far simpler. At the network layer, multicast protocols are unreliable. Hence, no tracking of per-packet reception is required. Utilizing multicast addressing allows new payloads to enter the switching fabric once, and then deliver them to N destinations. The protocol may then advantageously track the delivery of the entire message rather the tracking the delivery of individual packets. When congestion control properly minimizes the risk of congestion drops, the resulting delivery becomes reliable enough that per packet acknowledgements are no longer required. Hence, in accordance with an embodiment of the present invention, reliable delivery may be achieved using a simpler and more efficient transport protocol. In addition, the utilization of the switching fabrics buffers may be radically reduced, achieving more efficient distribution and more effective utilization of the network. Conventional point-to-point transport protocols rely on per-packet error detection. However, with modern wired networks, applicants believe that protecting data integrity for the entire message is more effective. This is because layer 2 error checking ensures that very few packets have undetected errors, and retransmission of the entire message is acceptable when it is seldom required. As described herein, a congestion control protocol may be designed for environments where dropped packets are extremely rare by avoiding congestion on the edge links to Ethernet end stations. In particular, a congestion control protocol that prevented concurrent bulk transfers to a given egress link would make it safe to transmit the entire chunk with a single ACK/NAK. Retransmission of the entire chunk would be required after an unsuccessful delivery attempt, but this is a cost easily carried if congestion drops have been avoided and dropped frames are extremely rare. The benefits of a simplified protocol, and lesser bandwidth required for acknowledgements themselves, would compensate for the extremely rare retransmission. Combined with the benefits of multicasting, such a congestion control protocol that enables coordination of bulk data transfers in a way that avoids edge link congestion-induced packet drops should generally improve overall network utilization. Note that the ability to avoid congestion by scheduling delivery of messages at a higher layer is dependent on networking layer 2 providing some basic traffic shaping and congestion avoidance on its own. L2 Traffic Shaping Capabilities The presently-disclosed solution utilizes edge-based congestion control for multicast messages. To understand how edge-based congestion control can avoid congestion-based drops of the layer 2 (L2) frames, it is useful to first review the traffic shaping capabilities of advanced Ethernet switches. In relation to such traffic shaping capabilities, the following discusses a non-blocking switch, a non-blocking core, multiple traffic classes, and protected traffic classes. 1) Non-Blocking Switch A switch can be considered to be non-blocking if it is capable of running every one of its links at full capacity without dropping frames, as long as the traffic was distributed such that it did not exceed the capacity on any one of its links. For example, a non-blocking eight-port switch could relay traffic between four pairs of end stations at full wire speed. More usefully, each of the eight ports could be sending 1/7th of the wire speed to each of the other ports. A non-blocking switch has sufficient internal buffering so it can queue the output frames to any one of its ports. The other ports can “share” this output without having to synchronize their transmissions. If they each have a sustained rate of 1/7th of the wire capacity then the output queue for the target port may grow temporarily, but it will not grow indefinitely. There are well-known algorithms to determine the maximum buffering capacity required. A non-blocking switch may offer service level agreements (SLAs) to its end stations that are capable of providing a sustained level of throughput to each of its ports, as long as no egress port is over-subscribed on a sustained basis. Referring now to FIG. 1, the illustrated switch provides a non-blocking switch fabric, such that a flow from X to Y cannot be adversely impacted by any flow from I to J. 2) Non-Blocking Core A non-blocking core is a collection of non-blocking switches that have sufficient bandwidth between the switches such that they can effectively act as though they were simply a large aggregate switch. 3) Multiple Traffic Classes Switches typically offer multiple traffic classes. Frames are queued based upon the egress port, Ethernet class of service, and other factors such as VLAN (virtual local area network) tags. Usually these queues do not represent buffers permanently assigned to separate queues, but rather just a method for accounting for buffer usage. When a queue is assigned N buffers it does not mean that N buffers are identified in advance. Rather it means that the number of buffers the queue is using is tracked, and if it exceeds N the excess buffers are subject to being dropped. Advanced switches are capable of monitoring the depth of queues for multiple traffic classes and potentially taking action based on queue depth (marking excessive traffic, generating congestion notifications, or simply dropped non-compliant frames). The traffic class configuration is typically a steady configuration item for any switch. Well known algorithms allow the switch to enforce that a given traffic class will be able to sustain X Gb/sec without requiring the switch to track the state of each flow through the switch. 4) Protected Traffic Classes A Protected Traffic Class is a traffic class that is reserved for a specific use. The network forwarding elements are configured to know which ports are members of the protected traffic class. L2 frames that are marked as part of a protected traffic class, but arrive from unauthorized ports, are simply dropped. Typically switches will also block, or at least limit, relaying frames in a protected traffic class to non-member ports. FCoE (Fibre Channel Over Ethernet) is one example of a protocol which is dependent on a protected traffic class. The protocol is not robust if non-compliant frames can be accepted from unauthorized ports. Replicast Transport Layer The present section details a “Replicast” transport layer in accordance with an embodiment of the present invention. In one implementation, the Replicast transport layer operates in conjunction with a distributed storage layer for a distributed object storage system. FIG. 2 illustrates the conventional model and where Replicast transport layer and distributed storage layers may be inserted into that model as “Layer 4.5” between the conventional transport layer 4 and the conventional session layer 5. While the present section details a Replicast transport service that is intended for usage by a distributed object storage system, the specific multicast messaging capabilities provided are not constrained to support only distributed object storage. Other applications can benefit from the presently-disclosed Replicast transport services. For example, a method for replicating file system images between file servers could also use the Replicast transport services disclosed herein. One example of this would be ZFS file servers replicating the output of the “zfs send” utility. The following Table 1 itemizes the assignment of responsibilities between the Replicast transport layer disclosed in the present section and an example distributed storage layer that may be supported by the Replicast transport layer: TABLE 1 Division of responsibilities between Replicast transport and distributed storage layers Responsibility Layer Comment Detection of Transport A hash signature is calculated for all transmission transferred content. error. Detection of lost Transport Messages with lost packets will fail the L3 packets signature test for the complete message. Packets are sequenced within a message, allowing for checkerboard testing for reception of the complete message. Detection of lost Storage The transport layer will only detect L5 messages messages that are partially lost, not any that are completely lost. The storage layer must detect missing messages and responses. Determination of Storage The transport layer can detect whether success specific recipients successfully received a message without corruption, but it cannot determine whether the overall message was delivered to a sufficient set of recipients. Retry of L5 Storage The transport layer detects failure of message individual deliveries. The storage layer must determine when and if a given message will be retried. Pacing of Storage The transport layer indicates how often the Unsolicited storage layer may transmit unsolicited Messages messages, but it does not pace those deliveries itself. Rather it relies on the storage layer to choose which messages to submit to comply with the rate published. Congestion Transport The transport layer works with L2 Avoidance congestion avoidance techniques (such as IEEE 802.1 DCB—Data Center Bridging) protocols to provide delivery of unreliable datagrams without dropping packets due to congestion. Note that IEEE 802.1 DCB is only one mechanism for achieving drop-resistant L2 delivery that is protected from other traffic classes. Traffic Selection Storage The primary congestion avoidance (i.e. Shaping technique used is to perform most bulk Storage Traffic content transfer with reserved bandwidth. or Selection The transport layer enforces reservations, of content but the storage layer chooses which when competing reservations to grant by determining for limited when each storage server would be bandwidth) capable of doing a specified rendezvous transfer. The present invention does not specify what algorithm any storage server in the cluster will use when proposing rendezvous transfer times based upon its current workload. There are many well = -known algorithms for making such estimates, determination of which algorithms are most cost effective for which storage resources is left to each specific embodiment. Distributed Storage The storage layer determines when a Deduplication proposed transfer is not needed because the content is already stored locally. The transport layer merely relays this information. Management of Enhanced The L2 layer is responsible for protecting bandwidth L2 (such the traffic class from other traffic. between Traffic as DCB) Presumably, it also protects the other Classes traffic from the storage traffic class. Management of Transport The transport layer is responsible for bandwidth within allocating the bandwidth provided by L2 the storage layer to specific messages and transfers. traffic class Transmit at Transport Datagrams are transmitted once and are most once typically delivered to all members of the target multicast group. Datagrams cannot be delivered more than once because each L2 message is uniquely identified by source and sequence number. Further, each datagram is identified as a specific fragment of an L5 message. Packets that are duplicate receptions are discarded. Datagram Transport Unreliable datagrams are labeled as to sequencing their sequencing within an L5 message. Multicast unreliable Such as UDP/IP/Ethernet or addressing datagram UD/InfiniBand. service Edge Managed Flow Control The present disclosure combines the lack of central bottlenecks with the ability to factor dynamic storage-server specific metrics, such as available storage capacity, work queue depth and network congestion on the storage server's ingress ports. An overly simplified analysis would seek to have every storage server evaluate its own suitability for storing a specific chunk, and then having the source select the number (n) of storage servers with the highest score. However, this would not scale as the total number (N) of storage servers in a cluster increased. As disclosed herein, a scalable methodology, instead, controls the total number of requests made to each storage server. Ideally, as the cluster workload grows, the number of requests per server can be held nearly constant by adding servers and network bandwidth. This will allow the entire cluster to scale in a nearly linear fashion. The present disclosure accomplishes holding nearly constant the number of requests per server by selecting a subset of the storage servers to process requests related to any specific chunk. The present disclosure refers to this subset as the “Negotiating Group”. The Negotiating Group will select specific storage servers from the group to store the specific chunk. Generally, the number of members in a Negotiating Group should be kept stable even as the number of storage servers grows. The complexity of the negotiation process is determined by the number of storage servers in the Negotiating Group, not by the size of the entire cluster. Referring now to Table 2, an exemplary size (n) of the Negotiating Group is that it should scale to on the order of K multiplied by Log10(N) [i.e. should scale to O(K*Log10(N)], where K is a function of the storage replication count, and where N is the total number of cluster members. K may typically vary from one to five. Hence, as shown in Table 2, depending on the value of K, for 100 servers in the cluster, there should be two to ten members of the Negotiating Group, and for 10,000 servers in the cluster, there should be four to twenty members of the Negotiating Group. TABLE 2 Number of Designated Servers in a Negotiating Group for a Cluster Replication Cluster Members K = 1 K = 2 K = 3 K = 4 K = 5 100 2 4 6 8 10 1,000 3 6 9 12 15 10,000 4 8 12 16 20 In an exemplary implementation, the server performing the “put” operation for a chunk will select a set of servers from the Negotiating Group. The selection method is not dependent on a central process or bottleneck and is capable of adapting to storage server backlogs and capacity. In an exemplary selection method, all members of the Negotiating group receive a proposal to store the new chunk (i.e. a Put Proposal) via multicast-addressed UDP datagrams, without adding extra transmission burden on the source server. The source chooses the Negotiating Group by mapping the appropriate Chunk Hash ID to a Distributed Hash Allocation Table so as to specify the membership of the Negotiating Group and identify its members. A Chunk Hash ID may be a cryptographic hash of either a chunk's payload (for chunks that hold only payload) or of the identity of the object (for chunks holding metadata). In an exemplary embodiment, this mapping is accomplished by indexing one row from a shared Distributed Hash Allocation Table. In an exemplary implementation, each chunk may have a unique identifier that effectively incorporates distributed deduplication into the distribution algorithm, making the implementation highly tailored for document storage applications. There are existing techniques that allow distributed deduplication to co-exist with the provision of cryptographic protection for document content. It should be understood that the “Distributed Hash Allocation Table” need not be an actual table fully implemented on every node of the network. It is sufficient that each row of the table maps to a multicast address, and that the network's multicast forwarding be configured so that a multicast message will be delivered to the members of the row. Existing protocols for controlling multicast forwarding can therefore be used to implement the “Distributed Hash Allocation Table” even if they do not consider the tables they manipulate to be anything more than multicast forwarding tables. Referring back to the exemplary selection method, each recipient of the Put Proposal calculates when and if it could accept the chunk, or whether it already has the indicated chunk. The recipient returns a Put Accept message with the appropriate indication to not only the source, but to all other members of the Negotiation Group. Limitations on the recipient's available storage that make this specific storage server less desirable as a target are reflected by making this storage server less prompt in acknowledging the proposal or in scheduling the receipt of the chunk. Additional considerations are possible to indicate that perhaps the recipient has a heavy workload and, if there are other recipients with less workload, that their responses may also be more prompt. While the present disclosure is not necessarily limited to storage servers, in most embodiments, the storage servers will utilize the entire bandwidth available to a single reservation at a time. In accordance with the present disclosure, there is no benefit to delivering part of a chunk. Therefore, it will generally be desirable to finish each request as early as possible, even if it means delaying the start of the transfer of a later request. It is the aggregate completion times for the transfers that matters. By contrast, conventional file systems will generally seek to make forward progress for all transfers in parallel. Upon the collection of the responses from the Negotiating Group within the timeout window, the Chunk Source decides whether to deliver the Chunk payload in order to increase the number of replicas. If so, it creates the Rendezvous Group, which is a subset of the Negotiating Group. In the exemplary implementation, other members of the negotiating group may also see this response and update their list of designated servers that hold a copy of the Chunk. The present disclosure also provides for efficient replication of content to the Rendezvous Group by relying on the rendezvous negotiation to eliminate the need for sustained congestion control for a multicast Chunk Put or conventional point-to-point reliable transport protocols. Exemplary Storage Cluster FIG. 3 is a simplified depiction of chunk transmission in an exemplary storage cluster in accordance with an embodiment of the invention. As shown in FIG. 3, there is a cluster 100 of servers and clients. The servers 102 through 112 are connected to Ethernet switches 115 and 116. While several servers and two switches are depicted, an implemented storage cluster may have a multitude of servers and a larger number of switches. The switches, in turn, may be connected to each other by one or more Ethernet trunk 127. In an exemplary implementation, the switches may be non-blocking switches, and the switches together with the interconnecting trunks may form a non-blocking core. In this depiction, Chunk A 117 is sent from user client 101 to network switch 115 and is multicast replicated to Server 103 as Chunk A 118, Server 106 as Chunk A 119, Server 112 as Chunk A 120. Chunk B 121 is sent from user client 114 to switch 115 and is replicated to Server 102 as Chunk B 122, through trunk 127 to switch 116 and then to Server 107 as Chunk B 123 and to Server 110 as Chunk B 124. At the same time Server 102 is returning Chunk C 125 via Switch 115 to client user 113 as Chunk C 126. Serial Transmission of Chunk Copies FIG. 4 depicts congestion on inbound links of a distributed storage system with serial transmission of chunk copies. Such serial transmission of chunk copies is utilized, for example, in the OpenStack Object Service. The servers, clients, switches and trunks of FIG. 4 may be arranged as described above in relation to FIG. 3. As depicted in FIG. 4, we see that Chunks A1 117, A2 118 and A3 119 (all copies of Chunk A) are transmitted sequentially through the connection between the user client machine 101 and the Ethernet switch 115. From that point, Chunk A1 121 is sent to the target server 106, Chunk A2 120 is sent to a different server 103 and Chunk A3 122 to a third server 112. Similarly the chunks B1 123, B2 124 and B3 125 (all copies of Chunk B) are transmitted from user client 114 to Ethernet switch 115 in sequential fashion, even though they are all copies of the same chunk B. From that point, the B1 Chunk 126, B2 Chunk 127 and B3 Chunk 128 are sent to separate servers. In addition, the C1 Chunk illustrates that, in addition to the PUT activities for Chunks A and B, users are also performing GETs of data. In particular, the C1 Chunk 129 is sent from the server 102 to the switch 115, and then the C1 Chunk 130 is sent to the user client 113. Relayed Unicast Transmission FIG. 5 depicts a distributed chunk put operation using relayed unicast transmission in a distributed storage system. Such a relayed unicast put is utilized, for example, in the HDFS. The servers, clients, switches and trunks of FIG. 5 may be arranged as described above in relation to FIG. 3. In the illustration of FIG. 5, we walk through a sequence of events. The first event is that the user Client 101 transmits the first copy of Chunk A, namely Chunk A1 117, which is received by Server 106 as Chunk A1 118. In preferred implementations, while the Chunk A1 118 is being received, the Server 106 begins a “Cut-through” transmission (illustrated here by the ingress and egress of 118 and 119 overlapping each other) to start transmitting Chunk A2 119 which is a copy of Chunk A1. Other implementations are possible, including waiting until Chunk A1 118 is completely received prior to transmitting Chunk A2 119, but are less optimal. Chunk A2 120 is received by Server 103, copied and retransmitted as Chunk A3 121 (here again illustrated by the ingress and egress of 120 and 121) and finally received by Server 112 as Chunk A3 123. Similarly, the user Client 114 transmits the first copy of Chunk B, namely Chunk B1 124, which is received by Server 107 as Chunk B1 125. In preferred implementations, while the Chunk B1 125 is being received, the Server 107 begins a “Cut-through” transmission (illustrated here by the ingress and egress of 125 and 126 overlapping each other) to start transmitting Chunk B2 126 which is a copy of Chunk B1. Chunk B2 127 is received by Server 102, copied and retransmitted as Chunk B3 128 (here again illustrated by the ingress and egress of 127 and 128) and finally received by Server 110 as Chunk B3 129. In this case, the retransmission of Chunk B3 128 may be delayed by the transmission of an asynchronous “get” operation which requested Chunk C 130. In this way, other operations on the Servers performing “get” operations (to retrieve data upon request) may slow down the replication of packets by the Servers. The C Chunk illustrates that, in addition to the PUT activities for Chunks A and B, users are also performing GETs of data. In particular, the C Chunk 130 is sent from the server 102 to the switch 115, and then the C Chunk 131 is sent to the user client 113. Overview of Replicast Transport Protocol The present disclosure provides a method of supporting effectively reliable message exchange and rendezvous payload transfers within a multicast group or subsets of the multicast group (possibly combined with an external client). An exemplary implementation of the disclosed method may be referred to herein as the “Replicast” transport protocol. The Replicast transport protocol sends unreliable datagrams over a protected traffic class. Protected traffic classes are a known networking mechanism used in many different IEEE 802.1 protocols. One example particularly relevant for storage networking is FCoE (Fibre Channel over Ethernet). The requirements for a protected traffic class may be summarized as follows. L2 frames are admitted to this traffic class only from explicitly authorized end stations. L2 frames are only delivered to members of the group. As long as this traffic class is in compliance with a bandwidth budget provisioned for it, its frames will not be dropped due to congestion caused by L2 frames from other traffic classes. Effective Reliability A goal of the Replicast transport layer (when it is used by a distributed storage application) is to enable effectively reliable transfer of chunks and associated tracking data within a storage cluster and to/from its clients. Distributed storage applications frequently need to make multiple replicas of storage chunks. Enabling an effectively reliable multicast replication may radically improve the efficiency of network utilization and the efficiency of server resources in a cluster. The Replicast transport layer disclosed herein is optimized for networks where actual transmission errors are rare. In such networks, packets are typically dropped due oversubscription of either forwarding or receiving buffers. Distributed storage applications supported by the Replicast transport protocol preferably may be expected to require more thorough validation of successful transfer of data than is supported by conventional point-to-point transport protocols (such as InfiniBand Reliable Connection, TCP/IP or SCTP/IP). To support more thorough validation of successful transfers, the Replicast transport protocol disclosed herein provides hash signatures for the entire chunk and self-validating tracking data which may be used to validate successful transfers. These measures allow incomplete or corrupted transfers to be detected by and reported to the upper layers. For example, a multicast transmission of a chunk may be successfully received by 5 out 7 target nodes. The question of whether that is a “successful” delivery may be properly answered at an upper layer; it is not something the transport layer can or should determine. Congestion Avoidance The present disclosure utilizes an assumption that the lower-layer transports (below the Replicast transport layer) provide at least minimal congestion avoidance features that can deal with short-lived congestion without dropping frames. The IEEE 802.1 Data Center Bridging (DCB) protocols are an example implementation of a suitable lower layer transport. Another goal of the Replicast transport layer disclosed herein is to further avoid congestion, particularly congestion over a sustained duration. The Replicast Transport layer strives to complement the existing lower layer solutions (such as DCB) to short-term congestion with solutions that avoid sustained over-commits of bandwidth. Unsolicited Vs. Solicited Bandwidth The present disclosure seeks to effectively eliminate the risk of a congestion drop by tracking its own usage of unsolicited bandwidth, issuing its own reservations for solicited bandwidth, and relying on the lower transport layers to resolve very short span over-subscriptions and protect the traffic class from traffic from other classes. Network administration will specify four bandwidth allocations for each participant in the protected traffic class: Unsolicited inbound rate: Using known techniques, this translates to a required amount of buffering to receive unsolicited packets. Unsolicited outbound rate: A base rate for transmission of unreliable datagrams that have no reservation. This rate may be adjusted dynamically by other sources of information. One source that must be used is the number of failed deliveries on prior attempts to transmit this datagram. This technique is known as the Aloha back-off algorithm. Reserved outbound rate: This may limit the aggregate bandwidth of all rendezvous transmissions from this storage node. This limit would seldom be reached, so some embodiments may omit this from their implementation. One deployment where it would be useful is when the same node was also originating traffic from a different traffic class. Reserved inbound rate: This node must limit the reservations it grants so that this rate is never exceeded. The disclosed congestion avoidance method may be, therefore: 1) Provision buffering for reception and in-network forward elements adequate for both the Unsolicited and Solicited traffic. Unsolicited traffic is subject to peaks because there is no advance permission granted before a request is transmitted. Therefore, more buffering is needed to support a specific bandwidth when using Unsolicited messaging than would be required for reserved bandwidth. 2) Limiting transmission rates of unsolicited messages so that the probability of packet drop is low. 3) Utilizing Aloha-style random back-offs of retransmissions of Unsolicited messages used for requests. Distributing and Retrieving Chunks The presently-disclosed Replicast transport layer relies on the layer above it, a distributed storage system in one embodiment, to specify the following: 1) A Negotiating Group, which is a multicast group that will conduct the negotiation to determine the Rendezvous Group, and may determine the source for a data transfer in a get transaction. 2) A Rendezvous Group, which is a multicast group that will receive a data transfer. For a put transaction this group will be a subset of the Negotiating Group. For a get transaction this group will include the client or proxy that initiated the get transaction and may include other members of the Negotiating Group that wish to receive opportunistic copies of the chunk that has been requested. 3) A base bandwidth quota for unsolicited traffic that this node may generate to a specified traffic class. This quota is across support for all requests. This quota may be dynamically adjusted by many sources of information as to the congestion state of the network. At the minimum this set must include the number of recent messages sent by this node for which there was no timely response. It may include other sources of network status that are correlated with the congestion state of the network, including: a) Measured queue depths on forwarding elements for queues that support this traffic class. b) Receipt of packets for this traffic class which were explicitly marked to indicate congestion. c) An increase in the one-way delay of packets for this traffic class through the network. d) Reports of congestion from other nodes participating in this traffic class. 4) A bandwidth for this node to receive solicited transfers. The node will not grant reservations that exceed this quota. Messages may be addressed to either Negotiating Groups and/or to Rendezvous Groups. A negotiation is conducted within the Negotiation Group using unreliable datagrams sent with multicast addressing, as described in further detail below, to select a subset of those servers to which the bulk message must be delivered (or replicated at). The purpose of the presently-disclosed transport is to deliver “chunks”, which are large collection of bytes used by the upper layer, to the Rendezvous Group negotiated in the transaction. Additionally, a set of opaque “transaction tokens” may be associated with each chunk and updated in each transfer. Typical uses of “chunks” by a distributed storage layer would include: 5) Large slices of object payload, typically after compression. 6) Metadata for versions of named objects, which will reference the payload chunks to allow the full object to be retrieved. The presently-disclosed transport requires each chunk to have the following naming attributes: A Chunk ID which uniquely identifies the chunk and which will never reference a different payload. In an exemplary implementation, the Chunk ID must be effectively globally unique for at least twice the lifetime that the chunk will be retained or referenced. A Content Hash ID: If the selected hash algorithm is a cryptographic hash with strong immunity from pre-image attacks, such as SHA-2 or SHA-3, then the Content Hash ID may also serve as the Chunk ID. When only used to validate content the hash algorithm merely has to be resistant to coincidental collisions. Whether or not the Content Hash ID is used to identify the chunk, the Content Hash ID is used to validate the content of transmitted chunks or chunks retrieved from persistent storage. In an exemplary implementation, the Chunk ID must have a uniform distribution so that it can efficiently index locally retained chunks on storage servers. In the preferred embodiment, the Chunk ID is always the Content Hash ID. Cryptographic hash algorithms always
{'application_number': '<application-reference appl-type="utility">\n<document-id>\n<country>US</country>\n<doc-number>14821698</doc-number>\n<date>20150808</date>\n</document-id>\n</application-reference>', 'publication_number': '<publication-reference>\n<document-id>\n<country>US</country>\n<doc-number>20160191250</doc-number>\n<kind>A1</kind>\n<date>20160630</date>\n</document-id>\n</publication-reference>', 'source_file': './application/2016/ipa160630.xml', 'source_data': 'USPTO-Application'}