text:
Ben Black always has interesting things on the go. He's now down in San Francisco working on his startup Fastip which he describes as "an incredible platform for operating, exploring, and optimizing data networks." A couple of days ago Deepak Singh sent me to a recent presentation of Ben's I found interesting: Challenges and Trade-offs in Building a Web-scale Real-time Analytics System.
The problem described in this talk was "Collect, index, and query trillions of high dimensionality
records with seconds of latency for ingestion and response." What Ben is doing is collecting per flow networking data with tcp/ip 11-tuples (src_mac, dst_mac, src_IP, dest_IP, …) as the dimension data and, as metrics, he is tracking start usecs, end usecs, packets, octets, and UID. This data is interesting for two reasons: 1) networks are huge, massively shared resources and most companies haven't really a clue on the details of what traffic is clogging it and have only weak tools to understand what traffic is flowing – the data sets are so huge, the only hope is to sample it with solutions like Cisco's NetFlow. The second reason I find this data interesting is closely related: 2) it is simply vast and I love big data problems. Even on small networks, this form of flow tracking produces a monstrous data set very quickly. So, it's an interesting problem in that it's both hard and very useful to solve.
Ben presented 3 possible solutions and why they don't work before offering a solution. The failed approaches that couldn't cope with high dimensionality and the sheer volume of the dataset:
1. HBase: Insert into HBase then retrieve all records in a time range and filter, aggregate, and sort
2. Cassandra: Insert all records into Cassandra partitioned over a large cluster with each dimension indexed independently. Select qualifying records on each dimension, aggregate, and sort.
3. Statistical Cassandra: Run a statistical sample over the data stored in Cassandra in the previous attempt.
The end solution proposed by the presenter is to treat it as an Online Analytic Processing (OLAP) problem. He describes OLAP as "A business intelligence (BI) approach to swiftly answer multi-dimensional analytics queries by structuring the data specifically eliminate expensive processing at query time, even at a cost of enormous storage consumption." OLAP is essentially a mechanism to support fast query over high-dimensional data by pre-computing all interesting aggregations and storing the pre-computed results in a highly compressed form that is often kept memory resident. However, in this case, the data set is far too large to be practical for an in-memory approach.
Ben draws from two papers to implement an OLAP based solution at this scale:
· High-Dimensional OLAP: A Minimal Cubing Approach by Li, Han, and Gonzalez
· Sorting improves word-aligned bitmap indexes by Lemire, Kaser, and Aouiche
The end solution is:
· Insert records into Cassandra.
· Materialize lower-dimensional cuboids using bitsets and then join as needed.
· Perform all query steps directly in the database
Ben's concluding advice:
· Read the literature
· Generic software is 90% wrong at scale, you just don't know which 90%.
· Iterate to discover and be prepared to start over
If you want to read more, the presentation is at: Challenges and Trade-offs in Building a Web-scale Real-time Analytics System.
James Hamilton
e: email@example.com
w: http://www.mvdirona.com
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
Last week, Sudipta Sengupta of Microsoft Research dropped by the Amazon Lake Union campus to give a talk on the flash memory work that he and the team at Microsoft Research have been doing over the past year. Its super interesting work. You may recall Sudipta as one of the co-authors on the VL2 Paper (VL2: A Scalable and Flexible Data Center Network) I mentioned last October.
Sudipta's slides for the flash memory talk are posted at Speeding Up Cloud/Server Applications With Flash Memory and my rough notes follow:
· Technology has been used in client devices for more than a decade
· Server side usage more recent and the difference between hard disk drive and flash characterizes brings some challenges that need to be managed in the on-device Flash Translation Layer (FTL) or in the operating systems or Application layers.
· Server requirements are more aggressive across several dimensions including required random I/O rates and higher reliability and durability (data life) requirements.
· Key flash characteristics:
· 10x more expensive than HDD
· 10x cheaper than RAM
· Multi Level Cell (MLC): ~$1/GB
· Single Level Cell (SLC): ~$3/GB
· Laid out as an linear array of flash blocks where a block is often 128k and a page is 2k
· Unfortunately the unit of erasure is a full block but the unit of read or write is 2k and this makes the write in place technique used in disk drives not workable.
· Block erase is a fairly slow operation at 1500 usec whereas read or write is 10 to 100 usec.
· Wear is an issue with SLC supporting O(100k) erases and MLC O(10k)
· The FTL is responsible for managing the mapping between logical pages and physical pages such that logical pages can be overwritten and hot page wear is spread relatively evenly over the device.
· Roughly 1/3 the power consumption of a commodity disk and 1/6 the power of an enterprise disk
· 100x the ruggedness over disk drives when active
· Research Project: FlashStore
· Use flash memory as a cache between RAM and HDD
· Essentially a flash aware store where they implement a log structured block store (this is essentially what the FTLs do in the device implemented at the application layer.
· Changed pages are written through to flash sequentially and an in-memory index of pages is maintained so that pages can be found quickly on the flash device.
· On failure the index structure can be recovered by reading the flash device
· Recently unused pages are destaged asynchronously to disk
· A key contribution of this work is a very compact form for the index into the flash cache
· Performance results excellent and you can find them in the slides and the papers referenced below
· Research Project: ChunkStash
· A very high performance, high throughput key-value store
· Tested on two production workloads:
· Xbox Live Primetime online gaming
· Storage deduplication
· The storage dedeuplication test is a good one in that dedupe is most effective with a large universe of objects to run deduplication over. But a large universe requires a large index. The most interesting challenge of deduplication is to keep the index size small through aggressive compaction
· The slides include a summary of dedupe works and shows the performance and compression ratios they have achieved with ChunkStash
For those interested in digging deeper, the VLDB and USENIX papers are the right next stops:
· http://research.microsoft.com/apps/pubs/default.aspx?id=141508 (FlashStore paper, VLDB 2010)
· http://research.microsoft.com/apps/pubs/default.aspx?id=131571 (ChunkStash paper, USENIX ATC 2010)
· Slides: http://mvdirona.com/jrh/talksandpapers/flash-amazon-sudipta-sengupta.pdf
NVIDIA has been an ARM licensee for quite some time now. Back in 2008 they announced Tegra, an embedded client processor including an ARM core and NVIDIA graphics aimed at smartphones and mobile handsets. 10 days ago, they announced Project Denver where they are building high-performance ARM-based CPUs, designed to power systems ranging from "personal computers and servers to workstations and supercomputers". This is interesting for a variety of reasons, first they are entering the server CPU market. Second NVIDIA is joining Marvell and Calxeda (previously Smooth-Stone) in taking the ARM architecture and targeting server-side computing.
ARM is an interesting company in that they produce designs and these designs get adapted by licensees including Texas instruments, Samsung, Qualcomm, and even unlikely players such as Microsoft. These licensees may fab their own parts or be fab-less design shops depending upon others for volume production. Unlike Intel, ARM doesn't really produce chips focusing just on design.
ARM has become an incredibly important instruction set architecture powering smartphones, low-end network routers, printers, copiers, tablets, and other embedded applications. But things are changing, arm is now producing designs appropriate for server-side computing at the same time that power consumption is becoming a key measure of server-side computing cost. The ARM design team are masters of low power designs and generations of ARMs have focused on power management. ARM has an impressively efficiently design.
Linux powers many of the ARM-based devices mentioned above and consequently Linux runs well on ARM processors. Completing the picture, Microsoft has announced that the next version of windows will also support ARM devices: Microsoft Announced Support of System on a Chip Architectures from Intel, AMD, and ARM for next Version of Windows.
Other articles are ARM and micro-slice computing:
· CEMS: Low-Cost, Low-power Servers for Internet Scale Services
· A Fast Array of Wimpy Nodes
· Very Low-Cost, Low-Power Servers
· Linux/Apache on ARM Processors
· ARM Cortex-A9 SMP Design Announced
Other articles on the NVIDIA announcement:
· Nvidia Developing ARM Processors for Servers
· Nvidia Turns to ARM for Server Chips to Kill Intel
A closely related article from the Barry Evans and Karl Freund of ARM-powered, server startup Calxeda: http://gigaom.com/cloud/look-out-your-data-center-is-about-to-change-again/.
We are on track for renewed competition in the server-side computing market segment and intense competition on power efficiency at the same time as internet-scale service operators are willing to run whatever processor is least expensive and most power efficient. With competition comes innovation and I see a good year coming.
If you have experience in database core engine development either professionally, on open source, or at university send me your resume. When I joined the DB world 20 years ago, the industry was young and the improvements were coming ridiculously fast. In a single release we improved DB2 TPC-A performance by a factor of 10x. Things were changing quickly industry-wide. These days single-server DBs are respectably good. It's a fairly well understood space. Each year more features are added and a few percent performance improvement may happen but the code bases are monumentally large, many of the development teams are over 1,000 engineers, and things are happening anything but quickly.
If you are an excellent engineer, have done systems or DB work in the past, and are interested in working on the next decade's problems in database, drop me a note (email@example.com).
Megastore is the data engine supporting the Google Application Engine. It's a scalable structured data store providing full ACID semantics within partitions but lower consistency guarantees across partitions.
I wrote up some notes on it back in 2008 Under the Covers of the App Engine Datastore and posted Phil Bernstein's excellent notes from a 2008 SIGMOD talk: Google Megastore. But there has been remarkably little written about this datastore over the intervening couple of years until this year's CIDR conference papers were posted. CIDR 2011 includes Megastore: Providing Scalable, Highly Available Storage for Interactive Services.
My rough notes from the paper:
· Megastore is built upon BigTable
· Bigtable supports fault-tolerant storage within a single datacenter
· Synchronous replication based upon Paxos and optimized for long distance inter-datacenter links
· Partitioned into a vast space of small databases each with its own replicated log
· Each log stored across a Paxos cluster
· Because they are so aggressively partitioned, each Paxos group only has to accept logs for operations on a small partition. However, the design does serialize updates on each partition
· 3 billion writes and 20 billion read transactions per day
· Support for consistency unusual for a NoSQL database but driven by (what I believe to be) the correct belief that inconsistent updates make many applications difficult to write (see I Love Eventual Consistency but …)
· Data Model:
· The data model is declared in a strongly typed schema
· There are potentially many tables per schema
· There are potentially many entities per table
· There are potentially many strongly typed properties per entity
· Repeating properties are allowed
· Tables can be arranged hierarchically where child tables point to root tables
· Megastore tables are either entity group root tables or child tables
· The root table and all child tables are stored in the same entity group
· Secondary indexes are supported
· Local secondary indexes index a specific entity group and are maintained consistently
· Global secondary indexes index across entity groups are asynchronously updates and eventually consistent
· Repeated indexes: supports indexing repeated values (e.g. photo tags)
· Inline indexes provide a way to denormalize data from source entities into a related target entity as a virtual repeated column.
· There are physical storage hints:
· "IN TABLE" directs Megastore to store two tables in the same underlying BigTable
· "SCATTER" attribute prepends a 2 byte hash to each key to cool hot spots on tables with monotonically increasing values like dates (e.g. a history table).
· "STORING" clause on an index supports index-only-access by redundantly storing additional data in an index. This avoids the double access often required of doing a secondary index lookup to find the correct entity and then selecting the correct properties from that entity through a second table access. By pulling values up into the secondary index, the base table doesn't need to be accessed to obtain these properties.
· 3 levels of read consistency:
· Current: Last committed value
· Snapshot: Value as of start of the read transaction
· Inconsistent reads: used for cross entity group reads
· Update path:
· Transaction writes its mutations to the entity groups write-ahead log and then apply the mutations to the data (write ahead logging).
· Write transaction always begins with a current read to determine the next available log position. The commit operation gathers mutations into a log entry, assigns an increasing timestamp, and appends to log which is maintained using paxos.
· Update rates within a entity group are seriously limited by:
· When there is log contention, one wins and the rest fail and must be retried.
· Paxos only accepts a very limited update rate (order 10^2 updates per second).
· Paper reports that "limiting updates within an entity group to a few writes per second per entity group yields insignificant write conflicts"
· Implication: programmers must shard aggressively to get even moderate update rates and consistent update across shards is only supported using two phase commit which is not recommended.
· Cross entity group updates are supported by:
· Two-phase commit with the fragility that it brings
· Queueing and asynchronously applying the changes
· Excellent support for backup and redundancy:
· Synchronous replication to protect against media failure
· Snapshots and incremental log backups
Overall, an excellent paper with lots of detail on a nicely executed storage system. Supporting consistent read and full ACID update semantics is impressive although the limitation of not being able to update an entity group at more than a "few per second" is limiting.
The paper: http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf.
Thanks to Zhu Han, Reto Kramer, and Chris Newcombe for all sending this paper my way.
Years ago I incorrectly believed special purpose hardware was a bad idea. What was a bad idea is high-markup, special purpose devices sold at low volume, through expensive channels. Hardware implementations are often best value measured in work done per dollar and work done per joule. The newest breed of commodity networking parts from Broadcom, Fulcrum, Dune (now Broadcom), and others is a beautiful example of Application Specific Integrated Circuits being the right answer for extremely hot code kernels that change rarely.
I've long been interested in highly parallel systems and in heterogeneous processing. General Purpose Graphics Processors are firmly hitting the mainstream with 17 of the Top 500 now using GPGPUs (Top 500: Chinese Supercomputer Reigns). You can now rent GPGPU clusters from EC2 $2.10/server/hour where each server has dual NVIDIA Tesla M2050 GPUs delivering a TeraFLOP per node. For more on GPGPUs, see HPC in the Cloud with GPGPUs and GPU Clusters in 10 minutes.
Some time ago Zach Hill sent me a paper writing up Radix sort using GPGPUs. The paper shows how to achieve a better than 3x on the NVIDIA GT200-hosted systems. For most of us, sort isn't the most important software kernel we run, but I did find the detail behind the GPGPU-specific optimizations interesting. The paper is at http://www.mvdirona.com/jrh/TalksAndPapers/RadixSortTRv2.pdf and the abstract is below.
This paper presents efficient strategies for sorting large sequences of fixed-length keys (and values) using GPGPU stream processors. Compared to the state-of-the-art, our radix sorting methods exhibit speedup of at least 2x for all generations of NVIDIA GPGPUs, and up to 3.7x for current GT200-based models. Our implementations demonstrate sorting rates of 482 million key-value pairs per second, and 550 million keys per second (32-bit). For this domain of sorting problems, we believe our sorting primitive to be the fastest available for any fully-programmable microarchitecture. These results motivate a different breed of parallel primitives for GPGPU stream architectures that can better exploit the memory and computational resources while maintaining the flexibility of a reusable component. Our sorting performance is derived from a parallel scan stream primitive that has been generalized in two ways: (1) with local interfaces for producer/consumer operations (visiting logic), and (2) with interfaces for performing multiple related, concurrent prefix scans (multi-scan).
As part of this work, we demonstrate a method for encoding multiple compaction problems into a single, composite parallel scan. This technique yields a 2.5x speedup over bitonic sorting networks for small problem instances, i.e., sequences that can be entirely sorted within the shared memory local to a single GPU core.
Even working in Amazon Web Services, I'm finding the frequency of new product announcements and updates a bit dizzying. It's amazing how fast the cloud is taking shape and the feature set is filling out. Utility computing has really been on fire over the last 9 months. I've never seen an entire new industry created and come fully to life this fast. Fun times.
Before joining AWS, I used to say that I had an inside line on what AWS was working upon and what new features were coming in the near future. My trick? I went to AWS customer meetings and just listened. AWS delivers what customers are asking for with such regularity that it's really not all that hard to predict new product features soon to be delivered. This trend continues with today's announcement. Customers have been asking for a Domain Name Service with consistency and, today, AWS is announcing the availability of Route 53, a scalable, highly-redundant and reliable, global DNS service.
The Domain Name System is essentially a global, distributed database that allows various pieces of information to be associated with a domain name. In the most common case, DNS is used to look up the numeric IP address for an domain name. So, for example, I just looked up Amazon.com and found that one of the addresses being used to host Amazon.com is 188.8.131.52. And, when your browser accessed this blog (assuming you came here directly rather than using RSS) it would have looked up perspectives.mvdirona.com to get an IP address. This mapping is stored in an DNS "A" (address) record. Other popular DNS records are CNAME (canonical name), MX (mail exchange), and SPF (Sender Policy Framework). A full list of DNS record types is at: http://en.wikipedia.org/wiki/List_of_DNS_record_types. Route 53 currently supports:
• A (address record)
• AAAA (IPv6 address record)
• CNAME (canonical name record)
• MX (mail exchange record)
• NS (name server record)
• PTR (pointer record)
• SOA (start of authority record)
• SPF (sender policy framework)
• SRV (service locator)
• TXT (text record)
DNS, on the surface, is fairly simple and is easy to understand. What is difficult with DNS is providing absolute rock-solid stability at scales ranging from a request per day on some domains to billions on others. Running DNS rock-solid, low-latency, and highly reliable is hard. And it's just the kind of problem that loves scale. Scale allows more investment in the underlying service and supports a wide, many-datacenter footprint.
The AWS Route 53 Service is hosted in a global network of edge locations including the following 16 facilities:
· United States
• Ashburn, VA
• Dallas/Fort Worth, TX
• Los Angeles, CA
• Miami, FL
• New York, NY
• Newark, NJ
• Palo Alto, CA
• Seattle, WA
• St. Louis, MO
· Europe
• Amsterdam
• Dublin
• Frankfurt
• London
· Asia
• Hong Kong
• Tokyo
• Singapore
Many DNS lookups are resolved in local caches but, when there is a cache miss, it will need to be routed back to the authoritative name server. The right approach to answering these requests with low latency is to route to the nearest datacenter hosting an appropriate DNS server. In Route 53 this is done using anycast. Anycast is a cool routing trick where the same IP address range is advertised to be at many different locations. Using this technique, the same IP address range is advertized as being in each of the world-wide fleet of datacenters. This results in the request being routed to the nearest facility from a network perspective.
Route 53 routes to the nearest datacenter to deliver low-latency, reliable results. This is good but Route 53 is not the only DNS service that is well implemented over a globally distributed fleet of datacenters. What makes Route 53 unique is it's a cloud service. Cloud means the price is advertised rather than negotiated. Cloud means you make an API call rather than talking to a sales representative. Cloud means it's a simple API and you don't need professional services or a customer support contact. And cloud means its running NOW rather than tomorrow morning when the administration team comes in. Offering a rock-solid service is half the battle but it's the cloud aspects of Route 53 that are most interesting.
Route 53 pricing is advertised and available to all:
· Hosted Zones: $1 per hosted zone per month
· Requests: $0.50 per million queries for first billion queries and $0.25 per million queries over 1B month
You can have it running in less time than it took to read this posting. Go to: ROUTE 53 Details. You don't need to talk to anyone, negotiate a volume discount, hire a professional service team, call the customer support group, or wait until tomorrow. Make the API calls to set it up and, on average, 60 seconds later you are fully operating.
I love high-scale systems and, more than anything, I love data from real systems. I've learned over the years that no environment is crueler, less forgiving, or harder to satisfy than real production workloads. Synthetic tests at scale are instructive but nothing catches my attention like data from real, high-scale, production systems. Consequently, I really liked the disk population studies from Google and CMU at FAST2007 (Failure Trends in a Large Disk Population, Disk Failures in the Real World: What does a MTBF of 100,000 hours mean to you). These two papers presented actual results from independent production disk populations of 100,000 each. My quick summary of these 2 papers is basically "all you have learned about disks so far is probably wrong."
Disk failures are often the #1 or #2 failing component in a storage system usually just ahead of memory. Occasionally fan failures lead disk but that isn't the common case. We now have publically available data on disk failures but not much has been published on other component failure rates and even less on the overall storage stack failure rates. Cloud storage systems are multi-tiered, distributed systems involving 100s to even 10s of thousands of servers and huge quantities of software. Modeling the failure rates of discrete components in the stack is difficult but, with the large amount of component failure data available to large fleet operators, it can be done. What's much more difficult to model are correlated failures.
Essentially, there are two challenges encountered when attempting to modeling overall storage system reliability: 1) availability of component failure data and 2) correlated failures. The former is available to very large fleet owners but is often unavailable publically. Two notable exceptions are disk reliability data from the two FAST'07 conference papers mentioned above. Other than these two data points, there is little credible component failure data publically available. Admittedly, component manufacturers do publish MTBF data but these data are often owned by the marketing rather than engineering teams and they range between optimistic and epic works of fiction.
Even with good quality component failure data, modeling storage system failure modes and data durability remains incredibly difficult. What makes this hard is the second issue above: correlated failure. Failures don't always happen alone, many are correlated, and certain types of rare failures can take down the entire fleet or large parts of it. Just about every model assumes failure independence and then works out data durability to many decimal points. It makes for impressive models with long strings of nines but the challenge is the model is only as good as the input. And one of the most important model inputs is the assumption of component failure independence which is violated by every real-world system of any complexity. Basically, these failure models are good at telling you when your design is not good enough but they can never tell you how good your design actually is nor whether it is good enough.
Where the models break down is in modeling rare events and non-independent failures. The best way to understand common correlated failure modes is to study storage systems at scale over longer periods of time. This won't help us understand the impact of very rare events. For example, Two thousand years of history would not helped us model or predict that a airplane would be flown into the World Trade Center. And certainly the odds of it happening again 16 min and 20 seconds later would be close to impossible. Studying historical storage system failure data will not help us understand the potential negative impact of very rare black swan events but it does help greatly in understanding the more common failure modes including correlated or non-independent failures.
Murray Stokely recently sent me Availability in Globally Distributed Storage Systems which is the work of a team from Google and Columbia University. They look at a high scale storage system at Google that includes multiple clusters of Bigtable which is layered over GFS which is implemented as a user–mode application over Linux file system. You might remember Stokely from my Using a post I did back in March titled Using a Market Economy. In this more recent paper, the authors study 10s of Google storage cells each of which is comprised of between 1,000 and 7,000 servers over a 1 year period. The storage cells studied are from multiple datacenters in different regions being used by different projects within Google.
I like the paper because it is full of data on a high-scale production system and it reinforces many key distributed storage system design lessons including:
· Replicating data across multiple datacenters greatly improves availability because it protects against correlated failures.
o Conclusion: Two way redundancy in two different datacenters is considerably more durable than 4 way redundancy in a single datacenter.
· Correlation among node failures dwarfs all other contributions to unavailability in production environments.
· Disk failures can result in permanent data loss but transitory node failures account for the majority of unavailability.
To read more: http://research.google.com/pubs/pub36737.html
The abstract of the paper:
Highly available cloud storage is often implemented with complex, multi-tiered distributed systems built on top of clusters of commodity servers and disk drives. Sophisticated management, load balancing and recovery techniques are needed to achieve high performance and availability amidst an abundance of failure sources that include software, hardware, network connectivity, and power issues. While there is a relative wealth of failure studies of individual components of storage systems, such as disk drives, relatively little has been reported so far on the overall availability behavior of large cloud-based storage services. We characterize the availability properties of cloud storage systems based on an extensive one year study of Google's main storage infrastructure and present statistical models that enable further insight into the impact of multiple design choices, such as data placement and replication strategies. With these models we compare data availability under a variety of system parameters given the real patterns of failures observed in our fleet.
Achieving a PUE of 1.10 is a challenge under any circumstances but the vast majority of facilities that do approach this mark are using air-side economization. Essentially using outside air to cool the facility. Air-side economization brings some complexities such as requiring particulate filters and being less effective in climates that are both hot and humid. Nonetheless, even with the challenges, air-side economization is one of the best techniques, if not the best, of improving datacenter efficiency.
As a heat transport water is both effective and efficient. The challenges of using water in open-circuit datacenter cooling designs are largely social and regulatory. Most of the planet is enshrouded by two fluids, air and water. It's a bit ironic that emitting heat into one of the two is considered uninteresting while emitting heat into the other brings much more scrutiny. Dumping heat into air concerns few and is the cooling technique of choice for nearly all data centers (Google Saint-Ghislain Belgium being a notable exception). However, if the same heat in the same quantities is released in water, it causes considerably more environmental concern even though it's the same amount of heat being released to the environment. Heat pollution is the primary concern.
The obvious technique to avoid some of the impact of thermal pollution is massive dilution. Use seawater or very high volumes of river water such that the local change is immeasurably small. Seawater has been used in industrial and building cooling applications. Seawater brings challenges in that it is incredibly corrosive which drives up build and maintenance costs but it has been used in a wide range of high-scale applications with success. Freshwater cooling relieves some of the corrosion concerns and has been used effectively for many large-scale cooling requirements including nuclear power plants. I've noticed there is often excellent fishing downstream of these facilities so there clearly is substantial environmental impact caused by these thermal emissions but this need not be the case. There exist water cooling techniques with far less environmental impact.
For example, the cities of Zurich and Geneva and the universities of ETH and Cornell use water for some of their heating & cooling requirements. This technique is effective and its impact on the environment can be made arbitrarily small. In a slightly different approach, the city of Toronto employs deep lake water cooling to cool buildings in its downtown core. In this design, the warm water intake is taken in 3.1 miles off shore at a depth of 272'. The city of Toronto avoids any concerns about thermal pollution by using the exhaust water from the cooling system as their utility water intake so the slightly warmer water is not directly released back into the environment.
Given the advantages of water over air in cooling applications and given that the environmental concerns can be mitigated, why not use the technique more broadly in datacenters? One of the prime reasons is that water is not always available. Another is that regulatory concerns bring more scrutiny and, even excellent designs without measurable environmental impact, will still take longer to get approved than a conventional air-cooled approaches. However it can be done and it does produce a very power efficient facility. The DeepGreen datacenter project in Switzerland perhaps the best examples I've seen so far.
Before looking at the innovative mechanical systems used in Deepgreen, the summary statistics look excellent:
· 46MW with 129k sq ft of raised floor (with upgrade to 70MW possible)
· Estimated PUE of 1.1
· Hydro and nuclear sourced power
· 356 W/sq ft average
· 5,200 racks with average rack power of 8.8kW and maximum rack power of 20kW
· Power cost: $0.094/kW hr (compares well across EU).
· 28 mid-voltage 2.5 MW generators with 48 hours of onsite diesel
The 46MW facility is located in the heart of Switzerland on Lake Walensee:
Google Maps: http://maps.google.com/maps?f=q&source=s_q&hl=de&geocode=&q=Werkhof+Bi%C3%A4sche,+Mollis,+Schweiz&sll=37.0625,-95.677068&sspn=45.601981,84.990234&ie=UTF8&hq=&hnear=Werkhof,+8872+Mollis,+Glarus,+Schweiz&ll=47.129016,9.09462&spn=0.307857,0.663986&t=p&z=11
The overall datacenter is well engineered but it is the mechanical systems that are most worthy of note. Following on the diagram below, this facility is cooled using 43F source water from 197' below the surface. The source water is brought in through dual redundant intake pipes to the pumping station with 6 high-capacity pumps in a 2*(N+1) configuration. The pumps move 668,000 gallons per hour at full cooling load.
The fairly clean lake water is run through a heat exchanger (not shown in the diagram below) to cool the closed-system, chilled water loop used in the datacenter. The use of a heat exchanger avoids bringing impurities or life forms into the datacenter chilled water loop. The chilled water loop forms part of a conventional within-the-datacenter cooling system design. The difference is they have completely eliminated process-based cooling (air conditioning) and water towers avoiding both the purchase cost and the power these equipment would have consumed. In the diagrams below you'll see the Deepgreen design followed by a conventional datacenter cooling system for comparison purposes:
Conventional datacenter mechanical system for comparision:
The Deepgreen facility mitigates the impact of thermal pollution through a combination of dilution, low deltaT, and deep water release.
I've been conversing with Andre Oppermann, the CTO of Deepgreen for nearly a year on this project. Early on, I was skeptical they would be able to work through the environmental concerns in any reasonable time frame. I wasn't worried about the design – its well engineered. My concerns were primarily centered around slowdowns in permitting and environmental impact studies. They have done a good job of working through those issues and I really like the resultant design. Thanks to Andre for sending all this detail my way. It's a super interesting project and I'm glad we can now talk about it publically.
If you are interested in a state of the art facility in Switzerland, I recommend you contact Andre, the CTO of Deepgreen, at: firstname.lastname@example.com.
I'm interested in low-cost, low-power servers and have been watching the emerging market for these systems since 2008 when I wrote CEMS: Low-Cost, Low-Power Servers for Internet Scale Services (paper, talk). ZT Systems just announced the R1081e, a new ARM-based server with the following specs:
· STMicroelectronics SPEAr 1310 with dual ARM® Cortex™-A9 cores
· 1 GB of 1333MHz DDR3 ECC memory embedded
· 1 GB of NAND Flash
· Ethernet connectivity
It's a shared infrastructure design where each 1RU module has 8 of the above servers. Each module includes:
· 8 "System on Modules" (SOMs)
· ZT-designed backplane for power and connectivity
· One 80GB SSD per SOM
· IPMI system management
· Two Reatek 4+1 1Gb Ethernet switches on board with external uplinks
· Standard 1U rack mount form factor
· Ubuntu Server OS
· 250W 80+ Bronze Power Supply
Each module is under 80W so a rack with 40 compute modules would only draw 3.2kw for 320 low-power servers for a total of 740 cores/rack. Weaknesses of this approach are: only 2-cores per server, only 1GB/core, and the cores appear to be only 600 Mhz (http://www.32bitmicro.com/manufacturers/65-st/552-spear-1310-dual-cortex-a9-for-embedded-applications-). Four core ARM parts and larger physical memory support are both under development.
Competitors include SeaMicro with an Atom based design (SeaMicro Releases Innovative Intel Atom Server
) and the recently renamed Calxeda (previously Smooth-Stone) has an ARM-based product under development.
Other notes on low-cost, low-powered servers:
· SeaMicro Releases Innovative Intel Atom Server
· When Very Low-Power, Low-Cost Servers don't make snese
· Very Low-Power Server Progress
· The Case for Low-Cost, Low-Power Servers
· 2010 the Year of the Microslice Servers
· Linux/Apache on ARM processors
From Datacenter Knowledge: New ARM-Based Server from ZT systems
Earlier this week Clay Magouyrk sent me a pointer to some very interesting work: A Couple More Nails in the Coffin of the Private Compute Cluster: Benchmarks for the Brand New Cluster GPU Instance on Amazon EC2.
This detailed article has detailed benchmark results from runs on the new Cluster GPU Instance type and leads in with:
During the past few years it has been no secret that EC2 has been best cloud provider for massive scale, but loosely connected scientific computing environments. Thankfully, many workflows we have encountered have performed well within the EC2 boundaries. Specifically, those that take advantage of pleasantly parallel, high-throughput computing workflows. Still, the AWS approach to virtualization and available hardware has made it difficult to run workloads which required high bandwidth or low latency communication within a collection of distinct worker nodes. Many of the AWS machines used CPU technology that, while respectable, was not up to par with the current generation of chip architectures. The result? Certain use cases simply were not a good fit for EC2 and were easily beaten by in-house clusters in benchmarking that we conducted within the course of our research. All of that changed when Amazon released their Cluster Compute offering.
The author goes on to run the Saleable Heterogeneous cOmputing BenChmarking Suite and compare EC2 with Native performance and conclude:
With this new AWS offering, the line between internal hardware and virtualized, cloud-based hardware for high performance computing using GPUs has indeed been blurred.
Finally a run with a Cycle Computing customer workload:
Based on the positive results of our SHOC benchmarking, we approached a Fortune 500 Life Science and a Finance/Insurance clients who develop and use their own GPU-accelerated software, to run their applications on the GPU-enabled Cluster Compute nodes. For both applications, the applications perform a large number of Monte Carlo simulations for given set of initial data, all pleasantly parallel. The results, similar to the SHOC result, were that the EC2 GPU-enabled Cluster Compute nodes performed as well as, or better than, the in-house hardware maintained by our clients.
Even if you only have a second, give the results a scan: http://blog.cyclecomputing.com/2010/11/a-couple-more-nails-in-the-coffin-of-the-private-compute-cluster-gpu-on-cloud.html.
A year and half ago, I did a blog post titled heterogeneous computing using GPGPUs and FPGA. In that note I defined heterogeneous processing as the application of processors with different instruction set architectures (ISA) under direct application programmer control and pointed out that this really isn't all that new a concept. We have had multiple ISAs in systems for years. IBM mainframes had I/O processors (Channel I/O Processors) with a different ISA than the general CPUs , many client systems have dedicated graphics coprocessors, and floating point units used to be independent from the CPU instruction set before that functionality was pulled up onto the chip. The concept isn't new.
What is fairly new is 1) the practicality of implementing high-use software kernels in hardware and 2) the availability of commodity priced parts capable of vector processing. Looking first at moving custom software kernels into hardware, Field Programmable Gate Arrays (FPGA) are now targeted by some specialized high level programming languages. You can now write in a subset of C++ and directly implement commonly used software kernels in hardware. This is still the very early days of this technology but some financial institutions have been experimenting with moving computationally expensive financial calculations into FPGAs to save power and cost. See Platform-Based Electronic System-Level (ESL) Synthesis for more on this.
The second major category of heterogeneous computing is much further evolved and is now beginning to hit the mainstream. Graphics Processing Units (GPUs) essentially are vector processing units in disguise. They originally evolved as graphics accelerators but it turns out a general purpose graphics processor can form the basis of an incredible Single Instruction Multiple Data (SIMD) processing engine. Commodity GPUs have most of the capabilities of the vector units in early supercomputers. What's missing is they have been somewhat difficult to program in that the pace of innovation is high and each model of GPU have differences in architecture and programming models. It's almost impossible to write code that will directly run over a wide variety of different devices. And the large memories in these graphics processors typically are not ECC protected. An occasional pixel or two wrong doesn't really matter in graphical output but you really do need ECC memory for server side computing.
Essentially we have commodity vector processing units that are hard to program and lack ECC. What to do? Add ECC memory and a abstraction layer that hides many of the device-to-device differences. With those two changes, we have amazingly powerful vector units available at commodity prices. One abstraction layer that is getting fairly broad pickup is Compute Unified Device Architecture or CUDA developed by NVIDIA. There are now CUDA runtime support libraries for many programming languages including C, FORTRAN, Python, Java, Ruby, and Perl.
Current generation GPUS are amazingly capable devices. I've covered the speeds and feeds of a couple in past postings: ATI RV770 and the NVIDIA GT200.
Bringing it all together, we have commodity vector units with ECC and an abstraction layer that makes it easier to program them and allows programs to run unchanged as devices are upgraded. Using GPUs to host general compute kernels is generically referred to as General Purpose Computation on Graphics Processing Units. So what is missing at this point? The pay-as-you go economics of cloud computing.
You may recall I was excited last July when Amazon Web Services announced the Cluster Compute Instance: High Performance Computing Hits the Cloud. The EC2 Cluster Compute Instance is capable but lacks a GPU:
· 23GB memory with ECC
· 64GB/s main memory bandwidth
· 2 x Intel Xeon X5570 (quad-core Nehalem)
· 2 x 845GB HDDs
· 10Gbps Ethernet Network Interface
What Amazon Web Services just announced is a new instance type with the same core instance specifications as the cluster compute instance above, the same high-performance network, but with the addition of two NVIDIA Tesla M2050 GPUs in each server. See supercomputing at 1/10th the Cost. Each of these GPGPUs is capable of over 512 GFLOPs and so, with two of these units per server, there is a booming teraFLOP per node.
Each server in the cluster is equipped with a 10Gbps network interface card connected to a constant bisection bandwidth networking fabric. Any node can communicate with any other node at full line rate. It's a thing of beauty and a forerunner of what every network should look like.
There are two full GPUs in each Cluster GPU instance each of which dissipates 225W TDP. This felt high to me when I first saw it but, looking at work done per watt, it's actually incredibly good for workloads amenable to vector processing. The key to the power efficiency is the performance. At over 10x the performance of a quad core x86, the package is both power efficient and cost efficient.
The new cg1.4xlarge EC2 instance type:
· 2 x NVIDIA Tesla M2050 GPUs
· 22GB memory with ECC
With this most recent announcement, AWS now has dual quad core servers each with dual GPGPUs connected by a 10Gbps full-bisection bandwidth network for $2.10 per node hour. That's $2.10 per teraFLOP. Wow.
· The Amazon Cluster GPU Instance type announcement: Announcing Cluster GPU Instances for EC2
· More information on the EC2 Cluster GPU instances: http://aws.amazon.com/ec2/hpc-applications/
The Top 500 Super Computer Sites list was just updated and AWS Compute Cluster is now officially in the top ½ of the list. That means when you line up all the fastest, multi-million dollar, government lab sponsored super computers from #1 through to #500, the AWS Compute Cluster Instance is at #231.
Amazon Web ServicesUnited States
Amazon EC2 Cluster Compute Instances - Amazon EC2 Cluster, Xeon X5570 2.95 Ghz, 10G Ethernet / 2010Self-made
One of the fastest supercomputers in the world for $1.60/node hour. Cloud computing changes the economics in a pretty fundamental way.
More details at: High Performance Computing Hits the Cloud.
Kushagra Vaid presented at Datacenter Power Efficiency at Hotpower '10. The URL to the slides is below and my notes follow. Interesting across the board but most notable for all the detail on the four major Microsoft datacenters:
· Services speeds and feeds:
o Windows LiveID: More than 1B authentications/day
o Exchange Hosted Services: 2 to 4B email messages per day
o MSN: 550M unique visitors monthly
o Hotmail: 400M active users
o Messenger: 320M active users
o Bing: >2B queries per month
· Microsoft Datacenters: 141MW total, 2.25M sq ft
o Quincy:
§ 500k sq ft total
§ 54W/sq ft
o Chicago:
§ 707k sq ft total
§ 85W/sq ft
§ Over $500M invested
§ $8.30/W (assuming $500M cost)
§ 3,400 tons of steel
§ 190 miles of conduit
§ 2,400 tons of copper
§ 26,000 yards of concrete
§ 7.5 miles of chilled water piping
§ 1.5M man hours of labor
§ Two floors of IT equipment:
· Lower floor: medium reliability container bay (standard IOS containers supplied by Dell)
· Upper floor: high reliability traditional colo facility
§ Note the picture of the upper floor shows colo cages suggesting that Microsoft may not be using this facility for their internal workloads.
§ PUE: 1.2 to 1.5
§ 3,000 construction related jobs
o Dublin:
§ 570k sq ft total
§ 47W/sq ft
o San Antonio:
§ 477k sq ft total
§ 57W/sq ft
· The power densities range between 45 and 85W/sq ft range which is incredibly low for relatively new builds. I would have expected something in the 200W/sq ft to 225W/sq ft range and perhaps higher. I suspect the floor space numbers are gross floor space numbers including mechanical, electrical, and office space rather than raised floor.
· Kushagra reports typical build costs range from $10 to $15/W
o based upon the Chicago data point, the Microsoft build costs are around $8/sq ft. Perhaps a bit more in the other facilities in that Chicago isn't fully redundant on the lower floor whereas the generator count at the other facilities suggest they are.
· Gen 4 design
o Modular design but the use of ISO standard containers has been discontinued. However, they modules are prefabricated and delivered by flatbed truck
o No mechanical cooling
§ Very low water consumption
o 30% to 50% lower costs
o PUE of 1.05 to 1.1
· Analysis of AC vs DC power distribution concluding that DC more efficient at low loads and AC at high loads
o Over all the best designs are within 1 to 2% of each other
· Recommends higher datacenter temperatures
· Key observation:
o The best price/performance and power/performance is often found in lower-power processors
· Kushagra found substantial power savings using C6 and showed server idle power can be dropped to 31% of full load power
o Good servers typically run in the 45 to 50% range and poor designs can be as bad as 80%
The slides: http://mvdirona.com/jrh/TalksAndPapers/KushagraVaidMicrosoftDataCenters.pdf
I did a talk earlier this week on the sea change currently taking place in datacenter networks. In Datacenter Networks are in my Way I start with an overview of where the costs are in a high scale datacenter. With that backdrop, we note that networks are fairly low power consumers relative to the total facility consumption and not even close to the dominant cost. Are they actually a problem? The rest of the talk is arguing networks are actually a huge problem across the board including cost and power. Overall, networking gear lags behind the rest of the high-scale infrastructure world, block many key innovations, and actually are both cost and power problems when we look deeper.
The overall talk agenda:
· Datacenter Economics
· Is Net Gear Really the Problem?
· Workload Placement Restrictions
· Hierarchical & Over-Subscribed
· Net Gear: SUV of the Data Center
· Mainframe Business Model
· Manually Configured & Fragile at Scale
· New Architecture for Networking
In a classic network design, there is more bandwidth within a rack and more within an aggregation router than across the core. This is because the network is over-subscribed. Consequently, instances of a workload often needs to be placed topologically near to other instances of the workload, near storage, near app tiers, or on the same subnet. All these placement restrictions make the already over-constrained workload placement problem even more difficult. The result is either the constraints are not met which yields poor workload performance or the constraints are met but overall server utilization is lower due to accepting these constraints. What we want is all points in the datacenter equidistant and no constraints on workload placement.
Continuing on the over-subscription problem mentioned above, data intensive workloads like MapReduce and high performance computing workloads run poorly on oversubscribed networks. Its not at all uncommon for a MapReduce workload to transport the entire data set at least once over the network during job run. The cost of providing a flat, all-points-equidistant network are so high, that most just accept the constraint and other run MapReduce poorly or only run them in narrow parts of the network (accepting workload placement constraints).
Net gear doesn't consume much power relative to total datacenter power consumption – other gear in the data center are, in aggregate much worse. However, network equipment power is absolutely massive today and it is trending up fast. A fully configured Cisco Nexus 7000 requires 8 circuits of 5kw each. Admittedly some of that power is for redundancy but how can 120 ports possibly require as much power provisioned as 4 average sized full racks of servers? Net gear is the SUV of the datacenter.
The network equipment business model is broken. We love the server business model where we have competition at the CPU level, more competition at the server level, and an open source solution for control software. In the networking world, it's a vertically integrated stack and this slows innovation and artificially holds margins high. It's a mainframe business model.
New solutions are now possible with competing merchant silicon from Broadcom, Marvell, and Fulcrum and competing switch designs built on all three. We don't yet have the open source software stack but there are some interesting possibilities on the near term horizon with OpenFlow being perhaps the most interesting enabler. More on the business model and why I'm interested in OpenFlow at: Networking: The Last Bastion of the Mainframe Computing.
Talk slides: http://mvdirona.com/jrh/TalksAndPapers/JamesHamilton_POA20101026_External.pdf
What happens when you really, really, focus on efficient infrastructure and driving down costs while delivering a highly available, high performance service? Well, it works. Costs really do fall and the savings can be passed on to customers.
AWS prices have been falling for years but this is different. Its now possible to offer an small EC2 instance for free. You can now have 750 hours of EC2 usage per month without charge.
The details:
· 750 hours of Amazon EC2 Linux Micro Instance usage (613 MB of memory and 32-bit and 64-bit platform support) – enough hours to run continuously each month*
· 750 hours of an Elastic Load Balancer plus 15 GB data processing*
· 10 GB of Amazon Elastic Block Storage, plus 1 million I/Os, 1 GB of snapshot storage, 10,000 snapshot Get Requests and 1,000 snapshot Put Requests*
· 5 GB of Amazon S3 storage, 20,000 Get Requests, and 2,000 Put Requests*
· 30 GB per of internet data transfer (15 GB of data transfer "in" and 15 GB of data transfer "out" across all services except Amazon CloudFront)*
· 25 Amazon SimpleDB Machine Hours and 1 GB of Storage**
· 100,000 Requests of Amazon Simple Queue Service**
· 100,000 Requests, 100,000 HTTP notifications and 1,000 email notifications for Amazon Simple Notification Service**
More information: http://aws.amazon.com/free/
Long time Amazon Web Services Alum Jeff Barr has written a book on AWS. Jeff's been with AWS since the very early days and he knows the services well. The new book Host Your Web Site in the Cloud: Amazon Web Services Made Easy, covers each of the major AWS services, how to write code against them, with code examples in PHP. It covers S3, EC2, SQS, EC2 Monitoring, Auto Scaling, Elastic Load Balancing, and SimpleDB. The table of contents:
Recommended if you are interested in Cloud Computing and AWS: http://www.amazon.com/Host-Your-Web-Site-Cloud/dp/0980576830.
This morning I came across an article written by Sid Anand, an architect at Netflix that is super interesting. I liked it for two reasons: 1) it talks about the move of substantial portions of a high-scale web site to the cloud, some of how it was done, and why it was done, and 2) its gives best practices on AWS SimpleDB usage.
I love articles about how high scale systems work. Some past postings:
The article starts off by explaining why Netflix decided to move their infrastructure to the cloud:
Circa late 2008, Netflix had a single data center. This single data center raised a few concerns. As a single-point-of-failure, it represented a liability – data center outages meant interruptions to service and negative customer impact. Additionally, with growth in both streaming adoption and subscription levels, Netflix would soon outgrow this data center – we foresaw an imminent need for more power, better cooling, more space, and more hardware.
Our option was to build more data centers. Aside from high upfront costs, this endeavor would likely tie up key engineering resources in data center scale out activities, making them unavailable for new product initiatives. Additionally, we recognized the management of multiple data centers to be a complex task. Building out and managing multiple data centers seemed a risky distraction.
Rather than embarking on this path, we chose a more radical one. We decided to leverage one of the leading IAAS offerings at the time, Amazon Web Services. With multiple, large data centers already in operation and multiple levels of redundancy in various web services (e.g. S3 and SimpleDB), AWS promised better availability and scalability in a relatively short amount of time.
By migrating various network and back-end operations to a 3rd party cloud provider, Netflix chose to focus on its core competency: to deliver movies and TV shows.
I've read considerable speculation over the years on the difficulty of moving to cloud services. Some I agree with – these migrations do take engineering investment – while other reports seem to less well thought through focusing mostly on repeating concerns speculated upon by others. Often, the information content is light.
I know the move to the cloud can be done and is done frequently because, where I work, I'm lucky enough to see it happen every day. But the Netflix example is particularly interesting in that 1) Netflix is a fairly big enterprise with a market capitalization of $7.83B – moving this infrastructure is substantial and represents considerable complexity. It is a great example of what can be done; 2) Netflix is profitable and has no economic need to make the change – they made the decision to avoid distraction and stay focused on the innovation that made the company as successful as it is; and 3) they are willing to contribute their experiences back to the community. Thanks to Sid and Netflix for the later.
For more detail, check out the more detailed document that Sid Anand has posted: Netflix's Transition to High-Availability Storage Systems.
For those SimpleDB readers, here's a set of SimpleDB best practices from Sid's write-up:
· Partial or no SQL support. Loosely-speaking, SimpleDB supports a subset of SQL
o Do GROUP BY and JOIN operations in the application layer
o One way to avoid the need for JOINs is to denormalize multiple Oracle tables into a single logical SimpleDB domain.
· No relations between domains
o Compose relations in the application layer
· No transactions
o Use SimpleDB's Optimistic Concurrency Control API: ConditionalPut and ConditionalDelete
· No Triggers
o Do without
· No PL/SQL
· No schema - This is non-obvious. A query for a misspelled attribute name will not fail with an error
o Implement a schema validator in a common data access layer
· No sequences
o Sequences are often used as primary keys
§ In this case, use a naturally occurring unique key. For example, in a Customer Contacts domain, use the customer's mobile phone number as the item key
§ If no naturally occurring unique key exists, use a UUID
o Sequences are also often used for ordering
§ Use a distributed sequence generator
· No clock operations
· No constraints. Specifically, no uniqueness, no foreign key, no referential & no integrity constraints
o Application can check constraints on read and repair data as a side effect. This is known as read-repair. Read repair should use the CondtionalPut or ConditionalDelete API to make atomic changes
If you are interested high sale web sites or cloud computing in general, this one is worth a read: Netflix's Transition to High-Availability Storage Systems.
Update: Netflix architects Adrian Cockcroft and Sid Anand are both presenting at QconSF between November 1st and 5th in San Francisco: http://qconsf.com/sf2010.
Hosting multiple MySQL engines with MySQL Replication between them is a common design pattern for scaling read-heavy MySQL workloads. As with all scaling techniques, there are workloads for which it works very well but there are also potential issues that need to be understood. In this case, all write traffic is directed to the primary server and, consequently is not scaled which is why this technique works best for workloads heavily skewed towards reads. But, for those fairly common read heavy workloads, the techniques works very well and allows scaling the read workload across over a fleet of MySQL instances. Of course, as with any asynchronous replication scheme, the read replicas are not transactionally updated. So any application running on MySQL read replica's must be tolerant of eventually consistent updates.
Load balancing high read traffic over multiple MySQL instances works very well but this is only one of the possible tools used to scale this type of workload. Another very common technique is to put a scalable caching layer in front of the relational database fleet. By far the most common caching layer used by high-scale services is Memcached.
Another database scaling technique is to simply not use a relational database. For workloads that don't need schema enforcement and complex query, NoSQL databases offer both a cheaper and a simpler approach to hosting the workload. SimpleDB is the AWS hosted NoSQL database with Netflix being one of the best known users (slide deck from Netflix's Adrian Cockcroft: http://www.slideshare.net/adrianco/netflix-oncloudteaser). Cassandra is another common RDBMS alternative in heavy use by many high-scale sites including Facebook where it was originally conceived. Cassandra is also frequently run on AWS with the Cassandra Wiki offering scripts to make it easy install and configure on Amazon EC2.
For those workloads where a relational database is the chosen solution, MySQL read replication is a good technique to have in your scaling tool kit. Last week Amazon announced read replica support for the AWS Relational Database Service. The press release is at: Announcing Read Replicas, Lower High Memory DB Instance Price for Amazon AWS.
You can now create one or more replicas of a given "source" DB Instance and serve incoming read traffic from multiple copies of your data. This new database deployment option enables you to elastically scale out beyond the capacity constraints of a single DB Instance for read-heavy database workloads. You can use Read Replicas in conjunction with Multi-AZ replication for scalable, reliable, and highly available production database deployments.
If you are running MySQL and wish you had someone else to manage it for you, check out Amazon RDS. The combination of read replicas to scale read workloads and Multi-AZ support for multi-data center, high availability make it a pretty interesting way to run MySQL.
A couple of years ago, I did a detailed look at where the costs are in a modern , high-scale data center. The primary motivation behind bringing all the costs together was to understand where the problems are and find those easiest to address. Predictably, when I first brought these numbers together, a few data points just leapt off the page: 1) at scale, servers dominate overall costs, and 2) mechanical system cost and power consumption seems unreasonably high. Both of these areas have proven to be important technology areas to focus upon and there has been considerable industry-wide innovation particularly in cooling efficiency over the last couple of years.
I posted the original model at the Cost of Power in Large-Scale Data Centers. One of the reasons I posted it was to debunk the often repeated phrase "power is the dominate cost in a large-scale data center". Servers dominate with mechanical systems and power distribution close behind. It turns out that power is incredibly important but it's not the utility kWh charge that makes power important. It's the cost of the power distribution equipment required to consume power and the cost of the mechanical systems that take the heat away once the power is consumed. I referred to this as fully burdened power.
Measured this way, power is the second most important cost. Power efficiency is highly leveraged when looking at overall data center costs, it plays an important role in environmental stewardship, and it is one of the areas where substantial gains continue to look quite attainable. As a consequence, this is where I spend a considerable amount of my time – perhaps the majority – but we have to remember that servers still dominate the overall capital cost.
This last point is a frequent source of confusion. When server and other IT equipment capital costs are directly compared with data center capital costs, the data center portion actually is larger. I've frequently heard "how can the facility cost more than the servers in the facility – it just doesn't make sense." I don't know whether or not it makes sense but it actually is not true at this point. I could imagine the infrastructure costs one day eclipsing those of servers as server costs continue to decrease but we're not there yet. The key point to keep in mind is the amortization periods are completely different. Data center amortization periods run from 10 to 15 years while server amortizations are typically in the three year range. Servers are purchased 3 to 5 times during the life of a datacenter so, when amortized properly, they continue to dominate the cost equation.
In the model below, I normalize all costs to a monthly bill by taking consumable like power and billing them monthly by consumption and taking capital expenses like servers, networking or datacenter infrastructure, and amortizing over their useful lifetime using a 5% cost of money and, again, billing monthly. This approach allows us to compare non-comparable costs such as data center infrastructure with servers and networking gear each with different lifetimes. The model includes all costs "below the operating system" but doesn't include software licensing costs mostly because open source is dominant in high scale centers and partly because licensing costs very can vary so widely. Administrative costs are not included for the same reason. At scale, hardware administration, security, and other infrastructure-related people costs disappear into the single digits with the very best services down in the 3% range. Because administrative costs vary so greatly, I don't include them here. On projects with which I've been involved, they are insignificantly small so don't influence my thinking much. I've attached the spreadsheet in source form below so you can add in factors such as these if they are more relevant in your environment.
Late last year I updated the model for two reasons: 1) there has been considerable infrastructure innovation over the last couple of years and costs have changed dramatically during that period and 2) because of the importance of networking gear to the cost model, I factor out networking from overall IT costs. We now have IT costs with servers and storage modeled separately from networking. This helps us understand the impact of networking on overall capital cost and on IT power.
When I redo these data, I keep the facility server count in the 45,000 to 50,000 server range. This makes it an reasonable scale facility –big enough to enjoy the benefits of scale but nowhere close to the biggest data centers. Two years ago, 50,000 servers required a 15MW facility (25MW total load). Today, due to increased infrastructure efficiency and reduced individual server power draw, we can support 46k servers in an 8MW facility (12MW total load). The current rate of innovation in our industry is substantially higher than it has been any time in the past with much of this innovation driven by mega service operators.
Keep in mind, I'm only modeling those techniques well understood and reasonably broadly accepted as good quality data center design practices. Most of the big operators will be operating at efficiency levels far beyond those used here. For example, in this model we're using a Power Usage Efficiency (PUE) of 1.45 but Google, for example, reports PUE across the fleet of under 1.2: Data Center Efficiency Measurements. Again, the spread sheet source is attached below so feel free to change to the PUE used by the model as appropriate.
These are the assumptions used by this year's model:
Using these assumptions we get the following cost structure:
For those of you interested in playing with different assumptions, the spreadsheet source is here: http://mvdirona.com/jrh/TalksAndPapers/PerspectivesDataCenterCostAndPower.xls.
If you choose to use this spreadsheet directly or the data above, please reference the source and include the URL to this pointing.
Disclaimer: The opinions expressed here are my own and do not necessarily represent those of current or past employers.


text:
How Do I Place an Order ?Our Automated Shopping Cart System Functions on Most All of Our Web PagesThe Shopping Cart is In Operation When You See Add to Cart Button' Click on The ' Add to Cart Button' To Buy Items From These Web Pages Adding Your First Item to the Cart Will Show Shipping Cost and Scheduled Ship Date IF You are Adding An Item(s) to the Cart That Does NOT Have A Shopping Cart Button Then Follow The Instructions Given Later in This page Types of PaymentsPayment Choice #1 (Preferred) - Pay by PayPal When Checking Out ONLINEThrough Our Shopping Cart - You Must have a REGISTERED Paypal Account (Credit Card on File)IMPORTANT PAYPAL NOTE: At Present, Paypal DOES NOT Tell Us if you are sending a gift. During CheckOut Check the Box to NOT to Include a Receipt in the Package Payment Choice #2 - Pay By Credit or Check Debit Card ONLINEWhen Checking Out Through Our Shopping Cart - VISA, MC and DISCOVERWe NO LONGER Accept American Express (You can Still Pay with Your AMEX Card Using Paypal) Our Secure Shopping Cart Page Accepts Your Items Ordered, Credit Card and Shipping Info Our Americart© Shopping Cart is Scanned and Tested daily by McAfee Secure© for Safety Once You Have Submitted Your Information During Checkout You Will Receive an Automated eMail Confirmation Payment Choice #3Postal Mail Us A US POSTAL Money OrderITEMS Are RESERVED for 7 Days Awaiting Payment A US Postal Money Order Mailed To Us - NO CHECKS - (Use Your Check Debit Card Instead?) US Postal Money Orders ONLY - NO OTHER Brand of MONEY ORDERS Will Be Accepted Make Sure You Have An eMail Confirmation with the Total Including Shipping Before Mailing Your Postal Money Order (Many One Of A Kind Items Are Available ONLY For Immediate Purchase by Credit Card) A Reservation is A Commitment to Purchase - Failure To Mail In Payment After Reserving Items Will Block Future Purchases Payment Choice #4 - (LEAST Desirable) Credit or Debit Card by TelephoneWe are an ONLINE Company and Have Designed Our Website for Easy Online OrderingOrders are EXPECTED to be Placed ONLINE The BEST Payment Choice Is To Order ONLINE AND Pay ONLINE It Is Best For YOU and For Us To Have The Order Confirmed in Print Plus You Receive An eMailed Copy of When and What You Ordered It's Often VERY Difficult to Get Through By Phone For YOUR Protection Phone Calls to Us That Show Unknown, Out of Area, an 800, 866, 877, 888 Number, No Caller ID, or Have the Caller Information Blocked Will NOT Get Through to Us - 412-885-7091 Have a Question ? Use the Contact Us Link found on Every Page and You Will Get a Prompt Reply Frequently Asked Questions (FAQ)What Options Do I Have During Checkout?1. Add Insurance (Optional - for Package Valued OVER $100)Packages Mailed PRIORITY Mail are Now Insured up to $50.00 by the Post Office at No extra charge However, For Claims the BUYER Must File a claim on the USPS.com Website 2. Have a Printed Receipt Postal mailed to you (Optional, for Gift Orders) 3. Choose NOT to have A Receipt Enclosed in the Package 4. Require a Signature to Accept the Package 5. Apply A Gift Certificate You Received to the Balance In Your Cart Does Your Website Support Secure Ordering ? Yes it Does... When you Click the Checkout Button to Enter Your Credit Card information, Notice that the website in the Address bar at the top of the page will now have an 's' after the http and now reads https:// - this Indicates a secure Site Orders and PaymentMinimum Merchandise Order is $10 - Minimum Shipping/Handling Charge is $3.50IF Your Total Merchandise Order is UNDER $10, then a $5.00 'Small Order Fee' is Added to the Cart (If you order additional merchandise, the $5.00 'Small Order Fee' is Removed) We Accept Credit Cards MasterCard, VISA, DISCOVER, Debit and Check Cards and PayPal Payment By Postal Mail - NO CHECKS - US Postal Money Orders ONLY NO Other Brands of Money Orders (Why Not Use Your Bank Check Card?) An eMail is Sent Immediately When You Click Process Order or Submit During Checkout, Often a Second eMail Confirming Stock and Ship Date, and an Additional eMail with the Postal Tracking Number(s) When Your Shipping Label is Printed Didn't Get even ONE eMail from us After Ordering ? Send us a Question Using Our Contact Us Link Found on the Left Side of the Every Page on the Floating Menu We Recommend Using a Reliable FREE eMail GMAIL account. Sign Up at google.com Order Status You can Check Your Order Shipping Status by Entering the Postal Tracking Number(s) at http://www.usps.com We eMailed you the Postal Tracking Number(s) when Your Label was Printed Typically the package is scanned at your nearest large Postal Facility or at the time of delivery to you. Delivery times vary from 2-10 business days depending on your proximity to Pittsburgh, PA and on the workload of USPS (Holidays, Natural Disasters, etc) Tracking information is updated throughout the day, but most data is transmitted in the evening. For example, if your Priority Express Mail® package is mailed at 10 am, you might not be able to view tracking info until evening or early next day. Adding An Item that Does NOT Have a Add to Cart ButtonUse The Form Found at the Bottom of the Web Page that do Not Have an Add to Cart ButtonAdd An Item That Do NOT Have An Add To Your Shopping Cart Button Any Instructions for Items Entered Here, Special Instructions if Needed Can Be Noted Later At Time of Checkout If You Are Paying by PayPal Wait For A Total From Us via eMail Enter The ITEM#: Enter Description: Enter ITEM Cost: Enter Quantity: Delivery and Shipping TimesOur Ship Date is Shown When You Add Your FIRST Item to the Shopping CartRemember Time Zone Differences When Ordering, We are Located in the Eastern Time Zone If you order during the Late Evening Hours Your Package Can Sometimes be Shipped the Next Day (This is Determined by When The Post Offices Picks Up Our Packages which is Typically before 9am EDT) US Post Office Estimated Delivery Times for Inside the US Are: First Class 2-7 Days Priority Mail 2-3 Days Priority Express Mail 1-2 Days Ship Postal Priority ExpressNeed it Faster?We Can Ship US Priority Express (Most Places Delivered Overnight - 2 Days Max) Check Delivery times Shipping From Pittsburgh ZIP 15227 here: http://postcalc.usps.gov/ We Still Ship the Date Shown in the Cart when the Order is Placed, but Express Mail is Faster Shipping Priority Express Mail is an Option You Choose in the Cart You can Choose to REQUIRE A Signature to Accept the Package, An Option During Checkout Check the Express Tracking Number we eMailed to you for Post Office Details Enter the Tracking Number here http://www.usps.com The Above Shipping Options are Available to you During Shopping Cart Checkout Pennsylvania Sales Tax7% Sales Tax (State and Allegheny County Pittsburgh) is Added to the Total Including ShippingONLY IF We Ship to a Pennsylvania Address - Food and Clothing Are Not Yet Taxed In PA Store HoursMay I Visit Your Store When I'm in Pittsburgh? - NO, We Do NOT Have a StorefrontPurchases are Made Only Through Online Web Sales 24 Hours a Day - NO Local Pickup Have a Question? - How to Contact Us?Click the Contact Us Button on the Left Hand side Floating Menu of Each Web page(VERY Difficult to get Through by Phone) I Have Collectibles to Sell - How Do I Do That ?We Sometimes Buy PITTSBURGH Sports Collectibles and Rolling Rock Beer CollectiblesThey MUST be Pittsburgh Sports Collectibles We Do Not Deal in Other Teams or Players from other Teams NO Sports cards - We Don't Deal in Sports Cards ROLLING ROCK - If you Have Quantities of NEW Tee Shirts, Sweat Shirts, Caps, Koozies, Tap Handles or Other LICENSED and NEW Latrobe Brewery Rolling Rock Beer Merchandise For Sale, We Would Like to Hear From You NO Cans, NO Reproductions or Items NOT Authorized by Rolling Rock or Anheuser Busch NO Breakables or Hard to Ship Items Such as: Mirrors, Neon Signs, Chalk Statues, Thermometers, Guitars, Bicycles etc CLICK On the Contact Us Button on the Side Menu on the Left of the Page Tell Us Your Zip Code, Then List the Items, Condition, Quantities and Price you are asking See What we Currently Buying Here We Are Buying What is My Item Worth?We Gladly Answer Questions About Your Order and About the Items WE Have for SaleDue to The High Volume of eMail that we Receive, We NO Longer Offer Opinions or Appraisals of Your Items There are Too many variables involved to determine an accurate value without personally examining the item You can get an idea of the value of your item by comparing it to similar items on our pages We NO Longer Answer Questions Such as What is My Item Worth?, What Can You Tell Me About My Item?, Can you get me.... , I'm looking for...?, Do you think you'll be getting.....? Backorders - About Our StockALL of Our Sports Products Penguins, Pirates, Steelers and Rolling Rock Are LICENSEDby the Appropriate Leagues and Licensed Holders IF The Item Is Listed On Our Web Page We Should Have It In Stock We do NOT Backorder Items Your Card is NOT Charged For Out of Stock Items (We will eMail you In The Unlikely Event An Item Listed on our webpage Out of Stock) Gift CertificatesGift Certificates are good for ONE Year can be Redeemed During Shopping Cart Check OutAutograph and Memorabilia CertificatesSince Authenticity is a Matter of Individual Opinion, We Offer a Money Back Guarantee.IF the Autographed Item Sold by us is Certified by PSA/DNA, GAI or ACE Authentication Services then It may be returned Within 30 Days from the Date of Purchase for a Refund. IF the Autographed Item is Certified by a Letter from Chuck Thomas Enterprises and is Disputed by Either PSA/DNA or ACE Authentication Services it Can be Returned for a Refund Within 60 Days from the date of purchase. These Disputed Autographed Items Must Include The Original Purchase Receipt AND the Original Certificate of Authenticity from Chuck Thomas Enterprises BOTH Must Be Returned Along With the Item in Question, with a copy of the Letter Disputing the Item. The Authenticity Must be Disputed in Writing by either PSA/DNA or ACE Authentication Services Personalized Certificates of Authenticity Are Not Transferable. Returns and ExchangesInclude a Note in Your Return Package with your 12 Digit order number. Indicate Exchange or Refund.If Exchanging check our website for available sizes in stock, Notify us right away Using the Contact Us Link with the ITEM# plus size and color if applicable, so we can hold that item for you for 7 days. You can Also apply the value of an exchange for another item on our Website. Your Credit Card will be Charged for any difference Including charges for reshipping Items. The Original Shipping Charges are Not Refundable. Items Must Be Returned or Exchanged Within 30 Days Mail Returns and Exchanges to: Chuck Thomas Enterprises 101 E Willock Rd Pittsburgh PA 15227 WE Pay Your Return Shipping Cost To Us When The Mistake is Ours! Otherwise You are Credited The Item Total Less 10% Restocking Fee Less Shipping Lost or Damaged PackagesWhat if I Don't Receive My Order ?The Good News is that Most All Packages We Ship Make it to their Destination Without Any Issues Occasionally Packages Get Lost or are Delivered to an Incorrect Address If this Happens, eMail Us Using the Contact Link so we Are Aware of the Situation We Will Investigate Where your Package May Be and Let You know What Happened Damage Claims Should be Reported Immediately KEEP All Damaged Items Received With Your Order Lost and Damage Insurance Claims MUST Be Reported to Us within 30 Days of Purchase Date Postal Regulations Require a 30 day Waiting Period Before a Claim is Filed for Lost Packages A Signature to Receive a Package is Your Choice (Unless the item is a Higher Value and we require it) You have the Option of Requiring Signature Confirmation for Delivery During Checkout The Post Office has recently implemented an Insurance Plan - When you choose PRIORITY Mail (NOT First Class) the Post Office will cover insurance for up to $50.00 If the package is lost or damaged the BUYER Can file a claim on the USPS.com Website Ordering From Outside The United States and APO and AE AddressesAPO and AE Destinations are OK - THANK YOU For ServingWe Require a Minimum of $50.00 in Merchandise for Some Countries Insurance On Some Overseas Shipments IS REQUIRED - IF An Insurance Claim has to be Filed, We may have the Postal Authorities Pay the Claim Directly to You Full Value is Declared to Customs - (Do Not Ask Us to Mark the Package as a 'Gift') Some Items are Too Large or Fragile To Ship Outside The USA We Will Tell You If Your Order Includes One of These Items How Much Does it Cost to Ship Outside the US ? Add the Item to the Cart. Choose Your Country from the Drop Down Menu Choose USPS First Class Int'l (International) or USPS Priority Mail Int'l (International) (You Can change this later) The Cart will Show you the Ship cost (You are Under NO Obligation to buy at this Point) How Long Does it take for my order to arrive ? Shipping Times can be found at the US Postal Web Site http://usps.com Enter Zip 15227 (Pittsburgh PA) as the Ship Zip Code and Your Postal Code as the Destination If Your Package does Not arrive within the suggested Guidelines, it's Best to Check with your Local Post Office. Customs Might be Holding Your Package Where Don't We Ship ?We Will NOT Ship to the Following US ZIP Codes: 10036, 10017, 94102 and 94108Some of The Overseas Locations We will NOT Ship to are: Afghanistan, Algeria, Angola, Bangladesh, Barbados, Benin, Bolivia, Bosnia, Brazil, Bulgaria, Burkina Faso, Burma, Burundi, Cambodia, Cameroon, Cape Verde, Central African Republic, Chad, China, Congo, Cuba, Djibouti, Dominican Republic, Ecuador, Ethiopia, Gabon, Ghana, Guinea, India, Indonesia, Iran, Iraq, Italy, Jordan, Kenya, Liberia, Libya, Malaysia, Mali, Mauritania, Morocco, Monaco, Myanmar, Niger, Nigeria, North and South Korea, Oman, Pakistan, Papua/New Guinea, Poland, Romania, Rwanda, Senegal, Seychelles, Sierra Leone, Slovenia, Solomon Islands, Somalia, Sudan, Syria, Tanzania, Togo, Tunisia, Uganda, Venezuela, Yemen, and Yugoslavia (US Service Personnel APO and AE Addresses are OK - THANK YOU! for Serving) If Your Live in One of These Countries, and Want to Buy From Us, Have A Friend or Relative in the US Order From Us and then they can Ship it to You Waiting For Your Package To ArriveYour Black and White Mailing Label is ALWAYS Computer Printed, NEVER Hand PrintedSo You Know What to Safely Expect, Our Return Address Will Read: FROM: Chuck Thomas Enterprises www.chuckthomas.com PO Box 98189 Pgh PA 15227 How Long Does it take for my package to Arrive? Check the http://usps.com website Shipping from Pittsburgh PA 15227 to your ZIP Code Do You Give Out My eMail Address?NO We Do NOT - See Our Complete Privacy PolicyWe Occasionally Offer Discount Promotions by eMail |
Copyright © 1997-2015 Chuck Thomas Enterprises All Rights Reserved

subset: commoncrawl


text:
Nicole Kidman exclusively chats with Richard Wilkins about her role in Baz Luhrmann's epic, Australia.
Kidman reveals that her faith in Luhrmann's creative ability is so strong that she signed on for the part as Lady Sarah Ashley without a script.
"I'll do anything for him – he knows that," she says.
Australia has already attracted Oscar buzz, but Kidman is simply excited to be promoting her home country.
"To have people go to the cinema and laugh and cry and really enjoy this and feel proud that this film was made here - that would be great."
While her onscreen love interest and off-screen friend, Hugh Jackman, hasn't been coy in discussing the kiss that has everybody talking, Kidman remains tight lipped.
"I will not be discussing the intricacies of the kiss," she asserts.
It's been four years in the making, but Australia finally premiered in Sydney last week.
The film has the potential to save Australia's flagging tourism and filmmaking industries. Tourism Australia has conducted a $50 million promotion campaign linked to the film, aiming to make Australia a coveted destination as the global financial crisis hits tourism, just as "Crocodile Dundee" did in the 1980s.
Australia will open nationwide on Wednesday, November 26.


text:
One of the other topics that my friendly archvillain Jason J Kee and I touched on via twitter on Friday was the term of copyright.
I remembered that when he was in front of the C-32 committee he claimed that the format shifting aspects of C-32 (Now C-11) didn't apply to video games. He started to make similar claims in our discussion, started by the claim that "by definition, software is never platform neutral", and later that there is "no reasonable consumer expectation to format shift games".
If you only consider the few months after a video game is released, when the most money is currently made by the game developer, Jason's suggestions my appear reasonable. Games tend to push the limits of the hardware they are designed for, and thus are tied to that hardware for the time when those limits still apply.
Unfortunately, the government granted monopoly of copyright last far more than a few months or a few years.


text:
wanting to try these new cucs in our NH garden. they have a short prolific growing season. experimenting with the right seed for a shorter growing season. thanks for your help if you can |
Follow-Up Postings:
HI eileen_shorty2002, Welcome to The Seed Exchange! You will likely get more offers to trade if you will update your exchange list with what you have available for trading hth |
Here is a link that might be useful: sticky: please read before posting! **required**
Please Note: Only registered members are able to post messages to this forum. If you are a member, please log in. If you aren't yet a member, join now!
Please Note: Only registered members from the U.S. are able to post messages here (this may be indicated by the title of the forum. All exchanges not indicated otherwise are restricted to those living in the U.S.) If you are a member from an area mentioned above, please log in.
Return to the Seed Exchange Forum
Information about Posting
- You must be logged in to post a message. Once you are logged in, a posting window will appear at the bottom of the messages. If you are not a member, please register for an account.
- Please review our Rules of Play before posting.
- Posting is a two-step process. Once you have composed your message, you will be taken to the preview page. You will then have a chance to review your post, make changes and upload photos.
- After posting your message, you may need to refresh the forum page in order to see it.
- Before posting copyrighted material, please read about Copyright and Fair Use.
- We have a strict no-advertising policy!
- If you would like to practice posting or uploading photos, please visit our Test forum.
- If you need assistance, please Contact Us and we will be happy to help.
Learn more about in-text links on this page here


text:
Lamar Jack, an Anderson University basketball player, passed away shortly before 11am today at AnMed Hospital in Anderson. The 19-year old from Piedmont, South Carolina was a redshirt freshman forward on the AU basketball team. He collapsed during routine pre-season workouts on the Anderson campus Friday morning and had been hospitalized since that time. He was the son of Karen and Patrick Jack of Piedmont. The following is the statement released by President Evans P. Whitaker to the Anderson campus this morning.
Dear Campus Family:
It is with great sadness that I share with you that our beloved student, Lamar Jack, passed away just a short time ago.
Our hearts ache for Lamar's parents and sister, and his larger family and circle of close friends.
He was a fine young man, appreciated and loved by many in our campus family, especially his teammates on the Men's Basketball Team. His death is a great and deeply hurtful loss to our campus.
Lamar knew our Lord and Savior and we are encouraged by the confidence that we will one day see him again.
I want to thank all of you who have wept and prayed for Lamar over the last few days. His family has been aware of your concern and they are very grateful. Your kindness and compassion have made a meaningful difference.
Please continue to lift the family and the basketball team members up to our Lord, asking for His tender mercies upon them in this difficult hour and their challenging journey ahead.
When we know more about plans, we will share that information with you. At the appropriate time, we will call the campus together to honor Lamar's life and his contributions to our campus.
Evans P. Whitaker, Ph.D.


text:
Mount Kimbie LIVE, XXYYXX debut SF @ 1015 Folsom 11/9
British electronic duo, Mount Kimbie will grace SF as one of their few stops on their 2012 tour. Minimalist and sometime ephemeral, Mount Kimbie experiment with embellishing tones from field recordings to make captivating rhythms. Sometimes labeled as post-dubstep or experimental, but more often described as completely unique. Mount Kimbie can be ambient, wobbling and wonky, or hip-hop. But almost always their music invokes emotion. Check them out LIVE.
XXYYXX is someone to note as an up and coming 16-yr old beat producer from Florida. Also a genre bending pioneer, reviewers attempt descriptions with chill-wave futurist, psychedelic with a hip-hop back beat, galactic soundscape, etc. What apparently can be expected, however, is a gorgeously expressive dance beat constantly pushing the listener forward on an underwater freeway of time.
Note: Headlining set at this venue usually plays from 1-3am or 2-4am.
Mount Kimbie LIVE
Friday Nov 9th, Doors at 10pm
21+ $17 tickets
with: XXYYXX (debut SF performance)
Ital Tek
Search for 1015 Folsom albums on AmazonPosted in San Francisco
Related Mount Kimbie Content:
High Fidelity
Mount Kimbie – Cold Spring Fault Less Youth
Pitchfork Offline SXSW 2011 Day Party and Official Showcase
St Vincent, Alt-J and Warpaint Contribute Artwork for Radio Station Fundraiser
WATCH: Mount Kimbie Release New Video for Song "You Took Your Time" Featuring Krule
Related San Francisco Content:
Sea of Dreams Announce Lineup for New Years Eve 2013 Extravaganza
Dean Wareham @ The Chapel 11/12
Related XXYYXX Content:
III Points Announces First-Ever III Points Festival and Reveals 2013 Lineup Featuring James Murphy, Jamie xx, Jacques Greene, and More
Hopscotch Festival Announces 2013 Lineup Featuring Big Boi, Spiritualized and The Breeders
St. Jerome's Laneway Festival Coming to America SXSW 2013 Day Party Announced
#FEED Powered by Twitter SXSW Interactive 2013 Parties Announced
House of Vans Boston Night Two ft. Earl Sweatshirt, Cities Aviv and XXYYXX
Related giraffage Content:
Binary Records' SXSW 2014 Launch Party Announced
Related experimental electronic Content:
LISTEN: Psapp Release Stream of New Song "Everything Belongs To The Sun"


text:
Japan is known as the land of the samurai, sushi, walkman, Nintendo, and Honda just to name a few. It's also known for creating some of the strangest...Continue reading »
Category Archives: Video Games
Eager PlayStation gamers across the UK could finally get their share of Sony's latest PlayStation 4 (PS4) console, following the official UK midnight...Continue reading »
The Baldur's Gate 2: Enhanced Edition has been quite a wait, but good news, folks! It's available to purchase as of today! What can you find...Continue reading »
Payday 2 has been met with a lot of success, and it's receiving a new DLC pack called Armored Transport. The pack comes with two new heists, focusing...Continue reading »
During Blizzard Entertainment's annual Blizzcon event, the developer announced what we've all been waiting for — Hearthstone's...Continue reading »
Blizzcon is underway, and things are getting exciting, as Blizzard just confirmed versions of its collectible card game, Hearthstone, for two mobile platforms,...Continue reading »
We haven't heard a whole lot about Diablo 3's new expansion, Reaper of Souls, however, a new trailer has hit the web detailing some of the...Continue reading »
In case you haven't had enough of Total War: Rome 2 yet, Creative Assembly just announced a version specifically for Valve's Steam Machines. Creative...Continue reading »
Company of Heroes 2 will be getting some much-needed DLC, as the Victory at Stalingrad premium DLC is set to launch next week, Sega announced. As for...Continue reading »
CD Projekt RED is well known for hit games like Baldur's Gate, and more recently, The Withcer, Witcher 2, and the upcoming Witcher 3. Oh, and let's...Continue reading »


text:
Enrique Iglesias Tickets
Singer, son of legend Julio Iglesias and all-round heart throb, Enrique Iglesias is one of the best selling Latin recording artists of our age. With multiple chart-topping albums selling millions of copies all over the world, Enrique is a genuine phenomenon. Tickets for Enrique's live shows are incredibly popular, as his concerts are notoriously passionate affairs. Get your Enrique Iglesias tickets now and get ready to be swept off your feet!
Enrique Iglesias
GuaranteedAll tickets covered by the viagogo guarantee
Your Money Is SafeWe don't pay the seller until you've been to the event
Secure DeliveryDelivered via e-ticket or courier
International Events
Enrique Iglesias Tour Dates and Tickets
Buy or sell your Enrique Iglesias tour tickets at viagogo. You can also sign up for Ticket Alerts above to stay informed about new and upcoming Enrique Iglesias 2015 tour dates. If you have any questions about using viagogo, visit our Frequently Asked Questions page.
viagogo is your online ticket marketplace for concert tickets, sports tickets and theatre tickets.


text:
Although most of us these days think of Labor Day Weekend as something of a time for one last outdoor party to celebrate the end of another summer, it is of course really a holiday that was decreed a federal holiday in 1894 to celebrate the economic and social contributions of workers and to reward them with an extra day off.
Although the real meaning of the final holiday of summer has got a little lost over the years, you can still incorporate the theme in your celebrations. One great way to do that is try out one of these great recipes that were created with hardworking Americans in mind:
Seattle Fire Station 25 Meatloaf
A firefighter's choice because it tastes as great cold as it does hot (as often a firehouse meal is interrupted for hours when duty calls) this meatloaf is robust and tasty without being too heavy for summertime eating.
1 pound bacon, sliced into1 inch pieces
2 onions, chopped
5 lbs ground beef
5 lbs spicy ground pork sausage
3 (10 ounce) packages chopped spinach , thawed, drained and squeezed dry
3 (4 ounce) cans sliced black olives, drained
3 cups pistachio nuts, chopped
1 cup ketchup
3 eggs
1 packet meatloaf seasoning mix
1 lb pepper jack cheese, cubed
Hows it's Done
Preheat the oven to 350F. Heat an oiled skillet over a medium flame, add bacon pieces and onions and cook until the bacon is crispy and the onions are translucent. Drain grease and set aside to cool.
In a mixing bowl combine ground beef, pork sausage, bacon and onions, spinach, olives, pistachio nuts, 1 cup of ketchup, eggs, and meatloaf seasoning and stir until well combined. Add mixture to a meatloaf tin and place cheese cubes by poking them into the top of the meatloaf and then re-covering them. Bake for 90 minutes of until the internal temperature of the meatloaf is 180 degrees F.
For many grillmasters, the Labor Day weekend is their last chance to don their barbecue chef's apron and show off their stuff before the grill is packed away for the season.
This marinated chicken dish was a favorite with factory workers at the turn of the last century as it is tasty and filling but uses inexpensive, easy to find ingredients, and, like the meatloaf is still delicious when eaten cold.
2 pounds chicken thighs
1 (16 ounce) bottle zesty Italian dressing
1 (16 ounce) jar salsa
How it's Done
To marinate the chicken thighs arrange them in a glass dish. Pour dressing and salsa over them, remembering to turn them all to properly coat. Cover the dish with aluminum foil and refrigerate for a minimum of three hours (overnight is best though.)
When you are ready to grill, remove chicken piece by piece, baste again with any remaining marinade, and grill for 15 minutes or until the juices run clear.
Minnesota Mail Carrier's Cookies
These originally cookies were created as gifts for hardworking mailmen and women, but they also make the perfect sweet treat at a Labor Day barbecue.
1 cup shortening
2 cups packed brown sugar
1 teaspoon baking soda
2 tablespoons warm water
3 eggs
1 cup flaked coconut
1 cup rolled oats
1 cup chopped dates
1/2 cup chopped walnuts
2 1/2 cups all-purpose flour
How it's Done
Preheat the oven to 400F. Dissolve the baking soda in the warm water. In a large bowl C-cream the shortening with the brown sugar until the mixture is light in color. Stir in the eggs, baking soda mixture, coconut, oatmeal, dates or raisins, nuts and flour.
Drop mixture onto a greased baking sheet in teaspoon sized dollops and then bake for ten minutes.
Unique Personalized Gifts for Life's Special Occasions. Shop Epic Personalized Gifts when an ordinary gift just won't do!