text:
Exactly one year ago, Jim Gray guided his sailboat Tenacious out of San Francisco's Gashouse Cove Marina into the Bay. He sailed under the Golden Gate Bridge and continued towards the Faralon Islands, some 27 miles off the California coast line. Until that morning, I chatted with Jim via email or phone several days a week. He has reviewed everything I've written of substance for many years. When I consider job changes, I'll always bounce them first off him first. When I come across something particularly interesting, I'll always send it Jim's way. Every month or so, he'll send me an interesting pre-published paper. If a conference deadline like CIDR or HPTS is approaching, he'll start pushing me to write something up and keep pushing until it happens. Every four to six months, he'll decide "I just have to meet" someone with overlapping interests, someone who's work is particularly interesting, or perhaps they are just super-clear thinkers and worth getting to know.
What's truly remarkable is that tens and perhaps hundreds of people can say exactly the same thing. He has time for everyone and everyone has similar stories of mentorship, advice, detailed explanation, patience, and insightful reviews. Jim's magic is that he does this for a huge cross-section of our industry. He knows no bounds and always manages to find the time to help without regard for who's asking.
Jim is still missing. Over the past year I've received no papers from Jim. There's been no advice. I've not had anything reviewed by him. And, I've not been able to talk to him about projects I've been working on. When I attend a conference, I don't get the usual surprise visits from Jim. It's been exactly one year and we know no more today than we did a year ago. Jim remains missing. We all miss him deeply.
James Hamilton, Windows Live Platform Services Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052 W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | email@example.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
If you are interested in boating, the Seattle Boat Show opened yesterday. Jennifer and I will be presenting on the red stage at 4:15 on Saturday February 2nd. Our presentation will be some of our favorite anchorages and cruising areas selected from Cruising the Secret Coast: Unexplored Anchorages on British Columbia's Inside Passage which just went to the printer a couple of weeks back. Drop by if you want to talk about boating (or high-scale distributed systems).
In Designing and Deploying Internet Scale Services (http://research.microsoft.com/~jamesrh/TalksAndPapers/JamesRH_Lisa.pdf) I've argued that all services should expect to be overloaded and all services should expect to have manage mass failures.
Degraded operations mode is a means of dealing with excess load that will happen at some point in the life of your service. Sooner or later, you'll get an unpredicted number of new customers, more concurrent users, or you'll have part of the server fleet down and get hit with unexpected load. Sooner or later you'll have more customer requests than you have resources to satisfy. When this occurs, many services just run slower and slower and eventually start failing in timeouts. Basically every user in the system gets a very bad experience. A more serious example is a login storm. For most services, steady state service is much less resource intensive than user login. So, in the event of global or broad service failure, millions of users will arrive back at once attempting to login. The service will fail again under the load and the cycle repeats. It's not a good place to be. A more drastic approach to avoid this problem is admission control. Only allow users into the service where you have resources left to be able to serve them. Essentially give a few customers a bad experience by not letting them onto the service in order to avoid giving all customers a bad experience.
There is much that can be done between the first options, service failure under high load, and the other end of the spectrum, admission control. I call this middle ground, degraded operations mode. In the limit all services need to have admission control to avoid complete and repeating service failure under extreme loads but you hope that admission control is never used. Degraded operations mode allows a service to continue to take on new load after it reaches capacity by shedding unnecessary tasks. Most services have batch jobs that run tasks that need to be done but there isn't actually a customer waiting on them. For example reporting, backup, index creation, system maintenance, copying data to warehouse servers, etc. In most services a substantial amount of this work can be deferred without negatively impacting the service. Clearly these operations need to be run eventually and how long each can be delayed is task and service specific. Temporarily shedding these batch jobs allows more customers to be served. The next level of degraded operations mode is to restrict the quality of service in some way. If some operations are far more expensive, you may only allow users to access a subset of the full service functionality. For example, if you may allow transactions but not reporting if that makes sense for your service. Finding these degraded modes of operation is difficult and very application specific but they are always there and its always worth finding them. There WILL be a time when you have more users than resources.
15 years ago I worked on an Ada language compiler and one of the target hardware platforms for this compiler was a Navy fire control system. This embedded system had a large red switch tagged as "Battle Ready Mode". This switch would disable all automatic shutdowns and put the server into a mode where it would continue to run when the room was on fire or water is beginning to rise up the base of the computer. In this mode, it runs until it dies. In the services world, this isn't exactly what we're after but it's closely related. We want all system to be able to drop back to a degraded operation mode that will allow them to continue to provide at least a subset of service even when under extreme load or suffering from cascading sub-system failures. We need to design and, most important, we need to test these degraded modes of operation in at least limited production or they won't work when we really need them. Unfortunately, all services but the very least successful will need these degraded operations modes at least once.
Degraded operation modes are service specific and, for many services, the initial developer gut reaction is that everything is mission critical and there exist no meaningful degraded modes for their specific service. But, they are always there if you take it seriously and look hard. The first level is to stop all batch processing and periodic jobs. That's an easy one and almost all services have some batch jobs that are not time critical. Run them later. That one is fairly easy but most are hard to come up with. It's hard to produce a lower quality customer experience that is still useful but I've yet to find an example where none were available. As an example, consider Exchange Hosted Services (an email anti-malware and archiving service). In that service, the mail must get delivered. What is the degraded operation mode? They actually can be found there as well. Here's some examples: turn up the aggressiveness of email edge blocks, defer processing of mail classified as Spam until later, process mail from users of the service ahead of non-known users, prioritize platinum customers ahead of others. There actually are quite a few options. The important point is to think what they are and ensure they are developed and tested prior to the operations team needing them in the middle of the night.
A few months back Skype had a problem recently where the entire service went down or mostly down for more than a day. What they report happened was that Windows Update forced many reboots and it lead to a flood of Skype login requests "that when combined with lack of peer to peer resources had a critical impact" (http://heartbeat.skype.com/2007/08/what_happened_on_august_16.html). There are at least two interesting factors here, one generic to all services and one Skype specific. Generically, it's very common for login operations to be MUCH more expensive than steady state operation so all services need to engineer for login storms after service interruption. The WinLive Messenger team has given this considerable thought and has considerable experience with this issue. They know there needs to be an easy way to throttle login requests such that you can control the rate with which they are accepted (a fine grained admission control for login). All services need this or something like this but it's surprising how few have actually implemented this protection and tested it to ensure it works in production. The Skype specific situation is not widely documented put hinted at by the "lack of peer-to-peer" resources note in the above referenced quote. In Skype's implementation, the lack of an available supernode will cause client to report login failure (http://www1.cs.columbia.edu/~salman/publications/skype1_4.pdf sent to me by Sharma Kunapalli). This means that nodes can't login unless they can find a supernode. This has a nasty side effect in that the fewer clients that can successfully login, the more likely it is that other clients won't successfully find a supernode. If they can't find a supernode, they won't be able to login either. Basically, the entire network can become unstable due to the dependence on finding a supernode to successfully log a client into the network. For Skype, a great "degraded operation" mode would be to allow login even when a supernode can't be found. Let the client get on and perhaps establish peer connectivity later.
Why wait for failure and the next post-mortem to design in and production test degraded operations for your services? Make it part of your next release.
A couple of weeks back I attended the Berkeley RAD Lab Retreat. At this retreat, the RAD Lab grad students present their projects and, as is typical of Berkeley retreats, the talks we're quite good. It was held up at Lake Tahoe which was great for the skiers but also made for an interesting drive up there. Chains were required for the drive from Reno to Lake Tahoe and I was in a rental car with less than great summer tires and, of course, no chains.
It snowed hard for much of the retreat. When leaving I took a picture of a pickup truck completely buried in the parking lot:
The talks included: Scalable Consistent Document Store, Prototype of the Instrumentation Backplane, Response time modeling for power-aware resource allocation, Using Machine Learning to Predict Performance of Parallel DB Systems, Diagnosing Performance Problems from Trace data using probabilistic models, Xtrace to find Flaws in UC Berkeley Wireless LAN, Exposing Network Service Failures with Datapath Traces, Owning Your Own Inbox: Attacks on Spam Filters, Declarative Distributed Debugging (D3), Policy Aware Switching Layer, Tracing Hadoop, Machine-Learning-Enabled Router to Deal with Local-Area Congestion, A Declarative API for Secure Network Applications, Deterministic Replay on multi-processor systems, and RubyOnRails.berkeley.edu.
Basically the list of talks presented came pretty close to what I would list as the most interesting challenges in services and service design. Great stuff. In addition to the talks, there are always an interesting group of folks from Industry and this year was no exception. I had a good conversation over dinner with Luiz Barroso (http://research.google.com/pubs/author77.html) and brief chat with John Ousterhout (http://www.electric-cloud.com/).
The flight back was more than a bit interesting as well. We left Reno heading towards Seattle in a small prop plane. Thirty minutes into the trip, I was starting to wonder what was wrong in that I could see the aircraft landing gear doors opening and closing repeatedly from my wing side seat. Shortly thereafter the pilot announced that we had a gear problem and we needed to return to Reno. We returned and did a low pass over the Reno airport so that the tower could check the landing gear position via binoculars. Then we circled back and landed with a fire trucking chasing us down the runway. We stayed out on the active taxi ways with the airport closed to incoming or outgoing traffic while crew came out to the aircraft and pinned the gear in the down position before moving the plane to the terminal.
Dave Dewitt and Michael Stonebraker posted an article worth reading yesterday titled: MapReduce: A Major Step Backwards (Thanks to Kevin Merrit and Sriram Krishnan for sending this one my way). Their general argument is that MapReduce isn't better than current generation RDBMS which is certainly true in many dimensions and it isn't a new invention which is also true. I'm not in agreement with the conclusion that MapReduce is a major step backwards but I'm fully in agreement with many of the points building towards that conclusion. Let's look at some of the major points made by the article:
1. MapReduce is a step backwards in database access
In this section, the authors argue that schema is good, separation of schema and application are good, and high level language access is good. On the first two points, I agree schema is good and there is no question that application/schema separation has long ago proven to be a good thing. The thing to keep in mind is that MapReduce is only an execution framework. The data store is GFS or sometimes Bigtable in the case of Google or HDFS or HBase in the case of Hadoop. MapReduce is only the execution framework so it's not 100% correct to argue that MapReduce doesn't support schema – that's a store issue and it is true that most stores that MapReduce is run over don't implement these features today.
I argue that a separation of execution framework from store and indexing technology is a good thing in that MapReduce can be run over many stores. You can use MapReduce over either BigTable (which happens to be implemented on GFS) or over GFS depending upon the type of data you have at hand. I think that Dewitt and Stonebraker would both agree that breaking up monolithic database management systems into extensible components is a very good thing to do. In fact much of the early work in extensible database management systems was done by David Dewitt. The point here is that Dewitt and Stonebraker would like to see schema enforcement as part of the store and, generally, I agree that this would be useful. However, MapReduce is not a store.
They also argue that high level languages are good. I agree and any language can be used with MapReduce systems so this isn't a problem and is supported today.
2. MapReduce is a poor implementation
The argument here is that any reasonable structured store will support indexes. I agree for many workloads you absolutely must have indexes. However, for many data mining and analysis algorithms, all the data in a data set is accessed. Indexes, in these cases, don't help. This is one of the reason why many data mining algorithms run poorly over RDBMS – if all they are going to do is repeatedly scan the same data, a flat file is faster. It depends upon application access pattern and the amount of data that is accessed. A common execution approach for data mining algorithms is to export the data to a flat file and then operate on it there. An index helps when you are looking at a small subset of the data and there is point N where if you are looking at less than N% of the data, the index helps and should be used. But, if looking at more than N%, you are better off table scanning. The point N is implementation dependent but storage technology trends have been pushing this number down over the years. Basically some algorithms look at all the data and aren't helped by indexes and some look at only a portion of the data and for those that look at more than N% of the data, the index again won't help.
There is no question that indexes are a good thing and there is no arguing that much of the worlds persistent storage access is done through indexes. Indexes are good. But, they are not good for all workloads and for all access patterns. Remember MapReduce is not a store – only an execution framework. To implement indexing in a store used by MapReduce would be easy and presumably someone will when it's need is broadly noticed. In the interim, indexes can be built using MapReduce jobs and then used by subsequent MapReduce jobs. Certainly more of a hassle than stores that automatically maintain indexes but acceptable for some workloads.
3. MapReduce is not novel
This is clearly true. These ideas have been fully and deeply investigated by the database community in the distant past. What is innovative is scale. I've seen MapReduce clusters of 3,000 nodes and I strongly suspect that clusters of 5,000+ servers can be found if you look in the right places. I've been around parallel database management systems for many years but have never seen multi-thousand node clusters of Oracle RAC or IBM DB2 Parallel Edition. The innovative part of MapReduce is that it REALLY scales and, for where MapReduce is used today, scale matters more than everything else. I'll claim that 3,000 server query engines ARE novel but I agree that the constituent technologies have been around for some time.
4. MapReduce is missing features
All of the missing features (bulk loader, indexing, updates, transactions, RI, views) are features that could be implemented in a store used by MapReduce. As these features become important in domains over which MapReduce is used, they can be implemented in the underlying stores. I suspect, as long as MapReduce is used for analysis and data mining workloads the pressing need for RI may never get strong enough to motivate someone to implement it. However, it clearly could be done and the absence of RI in many stores is not a shortcoming of MapReduce.
5. MapReduce is incompatible with the DBMS tools
I 100% agree. Tools are useful and today many of these tools target RDBMS. It's not mentioned by the authors but another useful characteristic of RDBMS is developers understand them and many people know how to write SQL. It's an data access and manipulation language that is broadly understood. The thing to keep in mind is that MapReduce is part of a componentized system. It's just the execution framework. I could easily write a SQL compiler that emitted MapReduce jobs (SQL doesn't dictate or fundamentally restrict the execution engine design). MapReduce can be run over simple stores as it mostly is today or over stores with near database level functionality if needed.
I'm arguing that the languages with which MapReduce jobs are expressed could be higher level and there have been research projects to do this (for example: http://research.microsoft.com/research/sv/dryad/). Even a SQL Compiler is possible over MapReduce. And I'm arguing that MapReduce could be run over very rich stores with indexes and integrity constraints should that become broadly interesting. MapReduce is just an execution engine that happens to scale extremely well. For example, in the MapReduce-like system used around Microsoft, there exist layers of languages above the execution engine that offer different levels of abstraction and control on the same engine.
An execution engine that runs on multi-thousand node clusters really is an important step forward. The separation of execution engine and storage engine into extensible parts isn't innovative but it is a very flexible approach that current generation commercial RDBMS could profit from.
I love MapReduce because I love high scale data manipulation. What can be frustrating for database folks is 1) most of the ideas of MapReduce have been around for years and 2) there has been decades of good research in the DB community focusing on execution engine techniques and algorithms that haven't yet been applied to the MapReduce engines. Many of these optimizations from the DB world will help make better MapReduce engines. But, for all these faults, MapReduce sure does scale and it's hard not to love being able to submit a job and see several thousand nodes churning over several petabytes of data. Priceless.
The article below is a restricted version of what I view to be the next big thing. If I was doing a start-up today, it would be data analysis and optimization as a service. The ability to run real time optimization over an understandable programming platform for a multi-thousand node cluster is very valuable to almost every industry. The smaller their margins, the harder it is for them to afford to not to use yield management and optimization systems. The airlines showed us years ago that yield management systems are a big part of what it takes to optimize the profitability of an airline. Walmart did the same thing to brick and mortar retail. Amazon applied the same for online commerce. The financial community has been doing this for years. The online advertising business is driven by data analysis. What's different as a service is that the technology can be made available to the remaining 99% of the business world. Those that don't have the scale to afford doing their own.
Storing blobs in the sky is fine but pretty reproducible by any competitor. Storing structured data as well as blobs is considerably more interesting but what has even more lasting business value is the storing data in the cloud AND providing a programming platform for multi-thousand node data analysis. Almost every reasonable business on the planet has a complex set of dimensions that need to be optimized. The example below is a good one but it's only one. Almost every business can be made more profitable by employing some form of yield management. And, it's ideally set up for a service based solution: the output data is small, the algorithms to be run are small, the data that forms the input is HUGE but it accumulates slowly over times. Historically large in aggregate but not that large per unit time. The input data per unit time is of manageable size and could be sent to the service. The service provides petabytes of storage and thousands of processors and a platform to run data analysis and optimization programs. This service would also be great for ISVs with companies springing up all over to write interesting algorithms that can be rented and run in the infrastructure.
From: Jeff CarnahanSubject: Fluidity better than perfection?
Good reading for a Saturday morning…
The math wizards at Dayjet are building a smarter air taxi--and it could change the way you do business.
From: Issue 115 | May 2007 | Page 100 | By: Greg Lindsay | Photographs By: Jill Greenberg and Courtesy DayJet
It's only fitting that a service pitched to traveling salesmen should find itself confronting an especially nasty version of what's known as the "traveling-salesman problem." Stated simply: Given a salesman and a certain number of cities, what's the shortest possible path he should take before returning home? It's a classic conundrum of resource allocation that rears its ugly head in industries ranging from logistics (especially trucking) to circuit design to, yes, flesh-and-blood traveling salesmen: How do you minimize the cost and maximize your efficiency of movement?
Solving for X: Once the FAA clears the way for the Eclipse 500, Iacobucci will get to see how good his models really are.
Back in 2002, that was the question facing DayJet, a new air-taxi service hoping to take off this spring. Based in Delray Beach, Florida, DayJet will fly planes, but its business model isn't built around its growing fleet of spanking-new Eclipse 500 light jets. It's built on math and silicon, and the near-prophetic powers that have in turn emerged from them. "We're a software and logistics company that only happens to make money flying planes," insists Ed Iacobucci, an IBM (NYSE:IBM) veteran and cofounder of Citrix Systems (NASDAQ:CTXS), who started DayJet as his third act.
The advent of affordable air taxis has been heralded by a steady drumbeat of press over the past few years, with an understandable fixation on the sexy new technology that's generally credited with making the market possible: the planes. The Eclipse 500 is a clean-sheet design for a tiny jet that seats up to six and costs about $1.5 million (the Federal Aviation Administration may clear it for mass production as early as next month). It is also the most fuel-efficient certified jet in the sky. Cessna, meanwhile, has rolled out its own, if pricier, "very light jet" (VLJ), with Honda's (NYSE:HMC) set to appear in 2010. No less an authority than The Innovator's Dilemma author and Harvard Business School professor Clayton Christensen has mused in print that the E500 and its ilk "could radically change the airline industry" by disrupting the hub-and-spoke system we all know and despise.
But Iacobucci, who wrote a check long ago for more than 300 orders and options on Eclipse's first planes, isn't relying on the aircraft to make or break him. Instead, it's his company's software platform--and the novel way it attacks the traveling-salesman problem--that will set DayJet apart. On day one of operations, flying from just five cities in Florida with only 12 planes, DayJet's dispatchers will already have millions of interlocking flight plans to choose from. As the company's geographic footprint spreads (with luck) across the Southeast--and as its fleet expands as well--the computational challenge only gets worse. Factor in such variables as pilot availability, plane maintenance schedules, and the downpours that drench the peninsula like clockwork in the summer, and well, you get the idea: Finding the shortest, fastest, and least-expensive combination of routes could take every computer in the universe until the end of time.
"I knew what the complexities were and how the problem degenerates once you reach a threshold," Iacobucci says. So he didn't try to find the optimal solution. Instead, DayJet began looking for a family of options that create positive (if imperfect) results--following a discipline known as "complexity science."
For the past five years, with no planes, pilots, or customers, DayJet has been running every aspect of its business thousands of times a day, every day, in silicon. Feeding in whatever data they could find, Iacobucci and his colleagues were determined to see how the business would actually someday behave. When DayJet finally starts flying, they'll switch to real-time flight data, using their operating system to shuttle planes back and forth the way computers shuttle around bits and bytes.
Iacobucci is an expert at building operating systems--he did it for decades at IBM and Citrix. Because of that, he has zero interest in the loosey-goosey world of Web 2.0. He sees the next great opportunities in business as a series of operating systems designed to model activities in the real world. DayJet looks to be the first, but he has no doubt there will be others, and that new companies, and even new industries, will appear overnight as computers tease answers out of previously intractable problems.
Which brings us back to the traveling salesmen. Iacobucci says his computer models predict that DayJet's true competitors are not the airlines, but Bimmers and Benzes--he says 80% of his revenue will come from business travelers who would otherwise drive. In other words, DayJet, which closed an additional $50 million round of financing in March, is creating a market where none exists, an astonishing mathematical feat. To get there, all Iacobucci needed was five years, a professor with a bank of 16 parallel processors, two so-called Ant Farmers, and a pair of "Russian rocket scientists" who, it turns out, are neither Russian nor rocket scientists.
"This is way nastier than any of the other airline-scheduling work we've ever done," says Georgia Tech professor George Nemhauser, whose PhD students have been helping to map the scope of DayJet's mountain-sized scheduling dilemma. "You can think of this as a traveling-salesman problem with a million cities, and that's a problem DayJet has to solve every day."
Tapping into the school's computing power, Nemhauser and his students have figured out how to calculate a near-perfect solution for 20 planes in a few seconds' worth of computing time and a solution for 300 planes in 30 hours. But as impressive as that is, in the real world, it's not nearly enough. That's because in order for DayJet's reservations system to succeed, Iacobucci and company need an answer and a price in less than five seconds, the limit for anyone conditioned to Orbitz or Expedia (NASDAQ:EXPE). Because DayJet has no preset schedule--and because overbooking is out of the question (DayJet will fly two pilots and three passengers maximum)--any request to add another customer to a given day's equation requires its software to crunch the entire thing again.
One of Iacobucci's oldest pals and investors, former Microsoft (NASDAQ:MSFT) CFO and Nasdaq chairman Mike Brown, pointed him toward a shortcut--a way to cheat on the math. Brown had retired with his stock options to pursue his pet projects in then bleeding-edge topics such as pattern recognition, artificial intelligence, nonlinear optimization, and computational modeling. His dabblings led him first to Wall Street, where he invested in a trading algorithm named FATKAT and eventually to Santa Fe, New Mexico, ground zero for complexity science.
Iacobucci says 80% of his revenues will come from travelers who would otherwise drive. DayJet, in other words, is creating a market where none existed, an astonishing mathematical feat.
Invented by scientists at the nearby Los Alamos National Laboratory in the 1980s, complexity science is a gumbo of insights drawn from fields as diverse as biology, physics, and economics. At its core is the belief that any seemingly complex and utterly random system or phenomenon--from natural selection to the stock market--emerges from the simple behavior of thousands or millions of individuals. Using computer algorithms to stand in for those individual "agents," scientists discovered they could build fantastically powerful and detailed models of these systems if only they could nail down the right set of rules.
When Brown arrived in town in the late 1990s, many of the scientists-in-residence at the Santa Fe Institute--the serene think tank dedicated to the contemplation of complexity--were rushing to commercialize their favorite research topics. The Prediction Co. was profitably gaming Wall Street by spotting and exploiting small pockets of predictability in capital flows. An outfit called Complexica was working on a simulator that could basically model the entire insurance industry, acting as a giant virtual brain to foresee the implications of any disaster. And the BiosGroup was perfecting agent-based models that today would fall under the heading of "artificial life."
By the time Iacobucci mentioned his logistical dilemma to Brown in 2002, however, most of Santa Fe's Info Mesa startups were bobbing in the dotcom wreckage. But Brown knew that Bios had produced astonishingly elegant solutions a few years earlier by creating virtual "ants" that, when turned loose, revealed how a few false assumptions or bottlenecks could throw an entire system out of whack. A model Bios built of Southwest's cargo operations, for example, cost $60,000 and found a way to save the airline $2 million a year.
Brown proposed that Iacobucci supplement his tool kit with a healthy dose of complexity science. Iacobucci was already hard at work building an "optimizer" program that employed nonlinear algorithms and other mathematical shortcuts to generate scheduling solutions in seconds. But what he really needed, Brown suggested, was an agent-based model (ABM) that would supply phantom traveling salesmen to train the optimizer. Without it, he'd essentially be guessing at the potential number and behavior of his future customers. "Eddy took no convincing," Brown says. "He was telling me, 'Get some guys down here and let's do this.'"
Brown dug up the Ant Farmers, a pair of Bios refugees and expert modelers named Bruce Sawhill and Jim Herriot. Sawhill had been a theoretical physicist at the Santa Fe Institute, while Herriot had been a member of the original team that invented Java at Sun Microsystems (NASDAQ:SUNW). Together, they're DayJet's own Mutt and Jeff, with Herriot playing congenial science professor and Sawhill his mischievous sidekick.
Meanwhile, to build the optimizer, Iacobucci recruited his pair of Russian rocket scientists: Alex Khmelnitsky and Eugene Taits, mathematical wizards he'd hired once before at Citrix. Rather than tackle every scheduling contingency via brute-force computing, the not-Russians cheated by slicing and dicing them into more manageable chunks. They used opaque mathematical techniques such as heuristics and algebraic multigrids, which elegantly subdivide a sprawling problem like this one into discrete patches that can be solved (within limits) simultaneously.
Ironically, the more they slaved over the problem, the less it seemed that throwing a perfect bull's-eye every time was the key to their salvation. The speed of their solutions was proving to be more crucial. If they could provide DayJet with a minute-to-minute snapshot of near- perfect solutions, the system could essentially run the company for them. DayJet would become faster--both in the air and operationally--than any of its competitors could ever hope to be.
With one team working on modeling demand and the other calculating baroque flight plans, Iacobucci and his engineers then concocted a third software system called the Virtual Operation Center. The VOC runs the company in silicon, feeding the phantom customers inside the ABM into the optimizer, which does its best to meet each of their demands with optimal efficiency and maximum gain. Seen on-screen, the VOC is a time-lapse photograph of DayJet's daily operations, also drawing upon maintenance and real-time weather information to produce a final data feed that factors in nearly every facet of the business. Iacobucci compares each run of the VOC with a game of baseball in which the ABM is continually pitching to the optimizer; DayJet has already played several thousand lifetimes' worth of seasons.
Armed with its real-time operating system, DayJet is pursuing a very different idea of optimality than, say, the airlines. With their decades of expertise in the dark arts of yield management, the airlines know exactly how to squeeze every last dollar out of their seats, which is indeed pretty optimal. But they also lack an effective plan B--let alone a plan C or D--in the event that the weather intervenes and schedules collapse. In fact, while, say, JetBlue (NASDAQ:JBLU) may now finally have a contingency plan or two, DayJet's business model is nothing but contingency plans.
Herriot offers another sports metaphor: "Total soccer," popularized by the Dutch in the 1970s, replaced brute-force attacks to the goal with continuous ball movement. "Moving straight to the goal is an excellent way to score, except for one slight problem--the other team," Herriot says. "They're a human version of Murphy's Law. In total soccer, you continually place the ball in a position with not the straightest but the greatest number of ways to reach the goal, the richest set of pathways."
"Each individual pathway may have a lower possibility of reaching the goal than a straight shot," Sawhill chimes in, "but the combinatorial multiplicity overwhelms the other team." The Dutch discovered that a better strategy was a series of good, seamlessly connected solutions rather than a single brittle one.
"The Dutch won a lot of games that way," Herriot adds. "It also created a different kind of player, a more agile, intelligent one. In some sense, we're teaching DayJet how to play total soccer."
In complexity lingo, a chart of all the pathways those Dutch teams exploited would be called a "fitness landscape," a sort of topographical map of every theoretical solution in which the best are visualized as peaks and the worst as deep valleys. "We're dealing with a problem where the problem specification itself is changing as you go along," Sawhill says. "You no longer want to find the best solution--you want to be living in a space of good solutions, so when the problem changes, you're still there." Fluidity is the greater goal than perfection.
To that end, the company has been changing the problem inside its simulators every day for the past four and a half years, looking for those broad mesas of good solutions. And after a million or so spins of the VOC, DayJet has produced a clear vision of the total market and its likely place in it. Iacobucci expects to siphon off somewhere between 1% and 1.5% of all regional business trips within DayJet's markets by 2008, with "regional trips" defined as being between 100 and 500 miles. In the southeast states the company initially has its eye on, that's 500,000 to 750,000 trips a year, out of a total of 52 million, more than 80% of which are currently traversed by car. Yes, DayJet's life-or-death competition is Florida's SUV dealerships, not the airlines. DayJet may even help the airlines slightly: The model predicts some customers who fly DayJet one way will take a commercial flight back home.
The reams of data produced by the VOC have already coalesced into a thick sheaf of battle plans framing best- to worst-case scenarios. And having run the scenarios so relentlessly for so long, Iacobucci is now utterly sanguine about his prospects. When I ask over dinner for the dozenth time about DayJet's presumptive break-even number, he flat out admits there isn't one. "Within the realm of all realistic possibilities--at least 25% of our projected demand to 125% demand--we maintain profitability." Even at 25%? "Sure," Iacobucci replies, "it just takes longer, and takes more [airports], and the margin is much lower. But this isn't going to be what the venture capitalists call the 'walking dead.' If it's a hit, it's going to be a hit pretty quickly."
"We'll see more companies integrate modeling," says former Microsoft CFO Mike Brown. "This is just like the Internet: One day no one had heard of it, the next day we were all using it."
I'm not the only one who has trouble wrapping his head around the numbers, or lack thereof. Iacobucci tells the story of one analyst asked to crunch the numbers ahead of an investment. "He asked a direct question: 'All I want to know is, what formula do I put into this cell to tell me how you come up with a revenue number?'" Iacobucci says. "I told him, 'There ain't no formula to put in that cell! It can't be done! We'll sit you down with our modelers, who will explain the range of numbers we came up with, but they can't be encapsulated in a spreadsheet.'" The would-be investors passed.
Not everyone is so put out by the math involved. Esther Dyson, the veteran technologist and venture capitalist, now runs an annual conference called "Flight School," in which DayJet has played a starring role. "I have no doubt it will work," she says, referring to the software, "and I have no doubt they will spend time refining it and that there will be glitches here and there. But I do think Ed knows how to design very highly available systems"--a reference to his days building operating systems--"and that's exactly what they're doing."
Mike Brown, who did ante up and today sits on DayJet's board, is convinced that businesses big and small will increasingly turn to modeling as a way of developing--or troubleshooting--their business plans, mapping out strategies and market expectations that go far, far beyond spreadsheets and PowerPoint (NASDAQ:MSFT) decks. "We'll see more and more companies integrate modeling into the heart of their business. This is just like the Internet: One day no one had heard of it, the next day we were all using it."
Since Iacobucci sees himself as being in the operating-systems business, he has no intention of giving that system away. (He learned that lesson the hard way at IBM.) He doesn't want to build what he calls "horizontal" software that gets shared, e.g., Web 2.0 and Windows, the two great platforms for which every programmer in Silicon Valley seems to be writing widgets these days. Where everyone else in the business sees limitless opportunities in snap-together applications, Iacobucci sees a playing field so flat as to have no barriers to entry at all, and he doesn't like it.
According to Dyson, DayJet's competitors have so far pooh-poohed its software, assuming they'll be able to buy their own off the shelf at some point. Eclipse Aviation's Vern Raburn hopes Iacobucci might be persuaded to license his tools, because Raburn's own business model depends upon air taxis' taking off. Iacobucci says that isn't going to happen. "There's a shift away from building another platform toward building highly integrated, vertical, special-purpose, high-performance systems," he argues. Iacobucci envisions more companies like his own, in which the competitive advantage resides in custom-built, deeply proprietary, real-world operating systems that don't just streamline accounting, but become the central nervous systems of entirely new, scalable businesses. He's looking to build barriers to entry out of brainpower--so much of it that rivals can never catch up. ("It's like in Dr. Strangelove," Sawhill quips. "'Our German scientists are better than their German scientists.'")
Iacobucci points to Google (NASDAQ:GOOG) as an example of what a vertical system can accomplish. While everyone raves about free services on Google, the largely invisible supercomputers in Google's data centers are themselves invisibly tackling a variation on the traveling-salesman problem: How do you solve millions of searches in parallel at any given second? "When you get into mesh computing," the name for Google's technique, "that's what it's all about: managing the complexity," Iacobucci insists.
But no company has ever built a business model around complexity from the ground up--until DayJet. Thumbing his nose at the prevailing ethos in software circles of "the wisdom of crowds," let alone that "IT doesn't matter," Iacobucci has set out to first invent and then dominate a market he might have otherwise just sold software to. "When we built generic software at IBM and Citrix, the other side would always reverse-engineer it," he says. "The only thing the customer sees here is an incredible service. This is 'software as a service.'"
Source: FastCompany.com
Tim O'Reilly of O'Reilly media spoke at Microsoft Research earlier today. It was a great, wide-ranging talk pounding through 103 slides roaming from social networking, through sensor and ambient computing, to Web2.0.
Four themes for the talk:
· Thoughts on social networking
· Sensors and Ambient Computing
· Web 2.0 and Wall Street
· Open Source Hardware
"The future is here. It's just not evenly distributed yet" -- William Gibson
Thoughts on social networking:
· Social networking sites are becoming personal CRM
· Need to have it reflect my REAL friends vs acquaintances, hangers on, etc.
· "What's new" is going to be everywhere
· Need to be able to have groups of friends based upon what "type" of friends they are. Where they were met, what context, more dimesions, etc.
· Want sharing of data with control across different social networking sites (need data portability)
· Want to make social networking applicable to enterprise problems (huge opportunities in that the existing tools are weak)
· Open social is weak. We're bringing together 60 or 70 social networking sites to get sharing that works
· Wallop was a GREAT as a research project but seemed to have lost what was cool about the original ideas when spun out.
· Photosynth is very cool as well. Use 'em and don't lose 'em?
· Need to bring social networking features into Outlook. It would be killer!
· What differentiated Web 2.0: real time user-facing services based upon data
· How do we take deep wells of data that all corporations serving customers have and turn it into value for your users.
o Note: my phone and my email know who my friends are. They should be running deep heuristics on this data. They should know who my real friends are.
· We grew up where the answers were black and white. We're entering a world where the answers are softer. We need close enough and more "pretty good" …
o I shouldn't have to tell the system who my friends are
o Xobni does a pretty good job in this dimension
· There are huge opportunities to re-invent enterprise software along Web 2.0 lines
o Social networking meets Web 2.0.
o Exploit the vast wells of data
· ½ of all mashups are Google Maps based (programmableweb.com)
· The most useful APIs:
o Aren't about you are your services
o Let developers try new things
o Exist to be stretched out of shape
o If someone violates your TOS, perhaps it is wrong
· Amazon AWS is doing this best (S3 is great)
· P2p hasn't been close to fully exploited yet.
· Amazon ASIN is an extension of ISBN. It's a name space for all products rather than just books.
· On the web, open source doesn't matter. It's from an old era. What matters today is open data.
· Open Source was the solution to software distribution over a fragmented set of computer architectures. It's no longer the problem. If I gave you the source to Google, you couldn't run it. What's interesting today is large aggregate data sources rather than access of the programs.
Sensors and Ambient Computing
· We are moving out o the world of people typing on keyboards. Increasingly apps will be driven by new kinds of sensors.
· There will be more and more sensor based applications.
· For example, Nintendo WII
· Photosynth is this amazing Web 2.0 application. All of our cameras become sensor in this large collective DB.
· Pathintellegence is using Gnu Radio to track cell phone carrying customers (everyone) to track shopping patters
· Jaiku is a smart presence application (using cell tower triangulations)
· LastFM delivers results (the music you like) without asking me to enter data. Just watch me using sensors. This is the future.
o PageRank found meaning in hidden data (page links). Finding data from large aggregations and extract meaning statistically.
· It's about "Programming Collective Intelligence"
Web 2.0 and Wall Street
· It's all about networked intelligence applications
· Dark Pools are used by Hedge funds to avoid moving the market by doing large numbers of small trades through a dark pool. Liuqidnet, for example, does several multiples of the traffic of the NYSE. Dark Pools are ultra-high speed, anonymous trading markets. Dark Pools are used by hedge funds to operate privately and anonymously. I expect to see a backlash across the web2.0 economy and a bigger focus on privacy and anonymous operation.
Open Source Hardware
· Didn't have time to cover it in this talk.
WS-* was a failed attempt by big companies to make it so hard that we needed their tools.
Why doesn't my address book, email, and phone software tell me about interesting things that relate to me. For example, I know Elop who just took over Office. Why couldn't this be done automatically?
There is LOTS of innovation happening right now in hardware. It's the next big frontier.
Many massively multi-player games have substantial parts of the game played and scored locally. The only way to get a sufficiently responsive gaming experience is to have the high speed game to player interactions local. Offloading some of the interactions from the server to the client is also an important way to reduce costs since the players pay for client side hardware and running some decisions locally offloads server and network traffic reducing the game providers costs. Off-loading some interactions to the client works extremely well at maintaining game responsiveness and reducing server-side costs but it does open up opportunity to cheat. If the client program can be modified or otherwise manipulated, a player can get an unfair advantage.
There are now markets for virtual merchandise like special armor, more powerful weapons, etc. These markets charge real money for "ownership" of virtual merchandise. That which another player has earned through playing the game can be transferred for use to another. Essentially there is a market for various forms of virtual merchandise and many of these transactions are conducted using eBay.
When the possibility of cheating by modifying or otherwise manipulating the client side game components is coupled with a financial incentive, cheating is almost assured. That's exactly what's happening in many instances. The December issue of IEEE Spectrum includes a story about Richard Thurman who, during his game manipulating days, had a fleet of 30 servers running day and night playing Ultima Online using modified game client software. At peak, Thurman claims he was making $25,000 each month.
Even without cheating, where there is value, there is a potential business model to be found. Companies, often called gold farmers, set up shop in low wage countries where employees play games 12 hours a day. Recent estimates peg the gold farming population in China at over 100,000 farmers. The bounty earned by each farmer is then sold for hard currency to players interested in the virtual asset. Gold farmers play by the same rules as ordinary players and typically use standard clients, but have some advantage in numbers. They can be more organized. Many players can gang up on a single foe that would be ordinarily very difficult to kill and execute the mission, reap the reward for their employer and move on to the next kill.
Clearly there are willing sellers of virtual wares. Some obtain their virtual goods by employing others, some using automata as described above, and some by the old fashioned way, by playing the game for hours at a time. There are also willing purchasers of virtual wares as evidenced by the traffic on eBay. Whenever there are buyers and sellers, there is opportunity to make the market more efficient by bringing together more buyers, more sellers, and helping to match their interests. Several companies have emerged to make markets in virtual goods. The largest example of a virtual to real world broker is IGE based in Hong Kong but there are many such companies out there matching buyers to sellers and taking a slice of each transaction as a fee.
The IEEE Spectrum article is posted at: http://spectrum.ieee.org/dec07/5719.
H:mvdirona.com | W:research.microsoft.com/~jamesrh | Msft internal blog: msblogs/JamesRH
It's finally done! Back in August of 2006 Joe Hellerstein asked me to join him and Mike Stonebraker in producing an article for Foundations and Trends in Database Systems. The project ended up being bigger than I originally understood, and the review process always takes longer than any of us expect. The goal for the paper is to document those aspects of commercial database management systems that aren't well-covered in the literature. We focused on those aspects of modern relational systems not broadly covered in the existing literature and let that criteria drive what material would be included.
The paper was published in Volume 1, Issue 2 of Foundations and Trends in Database Systems and is posted here: Architecture of a Database System.
The abstract:
Database Management Systems (DBMSs) are a ubiquitous and critical component of modern computing, and the result of decades of research and development in both academia and industry. Historically, DBMSs were among the earliest multi-user server systems to be developed, and thus pioneered many systems design techniques for scalability and reliability now in use in many other contexts. While many of the algorithms and abstractions used by a DBMS are textbook material, there has been relatively sparse coverage in the literature of the systems design issues that make a DBMS work. This paper presents an architectural discussion of DBMS design principles, including process models, parallel architecture, storage system design, transaction system implementation, query processor and optimizer architectures, and typical shared components and utilities. Successful commercial and open-source systems are used as points of reference, particularly when multiple alternative designs have been adopted by different groups.
FusionIO has released specs and pricing data on their new NAND flash SSD: http://www.fusionio.com/products.html (Lintao Zhang of msft Research sent it my way). 100,000 IOPS, 700 MB/s sustained read, and 600 MB/s sustained write. Impressive numbers but let's dig deeper. In what follows, I compare the specs of the FusionIO part with a "typical" SATA disk. For comparison with the 80GB FusionIO part, I'm using the following SATA disk specs: $200, 750GB, 70 MB/s sustained transfer rate, and 70 random IO operations per second (IOPS). Specs change daily but it's a good approximation of what we get from commodity SATA disk these days.
Obviously, the sustained read and write rates that FusionIO advertises are substantial. But to be truly interesting, they have to produce higher sustained I/O rates/dollar than magnetic disks, the high-volume commodity competitor. A SATA disk runs around $200 and produces roughly 70 MB/s sustained I/O. Looking at read and normalizing for price by comparing MB/s per dollar, we see that the FusionIO part can do roughly 0.29 MB/s/$ whereas the SATA disk will produced 0.35 MB/s/$. The disk produces slightly better sequential transfer rates per dollar. This isn't surprising, in that we know that disks are actually respectable at sequential access—this workload pattern is not where flash really excels. For sequential workloads or workloads that can be made sequential, at the current price point, I wouldn't recommend flash in general or the FusionIO SSD in particular. Where they really look interesting is in workloads with highly random I/Os.
Looking at capacity, there are no surprises. Normalizing to dollars/GB, we see the FusionIO part at $30/GB and the SATA disk at $0.26/GB. Where capacity is the deciding factor, magnetic media is considerably cheaper and will be for many years. Capacity per dollar is not where flash SSDs look best.
Where flash SSDs really excels, and where the FusionIO part is particularly good, is in random I/Os per second. They advertise over 100,000 random 4k IOPS (87,500 8k IOPS) whereas our SATA disk can deliver about 70. Again normalizing for costs and looking at IOPS per dollar, we see the FusionIO SSD at 41 IOPS/$ whereas the SATA disk is only 0.27 IOPS/$. Flash SSDs win, and win big, on random I/O workloads like OLTP systems (usually random-I/O-operation bound). These workloads typically run the smallest and fastest disks they can buy, and yet still can't use the entire disk since the workload I/O rates are so high. To support these extremely hot workloads using magnetic disk, you must spread the data over a large number of disks to effectively dilute the workload I/O rate to that which disks can support.
For workloads where the random I/O rates are prodigious and the overall database sizes fairly small, flash SSDs are an excellent choice. How do we define "fairly small"? I look at it this way: it's a question of I/O density and I define I/O density to be random IOPS per GB. The SATA disk we are using as an example can support 0.09 IOPS/GB (70/750). If the workload requires less than 0.09 IOPS/GB, then it will be disk-capacity bound whereas if it needs more than 0.09 IOPS/GB, then it's I/O bound. Assuming the workload is IO bound, how to decide whether SSDs are the right choice? Start by figuring out how many disks would be required to support the workload and what they would cost: take the sustained random IOPS required by the application and divide by the number of IOPS each disk can sustain (70 in the case of our example SATA drive or 180 to 200 if using enterprise disk). That defines the cost of supporting this application using magnetic disk. Now figure the same number for flash SSD. Aggregate workload required I/O rate divided by the sustained random IOPS the SSD under consideration can deliver. This will determine how many disks are needed to support the I/O rate. Given that flash SSDs can deliver very high I/O densities, you also need to ensure you have enough SSDs to store the entire database. Take the maximum of the number of SSDs required to store the database (if capacity bound) and the number of SSDs required to support the I/O rate (if IOPS bound), and that's the number of SSDS needed. Compare the cost of the SSDs with the cost of the disk required to support the same workload, and see if SSD is cheaper. For VERY hot workloads, flash SSDs will be cheaper than hard disk drives.
I should point out there are many other factors potentially worth considering when deciding whether a flash SSD is the right choice for your workload, including the power consumption of the disk farm, the failure rate and cost of service, and the wear-out rate and exactly how it was computed for the SSDs. The random I/O rate is the biggest differentiator and the most important for many workloads, so I haven't considered these other factors here.
Looking more closely at the FusionIO specs, we see they give specs on random IOPS but they don't specify the read-to-write ratios they can support. We really need to see the number of random read and write IOPS that can be sustained. This is particularly important for flash SSDs since, with these devices, getting extremely high random write I/O rates is VERY difficult and this is typically where the current generation devices fall down. In the case of the FusionIO SSD, we still don't have the data we need to make a decision and would still need to get third-party benchmark data before making a buying decision.
Another option to consider when looking at very hot workloads is to move the workload into memory when the I/O densities are extreamly high. Memory prices continue to fall and several memory appliance start-ups have recently emerged. I suspect hybrid devices that combine very large DRAM caches with 10 to 100x larger flash stores will likely emerge as great choices over the next year or so. Of the memory appliance vendors, I find Violin Memory to be the most interesting of those I've looked at (http://www.violin-memory.com/).
I do love the early performance numbers that FusionIO is now reporting—these are exciting results. But remember, when looking at flash SSDs, including the FusionIO part, you need to get the random write IOPS rate before making a decision. It's the hardest spec to get right in a flash SSD and I've seen results as divergent as random write IOPS at 1/100th the (typically very impressive) read rate. Ask for random write IOPS.
SpecPOWER is the first industry standard benchmark that evaluates power and performance characteristics of high volume servers. Yesterday the final spec was released.
I have quibbles with this benchmark but they really are mostly quibbles. Generally, I'm thrilled to see a benchmark out there that shines a light on server power consumption. The entire industry needs it. Benchmarks drive action and we have lots of low hanging fruit to grab when it comes to server power/performance.
The press release is up at: http://www.spec.org/power_ssj2008/ and early results are at http://www.spec.org/power_ssj2008/results/power_ssj2008.html.
I remember working on IBM DB2 years back when we first ran TPC-A. Prior to the benchmark we had a perfectly good database management system that customers were using successfully to run some very large businesses. When we first ran TPC-A, let's just say we didn't like what we saw. We got to work and ended up improving DB2 by a full factor of 10 in that release and then went on and improved by a further factor of 4 in the next release. Yes I know the only way to improve that much is to start off REALLY needing it. That's my point. As a direct result of the benchmark giving us and customers increased visibility into OLTP performance, the product improved 40x in less than 5. Customers gained, the product got better, and it gave the engineering team good goals to rally around. Benchmarks help customers.
As benchmarks age, it's harder to find the big differentiating improvements. Once the easy changes are found and even the more difficult improvements have been worked through, benchmark specials typically begin to emerge. Companies start releasing changes that help the benchmarks and do nothing and, in rare cases, can even hurt real customer workloads. Eventually, this almost always happens and causes industry veterans, myself included, to distrust benchmarks. We forget that in the early years of most benchmarks, they really did help improve the product and delivered real value to customers.
I'm very happy to see SPEC release a benchmark that measures power and I'm confident that it will help drive big power efficiency gains. Sure, we'll eventually see the game playing benchmark specials but, for now, I predict many of the improvements will be real. This will help the industry evolve more quickly. Now can someone please start working on a data center efficiency benchmark? Huge innovations are waiting in data center design and even more in how existing designs are deployed.
I'm online over the holidays but everyone's so busy there isn't much point in blogging during this period. More important things dominate so I won't be posting until early January.
Have a great holiday.
James Hamilton, Windows Live Platform Services Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052 W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201
Yesterday Nanosolar announced it has started to sell cheap solar panels at a materials cost of roughly $1/W and a finished panel cost of ~$2/W: http://www.reuters.com/article/technologyNews/idUSN1846022020071218. As a reference point, coal powered electricity is about $1/W when considering the fully burdened cost of the plant and operations. When a new technology is within a factor of two of an old, heavily invested technology, I get excited. This is getting interesting.
Nanosolar was founded in 2002 and has plants in San Jose, CA and another near Berlin, Germany. They received $20M from Mohr Davidow Ventures. Other investors include Sergey Brin and Larry Page.
More articles on Nanosolar: http://www.nanosolar.com/articles.htm. Thanks to Will Whitted for directing me to Nanosolar.
This note on the Google infrastructure was sent my way by Andrew Kadatch (Live Search) by way of Sam McKelvie (Cloud Infrastructure Services). It's not precise in all dimensions, but it does a good job of bringing together the few facts that have been released by Google, and it points to its references if you are interested in a deeper dive.
The infrastructure summary is at: http://highscalability.com/google-architecture. No new data here, but a good summary if you haven't read all the direct sources from which the summary was drawn.
James Hamilton, Windows Live Platform Services Bldg RedW-C/1279, One Microsoft Way, Redmond, Washington, 98052 W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | email@example.com
Last week Google announced their Renewable Energy Cheaper than Coal (RE<C) initiative. This is a two pronged effort combining external investment with internal Google research. The external investments include $10M in Makani Power, a startup aiming to harness high altitude winds. Makani explains that high altitude winds have the highest energy density of any renewable energy source. Google also invested in eSolar as part of this program. eSolar is a startup aiming to produce utility scale solar energy farms ranging from 25MW to 500MW.
More detail on:
· Makani: http://www.google.com/corporate/green/energy/makani.pdf
· eSolar: http://www.google.com/corporate/green/energy/esolar.pdf
In addition to the external investments, Google is staffing a team with a mission of assembling 1 Gigawatt of renewable energy capacity that is cheaper than coal. More detail at: http://www.google.com/corporate/green/energy/index.html.
It's hard to argue with the "cheaper than coal" target. If renewable resources actually could be less expensive than coal, they would be used without social pressure, government subsidies, or other external tweaks to the market. Looking at the business side, this looks like a good marketing investment for Google as well. Google's datacenter footprint continues to grow at an unprecedented pace while, at the same time, datacenter power consumption is becoming an social concern and attracting regulatory attention. This investment is clearly a well timed and well thought through business investment as well.
The number 1 Amazon AWS requirement just got met: structured storage. Amazon announced SimpleDB yesterday although it's not yet available for developers to play with. I'm looking forward to being able to write a simple application against it – I've had fun with S3. But, for now, the docs will have to do.
In the announcement (http://aws.amazon.com/simpledb), it's explained that SimpleDB is not a relational DB. AWS notes that some customers run relational DBs in EC2 and, those that need complex or strictly enforced schema, will continue to do this. Others that only need a simple structured store with much less administrative overhead will use SimpleDB. AWS explains that they will make it increasingly easy to do both.
The hard part of running a RDBMS in EC2 is that there is no data protection. The EC2 local disk is 100% ephemeral. Some folks are using block replicators such as DRBD (http://www.drbd.org/) to keep two MySQL systems in sync. It's kind of a nice solution but requires some skill to set up. When AWS says "make it easier" I suspect they are considering something along these lines. A block replicator would be a wonderful EC2 addition. However, for those that really only need a simple structured store, SimpleDB is (almost) here today.
My first two interests are data model and pricing. The data model is based upon domains, items, attributes and values. A domain roughly corresponds to a database and you are allowed up to 100 domains. All queries are within a domain. Within a domain, you can create items. Each item has an ID (presumably unique). Every Item has attributes and attributes have values. You don't need to (and can't) declare the schema of an item and any item can have any number of attributes up to 256 per item. Attributes have values and the values can repeat. So a given item may have an attribute or may not and it can have the attribute more than once. Attributes have values and value are simple UTF-8 strings limited to 1024 bytes. All attributes are indexed.
The space overhead gives a clue to storage format:
· Raw byte size (GB) of all item IDs + 45 bytes per item +
· Raw byte size (GB) of all attribute names + 45 bytes per attribute name +
· Raw byte size (GB) of all attribute-value pairs + 45 bytes per attribute-value pair
The storage format appears to be a single ISAM index of (item, attribute, value). It wouldn't surprise me if the index used in SimpleDB is the same code that S3 uses for metadata lookup.
The query language is respectable and includes: =, !=, <, > <=, >=, STARTS-WITH, AND, OR, NOT, INTERSECTION AND UNION. Queries are resource limited to know more than 5 seconds of execution time.
The storage model, like S3, is replicated asynchronously across data centers with the good and the bad that comes with this approach: the data is stored geo-redundantly which is wonderful but it is possible to update a table and, on a subsequent request, not even see your own changes. The consistency model is very weak and the storage reliability is very strong. I actually like the model although most folks I talk to complain that it's confusing. Technically this is true but S3 uses the same consistency model. I've spoken to many S3 developers and never heard a complaint (admittedly some just don't understand it).
Pricing was my second interest. There are three charges for SimpleDB:
· Machine Utilization: $0.14/machine hour
· Data Transfer:
o $0.10 per GB - all data transfer in
o $0.18 per GB - first 10 TB / month data transfer out
o $0.16 per GB - next 40 TB / month data transfer out
o $0.13 per GB - data transfer out / month over 50 TB
· Structured Storage: $1.50 GB/month
$1.50 GB/month or $18 GB/year is 10x the $1.80 GB/year charged by S3. Fairly expensive in comparison to S3 but a bargain compared to what it cost to manage an RDBMS and the hardware that supports it.
More data on SimpleDB from AWS: http://aws.amazon.com/simpledb.
Sriram Krishnan has an excellent review up at: http://www.sriramkrishnan.com/blog/2007/12/amazon-simpledb-technical-overview.html
Thanks to Sriram Krishnan (Develop Division) and Dare Obasanjo (WinLive Platform Services) for sending this one my way. I'm really looking forward to playing with SimpleDB and seeing how customers end up using it. Overall, I find it tastefully simple and yet perfectly adequate for many structured storage tasks.
SpecPOWER is the first industry standard benchmark that evaluates power and performance characteristics of high volume servers. Yesterday the final spec was released (see below thanks to Kushagra Vaid of GFS H/W Architecture & Standards).
From: email@example.com [mailto:email@example.com] On Behalf Of Alan AdamsonSent: Tuesday, December 11, 2007 8:44 AMTo: firstname.lastname@example.com: (osgmembers-363) SPECpower benchmark availability
I am delighted to announce the public availability of the initial SPECpower benchmark, SPECpower_ssj2008 - you can find the initial press release here http://www.spec.org/power_ssj2008/ , and see initial results here : http://www.spec.org/power_ssj2008/results/power_ssj2008.html This benchmark release is the result of a long and strenuous process (I cannot recall a committee meeting four days a week for several hours each day), needed partly because of the new ground being plowed. I expect that all forthcoming benchmark development groups will be thinking about how to incorporate power measurements, and that the power committee will be looking at extending the scope of its workloads, and also helping other committees learn from their experience. And now, on to several other benchmarks in or heading for general membership review. This is a pretty exciting time for the OSG! My thanks and congratulations to those who worked so hard on this initial SPECpower benchmark. Alan Adamson 905-413-5933 Tieline 969-5933 FAX: 905-413-4854OSG Chair, SPEC Internet: firstname.lastname@example.com Technology CentreIBM Toronto Lab
Google announced a project in October 2006 to install 9,212 solar panels on the roof of its headquarters complex. Currently over 90% of these panels are installed and active. They expect the installation will produce 30% of the peak power requirements of the headquarters buildings. For example, as I post this, they report 0.6 MW have been produced over the last 24 hour period. The roughly ½ megawatt is arguably in the noise when compared with the Google total data center power consumption which remains a carefully guarded secret. With high 10's of data centers world-wide each consuming in the 10WM range, the 0.6 produced on the roof of the HQ building is a very small number. Nonetheless, 0.6MW is MUCH better than nothing and it's great to see these resources committed to making a difference.
The real time power production report for this installation can be found at: http://www.google.com/corporate/solarpanels/home. Thanks to Lewis Curtis for sending this my way.
I've long argued that the firm and clear division between development and operations common in many companies is a mistake. Development doesn't feel the pain and understand what it takes to make their services more efficient to operate. Operations tends to hire more people to deal with the mess. It doesn't work, it isn't efficient, and it's slow. The right model has the development/ops line very blurred. The Amazon model takes this to an extreme with essentially a "you wrote it, you run it" approach. This is nimble and delivers the pain right back where it can be most efficiently solved: development. But it can be bit tough on the engineering team. The approach I like most is a hybrid, where we have a very small tier 1 support team operations with everything else going back to development.
However, if I had to err to one extreme or the other, I would head down the Amazon path. A clear division between ops and dev leads to an over-the-wall approach that is too slow and too inefficient. What's below is more detail on the Amazon approach, not so much because they do it perfectly but because they represent an extreme and therefore are a good data point to think though. Also note the reference to test-in-production found in the last paragraph. This is 100% the right approach in my opinion.
When I was at Amazon (and I don't think anything has changed) services were owned soup to nuts by the product team. There were no testers and no operations people. Datacenter operations, security, and operations were separate, centralized groups.
So 'development' was responsible for much of product definition and specification, the development, deployment and operation of services. This model came down directly from Jeff Bezos; his intention was to centralize responsibility and take away excuses. If something went wrong it was very clear who was responsible for fixing it. Having said that, Amazon encouraged a no-blame culture. If things went wrong the owner was expected to figure out the root cause and come up with a remediation plan.
This had many plusses and a few negatives. On the positive side:
· Clear ownership of problems
· Much shorter time to get features into production. There was no hand-off from dev to test to operations.
· Much less requirement for documentation, (both a plus and a minus)
· Very fast response to operational issues, since the people who really knew the code were engaged up-front.
· Significant focus by developers on reliability and operability, since they were the people responsible for running the service
· Model works really well for new products
· Developers have to carry pagers. On-call rotations, with people on-call required to respond to a sev-1 within 15 minutes of being paged 24 hours a day. Could lead to burn-out.
· Ramp up curve for new developers is extremely steep because of the lack of documentation and process
· For some teams the operations load completely dominated, making it very difficult to tackle new features or devote time to infrastructure work.
· Coordinating large feature development across multiple teams is tough.
The one surprise in this is that in my opinion code quality was as good as or better than here, despite having no testers. People developed an approach of deploying to a few machines and observing their behavior for a few days before starting a large roll out. Code could be reverted extremely quickly.
Michael Hunter, who authors the Testing and Debugging blog at Dr. Dobb's Journal, asked me for an interview on testing related topics some time back. I've long lamented that, industry-wide, there isn't nearly enough emphasis on test and software quality assurance innovation. For large projects, test is often the least scalable part of the development process. So, when Michael offered me a platform to discuss test more broadly, I jumped on it.
Michael structures these interviews, and his subsequent blog entry, around five questions. These ranged from where I first got involved in software testing, through the most interesting bug I've run into, what has most surprised me about testing, what's the most important thing for a tester to know, and what's the biggest challenge facing the test discipline over the next five years.
Michael's interview is posted at: http://www.ddj.com/blog/debugblog/archives/2007/12/five_questions_39.html.
Disclaimer: The opinions expressed here are my own and do not necessarily represent those of current or past employers.


text:
Let's Talk Education
By Rebecca Teti
Fridays: Education
(Join each day's Coffee Talk discussion: Mon: Parenting; Tues: Open Forum; Wed: NFP; Thu: Marriage; Fri: Education; Sat/Sun: Changing Roles)
Whether your children attend school or are homeschooled, this is the spot to ask questions about curricula, religious education, parent-teacher relationships, or academic concerns of any kind.
Please join us!
Anyone recommend materials for talking to your children about sexuality? It looks like we'll be having this conversation with our 10-year old sooner rather than later (thank you, Boy Scouts of America). We've been educating him generally his whole life, and we have a good handle on the Theology of the Body. Just wondering if anyone used any specific materials that they found helpful.
Dr. Colleen Kelly Mast has an entire curriculum called Love and Life. There is a student book and parent book. Also, I am looking at Jason Evert's Theology of the Body for Him/Her (it has two "fronts" with a flip over format) currently and it is very good. As a parent, I have found it best to educate myself so I will be ready when the opportunity arises rather than have formal discussions. God bless your discussions!
We enjoyed using Passport2Purity® - Family Life Today www.familylife.com/find-help/key-resources/passport2purity The Passport2Purity Getaway Kit helps build heart-to-heart communication between you and your pre-teen while laying a foundation of purity that will prepare ...
Here's a simple approach: http://www.patheos.com/blogs/duffy/2013/05/towards-a-true-sex-education/
"Listen Son" http://www.ewtn.com/library/FAMILY/LISTENSN.TXT Also, it can be purchased through AngelusPress http://angeluspress.org/Books/Family-Life/SET-Listen-Son-/-Mothers-Little-Helper

subset: commoncrawl


text:
Not much of an answer, however.
John Aravosis Follow me on Twitter: @aravosis | @americablog | @americabloggay | Facebook | Google+ | LinkedIn. John Aravosis is the editor of AMERICAblog, which he founded in 2004. He has a joint law degree (JD) and masters in Foreign Service from Georgetown (1989); and worked in the US Senate, World Bank, Children's Defense Fund, and as a stringer for the Economist. Frequent TV pundit: O'Reilly Factor, Hardball, World News Tonight, Nightline & Reliable Sources. Bio, article archive.
Donations are not tax-deductible.
If you like our progressive activism, and hard-hitting reporting, then please support our work.
For a while now, blogging has no long really paid the bills. I remain committed to keeping AMERICAblog afloat, and would appreciate your financial support in doing just that.
As always, thank you so much for your continued support. May you all have a wonderful new year.
JOHN and the Gang.


text:
Lace Dress
Style # 229913
Lacey racy. Our semi see-through mini dress styles well with a mid-section belt and is only transparent in the sexiest of all places: arms, back, and collar. Super fitted hip-hugging fit that ensures valet rushes to get your door!
- 100% Lace
- Hand wash
- Imported
- Center back to hem: 39" (99 cm)
- Model is 5'10" and wears a US size S
Rated 5 out of 5�by 3
Rated 5 out of 5�by Lala5 beautiful fit
This dress is not only stunning on, but it is very comfortable and elegant. I love this purchase.
July 18, 2014
Rated 5 out of 5�by RebNY2 Beautiful Lace Dress
I love this dress! I got the green and its so pretty in person! Its very classy and sexy at the same time. Conservative enough to wear to any function, its fitted and doesn't show too much skin, yet the hem is sexy and it hugs your curves beautifully! Fits true to size, 110 lbs, 32c 5'2" xs fits great! I highly recommend! You wont be disappointed!
March 25, 2014
Rated 5 out of 5�by AllyMarie gorgeous
this dress is just so pretty on so many levels its crazy! This is such a must have in white, black, red and a nice peachy colour (all with nude under the coloured lace)
March 13, 2014
1-3 of 3


text:
Please login to get more out of this site
You are currently not logged in, do so now to add comments, like articles as well as photos, submit to competitions, translate articles and gain access to unlimited photo upload! If you are not registered yet, go here to find out more about Lomography and join now! If you just want a limited account for now you can use Twitter or Facebook. We would love to have you as a member here!
One person likes this photo
Liking photos helps us to present the best ones to you and others. One person likes this photo. One person likes this photo: baitnicart.
No comments yet, be the first
Where is this?
Nearby popular photos – see more
Nearby LomoLocations
Nacogdoches - The Oldest Town in Texas
Nacogdoches is the oldest town in Texas, when people hear her name, question marks raise. Wi…


text:
There are a lot of things you can personalise in your phone to make it your own.
Personalise the start screen
- To pin an app to the start screen, swipe left, tap and hold the app, and tap pin to start. You can also pin contacts and websites.
- To move a tile, tap and hold the tile, drag and drop it to the new location, and tap the screen.
- To resize a tile, tap and hold the tile, and tap the arrow icon.
- To remove a tile from the start screen, tap and hold the tile, and tap the pin icon.
Change the background photo
- To change the photo on your lock screen, on the start screen, swipe left, tap Settings > lock screen > change photo, select a photo, adjust it to get the best possible fit, and tap the done icon.
- To change the background photo in the Photos Hub, tap Photos > … > choose background, select the photo, and tap the done icon.
Change your background or accent colour
- On the start screen, swipe left, and tap Settings > theme and Background or Accent colour.
- To enable black accent, go to Settings > ease of access and switch High contrast toggle to On. Note that this setting hides your lock screen background; you will see either white or black on lock screen depending upon the Background you have chosen in your lock screen settings. By default High Contrast mode is switched Off (Settings > ease of access > High contrast).
Add notifications to the lock screen
- You can freely decide which app notifications, such as missed calls or new messages, you want to see on the lock screen. On the start screen, swipe left, tap Settings > lock screen, tap a notification slot, and select the app you want to add.
Change your ringtone and phone tones
- On the start screen, swipe left, tap Settings > ringtones+sounds, select the type of ringtone or alert you want to change, and select the sound.
- If you have an mp3 file of a song on your PC, you can make it your ringtone. Connect your phone to your PC, and use the file manager of your PC to move the song you want to use as a ringtone to the ringtone folder in your phone. To set the song as your ringtone, in the ringtones+sounds view, tap Ringtone, and under Custom, tap the song. You can use a song as your ringtone if it's not protected with digital rights management (DRM), and it's smaller than 30 MB. Your PC must have Windows Vista, Windows 7, or Windows 8.
Change the font size
- On the start screen, swipe left, tap Settings > ease of access, and drag the Text size slider.
Filter your contacts
- If you do not want to see all your contacts from different social networking accounts in your contacts list, tap People > … > settings > filter my contacts list, and select the accounts whose contacts you want to see.
Switch off word suggestions when writing
- If you do not want your phone to automatically offer suggestions when you write, on the start screen, swipe left, tap Settings > keyboard, tap your language keyboard, and clear the Suggest text check box.


text:
I had no idea my "1/4 black" would raise so much comment. Maybe I had better give you more context.
Pete, Trey, and Dixon are watching the scene in which Trey bottoms for Dixon.
Trey: (watching himself sit on Dixon's cock) "Gettin' on that chocolate! What ethnicity are you?"
Dixon: "I'm three-quarters Caucasian and one-quarter African-American."
Trey: "Looks good on you."
Dixon: "It's a good mix. I like it."
From there they go back to discussing the scene. Pete never said a word on the subject.
My only observation is that mixed race is so common most big cities that it is no longer remarkable at all, just interesting sometimes, as in the case of Trey's question.


text:
Use logic, language and mathematics skills while solving 50 riddles and playing 4 arcade-style games. Teaches logic and reasoning, vocabulary, mathematics and problem solving. Logic and reasoning skills begin to develop at an early age and are used to solve problems and draw conclusions. The larger their vocabulary, the easier it is for children to comprehend what they read and hear and communicate with others. Mathematics involves recognizing patterns and solving problems using logic, number sense, and spatial relations. Children use logical reasoning skills and investigate multiple possibilities to develop creative problem-solving strategies.
Although we can only offer these items at full price, we love the products and are excited to bring them to our members.
Brand: LeapFrog
Headquartered in Emeryville, California, LeapFrog was founded in 1995 when entrepreneur Michael Wood was unable to find toys that would help teach his son to read. Since then, the company has grown exponentially, developing a number of products focusing on three main areas: educational gaming, grade school learning toys, and reading solutions. The LeapPad was LeapFrog's flagship product from 1999 to 2007, when it was replaced by the Tag reading system. Another notable product is the Leapster, a handheld gaming device with a touch screen.


text:
(01-01-2012 01:39 PM)0264 roadrunner1964 Wrote: Congrats on the Applewhite and McCoy autos.
(01-01-2012 04:25 PM)qbcollectorlionsfan Wrote: Nice break Wendy.....they really need to have a UConn or Ohio State version of this....lol
(01-01-2012 04:47 PM)alstott9adams Wrote: Sweet applewhite
(01-01-2012 05:03 PM)lpearce7 Wrote: Awesome hits, love that Applewhite. Are just the rookies and inserts /210?
(01-02-2012 02:41 PM)ftballcardz Wrote: Do you have a list of the autos your missing?
(01-02-2012 03:10 PM)Duchovny2 Wrote: Funny to see the Steve McMichael card
(01-02-2012 06:04 PM)ricky williams 34 Wrote: Don't even go there. Say more...
(01-03-2012 10:39 AM)ftballcardz Wrote: WHAT..... just curious
(01-03-2012 12:38 PM)whitewolf2367 Wrote: Awesome cards Wendy


text:
What Brewer would you trust with your girlfriend?
Off-Base October 27th, 2010
I do my best not to get jealous when it comes to my girlfriend. Whether it's her puzzling lifelong crush on Jason Mraz or the people who won't stop telling me how "lucky" I am to be with her or any of her assorted male best friends, I do my best not to let it bother me. After all she is with me and not them for a reason, I don't know what that reason might be (I suspect some sort of experiment for college credit), but the fact is that she is with me and not them. So I try not to get jealous, but she's attractive and she owns a vagina so she gets hit on a lot. I do my best not to let it bother me, but sometimes it does. Like the other day.
The other day she went to the corner store where we normally go to pick up alcohol and other drinks and the cashier (who never does this sort of thing) asked her if she had a boyfriend. She told him that she did and when he asked if it was "that guy you're always in here with" she told him that yes, that was me and I was her boyfriend. (Good work, btw girlfriend.) I would have preferred "Yeah he's my boyfriend you got a problem with that douchebag? His name is Vince and he will f*ckin' END you for this, dog", but she's not Lil' Kim and probably way too nice to even think to say something like that. Plus, she probably doesn't believe I could actually end anybody. Still, that should have shut him up right? He should be apologizing to her. Apologizing for the great disrespect that he has shown for the two of us and our relationship because we seem like nice people and are probably a really good couple. "Sorry," he should say, "I messed up," but of course he doesn't do that. No, he has something to say. He says "Oh...."
I heard that he was gay.
I heard that he was gay.
I heard that he was gay.
Seriously? That's your play, guy who I always talk to about sports? Really, guy who I once told to "start stocking condoms because I don't want another kid"? That's your move? Really? You sure about that, guy with a neck tattoo who is out of her league anyways?
(And I'm not gay. I watch professional wrestling! There's nothing gay about that, that's for sure!)
It sucks. I can't even trust my girlfriend to buy me a Vitamin Water without some jerkoff trying to hit on her, but the truth is that when it comes to your girlfriend you really can't trust anybody. How many movies are there about best friends in love with the same girl or the guy in love with his brother's fiance? How many vampires have to steal girls from werewolves before we learn that this stuff happens in real life? It happens all the time, every single day and there is only a few people in the world that you can trust.
I know that with my girlfriend I can trust the following people: Read the rest of this entry »


text:
You are visiting our UK site. Click here to view our US site
Glass Cup with Rubber - 65mm (CP08C)
Squeeze the rubber top to create a vacuum before placing on the body.
Easy to sterilise (detach the glass cup from rubber top before cleaning).
Size: 65mm diameter internal, 70mm diameter external.
Made in Germany
Weight (including packaging): 200g
Tax is not applicable to those living outside the European Economic Community. For more info click here.
99 in stock
Now: £11.50 (£13.80 inc. tax)
Customer Reviews
There are currently no reviews for this product. Be the first!
Review this product...Subscribe To CMT
Shopping Basket
Your basket is empty!
Change Language
Change Currency
Events Calendar
Auriculotherapy Seminars - The 2014 UK Weekend Workshops with Jim Chalmers
A brief history of Ear Acupuncture, Anatomy and Enervation of the Ear, Techniques to locate active points, Prioritisation of point selection, TCM integration... [more]
Master Tung Acupuncture with Philip Weeks (22nd/23rd January 2015)
Tung Acupuncture is well known for its quick and spontaneous results... [more]


text:
preston boardshort
by STEVEN ALAN
Having spent the past 20 years creating a mini fashion world grounded in classic American roots - fashioned with a twist - Steven Alan has quietly emerged an uncommon fashion pioneer. Upon opening his first retail store in 1994, Steven sold pieces from an eclectic group of emerging designers, becoming known for his keen eye. In 1999, Steven launched his first men's collection, including his now signature Reverse Seam Shirt, followed by a women's collection soon after. The collections build upon the same curatorial point of view using authentic fabrics and precise tailoring along with thoughtful details- resulting in easily wearable, refined casual looks.
Back To Product View All From This DesignerMen's Size Chart
Product Details
Our classic boardshort, offered in a starburst floral cotton print or a quick drying indigo cotton nylon blend. Fully lined polyester mesh, traditional lace-up drawstring tie closure and velcro fly.
• front coin pocket
• rear flap pocket with button closure and grommet for drainage
• signature logo patch on back waist
• 11" rise, 6.5" inseam
• starburst multi: 100% cotton
• indigo sulfide: 50% cotton, 50% nylon
• mesh lining: 100% polyester
• made in USA
PLEASE NOTE: all swimwear items are final sale


text:
Ethanol as government energy policy has been an economic and environmental bust. There's little debate: it inflates motor fuel prices, while compromising the environment. And that was before this summer's drought reducing corn and wheat supply. President Obama is under pressure from all sides to waive the ethanol requirement for less expensive fuel and food.
According to the U.S. Small Business Administration (SBA), a small business is one that employs fewer than 500 employees, a definition that applies to a wide swath of companies. In fact, the SBA says that small businesses account for 99.7 percent of employer firms, making them a significant sector of the economy.
Until last year, Social Security sent taxpayers an annual personalized statement showing a complete record of their taxable earnings as well as estimated retirement, disability and survivor benefits based on those earnings. Unfortunately, budget constraints brought put an end to those yearly mailings.
Basketball and bicycling rank highest for injuries among recreational sports, causing 1.5 million accidents per year.
Water ‑ and how to protect it in the face of worsening drought conditions ‑ is now a hot topic across America.
I saw an item on the nightly news recently about a fellow accused of trying to sell two bear cubs. He claimed he'd been attacked by the mama bear and had to kill her in self-defense. Rather than abandon the cubs to die, he had brought them into town. The story gets a little fuzzy from then on, but eventually the cubs were taken by Fish and Game personnel to a shelter that specializes in releasing critters back into the wild. The cubs were really cute little guys and I hope they make it in their transition back to ...
If your spouse-to-be is considering postponing the wedding because of fears about the so-called "marriage penalty," you two probably have bigger issues than whether you'll have to pay higher taxes as a married couple than when you were single.
California's Central Valley environment is getting healthier, but not fast enough. Its air quality is still among the worst in the nation, according to a report released this past week.
In California, the Affordable Care Act – the new health care law – helped 1,327,883 people with original Medicare get at least one preventive service at no cost to them during the first six months of 2012, Health and Human Services Secretary Kathleen Sebelius recently announced. This is 45 percent of those eligible for these services in California. This figure also includes 109,444 in California who have taken advantage of the Annual Wellness Visit provided by the Affordable Care Act.
Let's face it. Walking is a popular low-impact exercise. Now, new iPods are equipped with pedometers, half-marathons are packed with walkers and many apps can map and measure walking routes. But even with all of these new innovations, you'll still want to pay attention to basics.
Their names may sound funny but their financial consequences are not: "Phishing," "smishing," "vishing" and "pharming" are just a few of the ways criminals gain access to personal information via your computer or smartphone. If you're not careful, identity thieves can use harvested information to open fraudulent bank or credit card accounts, take out loans, rent apartments or even charge medical procedures to your insurance plan.
For many, the Fourth of July signals the start of summer and time for fun family outings and vacations, especially camping. However, with hot and dry conditions, summer is also the time for wildfires. Cal Fire would like to remind all campers and outdoor enthusiasts to keep these summer time safety tips in mind while enjoying the great outdoors.
The temperatures are rising, and so are the numbers of vehicles on the road. Summer is now officially here, bringing with it a number of road trips to the beach for many families.
A diagnosis of kidney failure could be a real shock.
Nearly 12 million adults were victims of identity fraud last year, up 13 percent from the previous year, according to Javelin Strategy & Research.


text: Put your wardrobe on top with tees from the NBC Universal T-Shirt Shop. Here you can shop tees from all the NBC networks you love, including Syfy and USA. Get the famous Swanson Pyramid of Greatness T-Shirt, unique T-shirts from the Office and even a Burton Guster tee from Psych. Whether youre looking for something to show youre a serious fan or the perfect comedic touch to add to your collection, youll find it in these fantastic NBC T-shirts.


text:
stadium in the storm
This photo was shot on film. Taken by mallorynox with a Lomography Diana F+ (available in our Online Shop) loaded with Kodak Ektachrome 100 (120) film in Přelouč, Czech Republic. These tags describe this photo: ektachrome, diana, xpro, and trip. Date: 2012-07-05. Time: Afternoon. This photo can be found in the album Diana - Trip to Prelouc and horses in Kladruby.Please login to get more out of this site
You are currently not logged in, do so now to add comments, like articles as well as photos, submit to competitions, translate articles and gain access to unlimited photo upload! If you are not registered yet, go here to find out more about Lomography and join now! If you just want a limited account for now you can use Twitter or Facebook. We would love to have you as a member here!
2 people like this photo
Liking photos helps us to present the best ones to you and others. 2 people like this photo. 2 people like this photo: rik041 and aton.


text:
The cookout will follow the Athletic Department's "Trunk or Treat," an event where goods will be given out to trick-or-treaters from the trunks of vehicles.
"It will be a free event — open to all," said Chamber Executive Director Judson Thigpen. "We really want to encourage people to come to this event. People can bring lawn chairs and picnic blankets. We will have tables as well. We will also have games for the kids, which the theater department is helping out with."
Food will include hamburgers and hotdogs and beer will also be available for purchase.
The bands, DMI All-Stars and 1964: The Tribute, will provide musical entertainment for the evening.
A pep rally will follow the cookout at 9 p.m. and that will be followed by a 10-minute firework show.
Delta State faculty, staff, students, and alumni are invited to attend LaForge's inauguration, which will be at 10:30 a.m. on Friday at the Bologna Performing Arts Center in Cleveland.
For more information on the cookout, contact the Cleveland-Bolivar Chamber of Commerce at (662) 843-2712.