text:
This morning, I was thinking about Apple. When I got started in this industry in the early 80s, it was on an Apple II+ writing first in BASIC and later in UCSD Pascal. I thought Apple was simply amazing, so it was tough watching the more than decade of decline before Jobs rejoined. Our industry has no memory and most great companies fade away. Sometimes this happens surprisingly quickly once the process has begun. Tandem, still one of the best database machines ever made, was eaten by DEC. DEC was eaten by Compaq. Compaq was eaten by HP. IBM took over the personal computer leadership from Apple and subsequently also fell from the top through a long period of decline. IBM eventually sold the client computer business to Lenovo and IBM continues to struggle to find growth with 20 consecutive quarters of declining revenues. Google and Amazon didn't yet exist back then, and Oracle was just getting started.
Apple has done the impossible of hitting the top, losing touch with their customer base, and declining to near irrelevance before finding their way back to the top. Apple's stock price has been high for years but the stock market gets excited (or depressed) in ways that are often disconnected from the success the company is actually achieving. To see how Apple was really doing, I looked up the world's top 50 revenue producers to see where they stood: List of Largest Companies by Revenue.
Four observations jumped out while scanning the list. Of course the first is that Apple is at #9 on this list after having had a brush with corporate death. They now bring in more revenue than all companies on the planet except 1 retail, 1 electric utility, 4 oil and gas, and 2 car companies. Incredible and it just about never happens that a company comes back after having nearly faded away.
The second is that Walmart is still bringing in more revenue than any other company period. $482B. That's ½ trillion dollars. Looking at the Gross Domestic Product of entire countries (all the finished goods and services produced by the country in a year), Walmart revenue when compared to country GDP would place just slightly behind Poland and well ahead of Belgium. A truly staggering number.
The third observation from reading the top 50 corporate revenue list is that most of the exciting companies you read about aren't even on the list and, ironically, most of their competitors are. Tesla is perhaps the best example. Tesla is in the press daily and there is no question they are having a massive impact on the automotive world. Tesla brings in a respectable $7B annually while 7 of the world's automotive companies bring in more than $100B. Revenue is obviously a trailing indicator. I'm not convinced the stock price is much better but it's amazing, at least to me, how much revenue is being brought in by companies that really haven't been innovating for quite some time.
A fourth observation is the massive amount of money spent on automobiles and related technology. 14 of the top 50 revenue producing companies in the world are from the automotive and oil and gas sectors. An extension of that observation is selling to consumers in general is always a huge opportunity. Even small profits times a very large multiplier can get impressively large.
Apple is a real outlier in many respects. One of my greatest regrets in my professional life is I never got a chance to meet Steve Jobs.]]>
For years I've been saying that, as more and more workloads migrate to the cloud, the mass concentration of similar workloads make hardware acceleration a requirement rather than an interesting option. When twenty servers are working on a given task, it makes absolutely no sense to do specialized hardware acceleration. When one thousand servers are working on the task, it certainly makes sense to do custom boards and/or Field Programmable Gate Arrays (FPGAs). But one thousand is actually a fairly small number in the cloud. When there might several hundred thousand servers all running the same workload, hardware specialization goes from an interesting idea to almost a responsibility. Hardware acceleration can reduce latency by a factor of ten, decrease costs by a factor of ten and, it's better for the environment, with power/performance improved by at least a factor of ten.
In the 80s I argued that specialized hardware was crazy and the future of our industry was innovative software on low-cost, commodity, general-purpose processors. For many decades that certainly appeared to be true and I've managed to make a respectable living on that basic approach: improving performance, availability, and lowering costs by using large numbers of commodity processors. If I ever was correct on this point, it's certainly not true any longer. We are entering the era of hardware acceleration.
In actuality hardware acceleration has been around in large numbers for a considerable length of time. Commercial routers have massive Application Specific Integrated Circuits (ASIC) at the core. Many specialized network appliances have workload specialize hardware doing the heavy lifting. Most network Interface cards have ASICs at the core. The most effective BitCoin mining engines are using hardware workload acceleration in custom ASICs.
It really wasn't that long ago that that X86 processors didn't include floating point on die and it was either done in software on in a separate floating point co-processor. In the somewhat more distant past, I've worked on processors that even lacked a fixed point multiply instruction. In the early days of high-performance computing, matrix work was done in software. Cray moved it into hardware with the Cray 1 vector units and, as the cost of a transistor continues to plummet, even hardware vector units are now standard fare in a modern X86 processor.
Hardware acceleration isn't new, but the operations being accelerated are moving up from the incredibly primitive to replacing ever larger hot kernels of higher-level applications. In the early days, adding the MultiplyAdd (also called Fused-Multiply Add or fmadd) instruction to IBM POWER was a reasonably big deal in allowing the two operations to be done as a single instruction rather than as two. Over time, more hardware accelerations like cryptography have been creeping into general purpose processors and I expect this process will not just continue but pick up pace. General purpose processors have the die real estate to spare and, even if the accelerators are used rarely, with 10x gains across many dimensions, hardware acceleration makes excellent economic sense. In fact, as Moore's law slows, higher-level hardware acceleration will become one of the most important ways that the next processor generation shows material advantage over the previous.
While general purpose processors will continue to get more higher–level accelerations, another trend emerging much more broadly over the last ten years is the offloading of important workloads from the general-purpose processor entirely. Where a workload has both high value and massive parallelism, it's a candidate for migration off of general purpose processors and onto graphics processors. It turns out the massive parallelism required for advanced computer graphics also supports some non-graphical workloads incredibly well. Some of the first examples I came across was the offloading of hot financial calculations and the acceleration of seismic studies used in oil exploration. These were important, but the golden workload that has absolutely exploded the general purpose graphics processor market is machine learning. Training neural networks is a highly parallel task that runs incredibly well on graphics processors. Five years ago there was lots of talk about this and some were doing it. Today, it's hard to justify not running these workloads on General Purpose Graphics Processor Units (GPGPUs) once the workload is being run at scale.
As an example, the Nvidia Tesla K80 GPGPU board has 2 sockets with 2496 CUDA cores each and the overall system is capable of 8.74 TFLOB. The individual cores aren't that powerful nor amazingly fast but, if the workload is highly parallel, there are a lot of cores available to host it. This part is a bit power intensive at 300W TDP (thermal design power) but that's not really a problem. What's both a massive problem and proof of how well this systems supports some workloads is the price. The retail price of the K80 when announced was $5,000. This board is just about as close as our industry gets to pure profit margin with a complete disregard to cost so large discounts are available. But, regardless of discount, this board will never be confused for a low-cost or commodity part.
The good news is our industry is self-correcting and there are many new solutions under development. In fact, partly because the price of this part is so crazy high and partly because cloud volumes are now large enough to justify customer processors, we are going to see more and more workloads hosted on custom ASICs. 15% of the 2016 ISCA papers were on machine learning hardware accelerators and Google, Microsoft, and Amazon all have digital design skills. Not much has been written about the semiconductor work done by the mega-providers but, as an example, at AWS we deploy many hundreds of thousands of custom ASICS each year.
I just read about another excellent example higher-level application acceleration. In fact this best example I've seen publicly disclosed so far. The paper "In-Datacenter Performance Analysis of a Tensor Processor Unit" will be presented at the upcoming 44th International Symposium on Computer Architecture (ISCA) to be held in Toronto Canada June 26, 2017.
In my opinion, this is excellent work, a well-written paper, and a balanced analysis of what they produced and started to deploy back in 2015. It's the normal practice in our industry to only show that which has already been replaced or is about to be replaced but that's just the reality of commercial innovation and I do the same thing myself.
What I found most striking is the elegant simplicity of what has been done. It wins over general purpose Intel processors and Nvidia GPGPUs of the same generation by the greater than 10x we would expect and yet, they have kept the part simple and shown good taste in what to include and what not to. The paper uses power/performance as a proxy for the price/performance they know they should be using but since this is commercial innovation, pricing needs to remains confidential.
Because the part and board have been in production since 2015, they will have likely done more than 10^6 volume by now. Since the volume is good by semiconductor standards and the part is not that complex, I would speculate that the part is less than $50 and the full PCIe board will cost under $100. For machine learning inference, this part is more than an order of magnitude faster than an Nvidia GPGPU while being more than an order of magnitude less expensive. This is the power of workload hardware specialization and we are going to see a lot more of this over the next decade.
Some key speeds and feeds from the paper:
Some interesting observations and lessons learned from the paper in bold with my short form notes in italics:
This is really fine work by Norman Jouppi, Cliff Young, Nishant Patil, the always excellent David Patterson, and others. If you can find the time, go ISCA2017 in Toronto and, for sure, read the paper: In-Datacenter Performance Analysis of a Tensor Processing Unit.
Last week Fortune asked Mark Hurd, Oracle co-CEO, how Oracle was going to compete in cloud computing when their capital spending came in at $1.7B whereas the aggregate spending of the three cloud players was $31B. Essentially the question was, if you assume the big three are spending roughly equally, how can $1.7B compete with more than $10B when it comes to serving customers? It's a pretty good question and Mark's answer was an interesting one "If I have two-times faster computers, I don't need as many data centers. If I can speed up the database, maybe I need one fourth as may data centers."
Of course, I don't believe that Oracle has, or will ever get, servers 2x faster than the big three cloud providers. I also would argue that "speeding up the database" isn't something Oracle is uniquely positioned to offer. All major cloud providers have deep database investments but, ignoring that, extraordinary database performance won't change most of the factors that force successful cloud providers to offer a large multi-national data center footprint to serve the world. Still, Hurd's offhand comment raises the interesting question of how many data centers will be required by successful international cloud service providers.
I'll argue the number is considerably bigger than that deployed by even the largest providers today. Yes this represents massive cost given that even a medium sized data center will likely exceed $200m. All the providers are very focused on cost and none want to open the massive number of facilities I predict, so let's look deeper at the myriad of drivers for large data center counts.
*N+1 Redundancy: The most efficient number of data centers per region is one. There are some scaling gains in having a single, very large facility. But one facility will have some very serious and difficult-to-avoid full-facility fault modes like flood and, to a lesser extent, fire. It's absolutely necessary to have two independent facilities per region and it's actually much more efficient and easy to manage with three. 2+1 redundancy is cheaper than 1+1 and, when there are 3 facilities, a single facility can experience a fault without eliminating all redundancy from the system. Consequently, whenever AWS goes into a new region, it's usual that three new facilities be opened rather than just one with some racks on different power domains.
*Too Big to Fail: Even when building three new data centers when opening up a new region, there are some very good reasons to have more than three data centers as a region grows. There is some absolute data center size where the facility becomes "too big to fail." This line is gray and open to debate but the limiting factor is how big of a facility can an operator lose before the lost resources and the massive network access pattern changes on failure can't be hidden from customers. AWS can easily build 100-megawatt facilities, but the cost savings from scaling a single facility without bound are logarithmic, whereas the negative impact of blast radius is linear. When facing seriously sub-linear gains for linear risk, it makes sense to cap the maximum facility size. Over time this cap may change as technology evolves but AWS currently elects to build right around 32MW. If we instead built to 100MW and just pocketed the slight gains, it's unlikely anyone would notice. But there is a slim chance of full-facility fault, so we elect to limit the blast radius in our current builds to around 32MW.
These groupings of multiple data centers in a redundancy group are often referred to as a region. As the region scales, to avoid allowing any of the facilities that make up the region to become too big to fail, the number of data centers can easily escalate to far beyond ten. AWS already has regions scaled far beyond 10 data centers.
What factors drive a large scale operator to offer more than a single region and how big might this number of regions get for successful international operators? Clearly the most efficient number of regions is one covering the entire planet just as one is the most efficient number of data centers if other factors are ignored. There are some significant scaling cost gains that can be achieved by only deploying a single region.
*Blast Radius: Just as we discovered that a single facility eventually gets too big to fail, the same thing happens with a very large, mega-region. If an operator were to concentrate their world-wide capacity in a single region it would quickly become too big to fail.
I'm proud to say that AWS hasn't had a regional failure in recent history but the industry continues to see them rarely. They have never been common but they still are within the realm of possibility, so a single region deployment model doesn't seem ideal for customers. The mega region would also suffer from decaying economics where, just as was the case in the single large data center, the gains from scaling become ever smaller while the downside risks continue to climb. Eventually the incremental cost reductions of scaling the region become quite small while the downside risk continues to escalate.
The mega-region downside risks can be at least partially mitigated by essentially dividing the region up into smaller independent regions but this increases costs and further decreases the scaling gains. Eventually it just make better sense to offer customers alternative regions rather than attempting to scale a single region and the argument in favor of multiple regions become even stronger when other factors are considered.
*Latency and the Speed of Light: The speed of light remains hard to exceed and the round trip time just across North America is nearly 100 ms (Why are there data centers in NY, Hong Kong, and Tokyo). Low latency is a very important success factor in many industries so, for latency reasons alone, the world will not be well served by a single data center or a single region.
Actually it turns out that the speed of light in fiber is about 30% less than the speed of light in other media so it actually is possible to run faster (Communicating data beyond the speed of light). But, without a more fundamental solution to the speed of light problem, many regions are the only practical way to effectively serve the entire planet for many workloads.
There are many factors beyond latency that will push cloud providers to offer a large number of regions and I'm going to argue that latency is not the prime driver of very large numbers of regions. If latency was the only driver the number of required regions would likely be in the 30 to 100 range. Akamai, the world leading Content Distribution Network (CDN), reports more than 1,500 PoPs (Points of Presence) but many experts see them 10x bigger than would be strictly required by latency. Another major CDN, Limelight, reports more than 80 PoPs. This number is closer to the one I would come up with for the number of PoPs required if latency was the only concern. However, latency isn't the only concern and the upward pressure from other factors appears to dominate latency.
*Networking Ecosystem Inefficiencies: The world telecom market is a bit of a mess with many regions being served by state sponsored agents, monopolies, or a small number of providers that, for a variety of reasons, don't compete efficiently. Many regions are underserved by providers that have trouble with the capital investment to roll out the needed capacity. Some providers lack the technical ability to roll out capacity at the needed rate. All these factors conspire to produce more than an order of magnitude difference in cost between the (sort of) competitive US market and some other important world-wide markets.
Imagine a $20,000 car in one market costing far more than $200,000 in another market. That's where we are in the network transit world. This is one of the reasons why all the major cloud providers have private world-wide networks. This is a sensible step and certainly does help but it doesn't fully address the market inefficiencies around last-mile networks. Most users are only served by a single access network and these last-mile network providers often can't or don't own the interconnection networks that link different access networks together. Each access network must be reached by all cloud providers and each of these access networks themselves face a challenge with sometimes unreasonable interconnection fees that increase their costs, especially for video content.
Netflix took an interesting approach to the access network cost problem. Their approach helps Netflix customers and, at the same time, helps access networks serve customers better. Netflix offers to place caching servers (essentially Netflix-specific CDN nodes) in the central offices of access networks. This allows the access network to avoid having to pay the cost to their transit providers to move the bits required to serve their Netflix customers. This also gives the customers of these access networks a potentially higher quality of service (for Netflix content). A further advantage for Netflix is in reducing the Netflix dependence on the large transit providers, it reduces the control these transit providers have over Netflix and Netflix customers. This was a brilliant move and it's another data point on how many points of presence might be required to serve the world. Netflix reports they have close to 1,000 separate locations around the world.
*Social and Political Factors: We have seen good reason to have order 10^3 regions to deliver the latency required by the most demanding customers. We have also looked at economic anomalies in networking costs requiring O(10^3) regions to fully serve the world economically. What we haven't talked about yet are the potentially more important social and political factors. Some cloud computing users really want to serve their customers from local data centers and this will impact their cloud provider choices. In addition, some national jurisdictions will put in place legal restrictions than make it difficult to fully serve the market without a local region. Even within a single nation, there will sometimes be local government restrictions that won't allow certain types of data to be housed outside of their jurisdiction. Even within the same country won't meet the needs of all customers and political bodies. These social and political drivers again require O(10^3) points of presence and perhaps that many full regions.
As the percentage of servers-side computing hosted in the cloud swings closer to 100%, the above factors will cause the largest of the international cloud providers to have between several hundred to as many as a thousand regions. Each region will require at least three data centers and the largest will run tens of independent facilities. Taking both the number of regions and the number of data centers required in each of these regions into account argues the total data center count of the world largest cloud operators will rise from the current O(10^2) to O(10^5).
It may be the case that there will be many regional cloud providers rather than a small group of international providers. I can see arguments and factors supporting both outcomes but, whatever the outcome, the number of world-wide cloud data centers will far exceed O(10^5) and these will be medium to large data centers. When a competitor argues that fast computers or databases will save them from this outcome, don't believe it.
Oracle is hardly unique in having their own semiconductor team. Amazon does custom ASICs, Google acquired an ARM team and has done custom ASIC for machine learning. Microsoft has done significant work with FPGAs and is also an ARM licensee. All the big players have major custom hardware investments underway and some are even doing custom ASICs. It's hard to call which company is delivering the most customer value from these investments, but it certainly doesn't look like Oracle is ahead.
We will all work hard to eliminate every penny of unneeded infrastructure investment, but there will be no escaping the massive data center counts outlined here nor the billions these deployments will cost. There is no short cut and the only way to achieve excellent world-wide cloud services is to deploy at massive scale.]]>
Airline overbooking isn't evil. In fact, if done properly, it's good for airlines, good for customers, and good for the environment. Sold seats are clearly good for airlines and their shareholders. High utilization is good for customers because it reduces seat costs for airlines which normally operate at single digit profit margins. In the consumer airline industry, customers typically do enjoy the gains from increased operator efficiency. Full seats are good for the environment since the fuel burned and the emissions produced are less per airline traveler. By overbooking "done properly" I mean there have to be controls in the system to ensure customers get what they bought and that airlines make decisions that are in the best interests of the environment, customers, and their shareholders.
The airline seat overbooking system as currently operating in the United States puts all the control in the hands of the airline and they get most of the gain rather than customers. United Airlines, in what appears to have been a truly exceptional entry into the corporate Darwin Awards, has been heavily covered in recent news for forcibly dragging a paying customer off a plane. It's hard to read the news and not see numerous mentions of this event but here's a good example: United Airlines Offers Refunds as Outrage at a Violent Removal Continues. In this event a 69-year-old physician was forcibly removed from the United Airlines flight. The customer was injured, will take legal action, and even though the customer is just about guaranteed to win a large settlement, it'll be dwarfed by the negative publicity of this event on the airline.
Digging deeper into the United removal of this paying passenger, the details are actually a bit worse than the explanation that the plane was over-booked. United reports the plane wasn't overbooked but they needed to transport crew so they elected to forcibly remove paying passengers. After the event, United Chief Executive Office, Munoz, is reported to have said in an internal memo that the passenger, a physician was "belligerent" (United Airlines says controversial flight was not overbooked; CEO apologizes again). The CEO has since reversed this statement and now says "no one should ever be mistreated this way."
Clearly United has serious issues starting at the very top of the company that won't likely be solved without management changes but that's not my focus here. What I'm arguing is that commercial airline seat overbooking isn't actually a bad way to manage expensive resources and it needn't be customer unfriendly. I'm going to argue that lawmakers need to make changes to make the overbooking system work properly, but the natural inclination to ban overbooking because of the clumsy handling of this event by United isn't necessarily the best approach.
Flying commercial aircraft is expensive and whenever an industry needs to sell a capital-intensive resource, utilization becomes one of the most important factors to the long term success of the industry. I work in cloud computing where data centers run several hundred million dollars each and the most successful providers are spending billions on capital expenses each year. Every basis point is huge and a few percentage points difference in utilization could be the difference between excellent profit margins or operating negative. Airlines are one of the prototypical cases of capital-intensive industries.
Airlines make their money moving people and cargo. The cargo hold and the cabin have fixed divisions but the overall plane is weight-limited, so the cargo sales and passenger sales are not actually fully independent. However, selling all the passenger seats is the goal of every passenger-carrying airline and successful airlines operate at very high seat-utilization levels. This is done using a combination of yield management and overbooking. Yield management is a pricing strategy based upon computer models of customer behavior on specific routes. All successful airlines do it and the goal is to sell all seats and to sell them at the highest possible profit.
These algorithmic pricing systems do a remarkably good job of selling all the seats and getting competitive prices from every seat sold. However there are latencies in the sales channel and other issues that prevent yield management from solving the problem entirely and, even if they did, some customers simply don't show up. Since an empty seat represents a substantial source of lost revenue, no airline wants to fly with an empty seat. The next tactic after yield management to ensure the maximum possible load of paying customers is to oversell seats. In this approach, the airline sells more seats than they have available knowing the statistical likelihood of customers not showing up.
Overselling sounds customer-unfriendly – why allow an airline to sell a seat that it doesn't actually have available? It's a great question and the United incident mentioned above clearly underlines that poor leadership certainly can yield very poor results but that's a management constant rather than specific to overbooking. But, overbooking as currently practiced, does have problems.
What's broken here is the airlines are allowed to simply fail to honor their commitment and forcibly remove a paying passenger from the airplane. The system would work fine if the airline had to pay passengers to accept alternative transport and if there were no bounds on what this might cost. This puts an economic safeguard on the airline to only sell that which they are highly likely to have. If they don't have sufficient seats, they are obligated to offer alternative transport on other airlines, pay customers whatever it takes to get off a plane or, in the worst case, charter transport for the remaining passengers. Certainly there is always a price that would get me off a plane. Recently the US President said the same (David Dao, Doctor Dragged Off Plane, Files Court Papers Demanding Airlines Preserve Evidence). I suspect he is a bit more expensive than I am, but it turns out that in the vast majority of times, a few hundred dollars will open up the needed seats. Sometimes it might cost more but that's the cost of over selling
What about the customer that don't show up? Doesn't there have to be some factor to prevent them from booking seats and not showing up since they are adding considerable risk to the airline? I agree. Customers should have to pay a no-show penalty. It's a competitive market and the airline should set it to whatever they believe covers their overbooking costs or encourages the desired behavior. Some airlines might chose to set this "no show" cost at free for some tickets, some passengers, or even all passengers. But it would be the airline's choice to set the cost of a no show. Currently they mostly don't charge now but, again, that's their choice.
In this proposed model airlines have to move all paying passengers or make arrangements with some of the passengers to accept compensation for different travel arrangements. They are free to charge no-show passengers a fee if they so choose to limit the airline passenger no-show risk. In this model, there is economic pressure on the airline to fly full but to also satisfy the needs of their customers. There is also economic pressure on passengers to show up for the flights they book. It's a good model that helps airlines and travelers. Airlines can overbook and customers can no-show but both have to pay the cost of that action. The current model where an airline is not obligated to fulfill their commitment on selling a ticket by either transporting the customer or offering them compensation acceptable to the customer doesn't work, and the United handling of this most recently situation really underlines that failing.
Airlines seat overbooking isn't evil but allowing airlines to cap the costs and be able to just refuse a serve a paying customer clearly doesn't work.
Update: On the topic of customer centricity, Jeff Bezos, annual letter to shareholders is an excellent read: 2016 Amazon Letter to Shareholders.
Update: Some of the right things are starting to happen in the airline industry:
From: LA Times
From: Washington Post
I'm a connoisseur of failure. I love reading about engineering failures of all forms and, unsurprisingly, I'm particularly interested in data center faults. It's not that I delight in engineering failures. My interest is driven by believing that the more faults we all understand, the more likely we can engineer systems that don't suffer from these weaknesses.
It's interesting, at least to me, that even fairly poorly-engineered data centers don't fail all that frequently and really well-executed facilities might go many years between problems. So why am I so interested in understanding the cause of faults even in facilities where I'm not directly involved? Two big reasons: 1) the negative impact of a fault is disproportionately large and avoiding just one failure could save millions of dollars and 2) at extraordinary scale, even very rare faults can happen more frequently.
Today's example is from a major US airline last summer and it is a great example of "rare events happen dangerously frequently at scale." I'm willing to bet this large airline has never before seen this particular fault under discussion and yet, operating at much higher scale, I've personally encountered it twice in my working life. This example is a good one because the negative impact is high, the fault mode is well-understood, and although a relatively rare event there are multiple public examples of this failure mode.
Before getting into the details of what went wrong, let's look at the impact of this failure on customers and the business. In this case, 1,000 flights were canceled on the day of the event but the negative impact continued for two more days with 775 flights canceled the next day and 90 on the third day. The Chief Financial Office reported that $100m of revenue or roughly 2% of the airline's world-wide monthly revenue was lost in the fall-out of this event. It's more difficult to measure the negative impact on brand and customer future travel planning, but presumably there would have been impact on these dimensions as well.
It's rare that the negative impact of a data center failure will be published, but the magnitude of this particular fault isn't surprising. Successful companies are automated and, when a systems failure brings them down, the revenue impact can be massive.
What happened? The report was "switch gear failed and locked out reserve generators." To understand the fault, it's best to understand what the switch gear normally does and how faults are handled and then dig deeper into what went wrong in this case.
In normal operation the utility power feeding a data center flows in from the mid-voltage transformers through the switch gear and then to the uninterruptible power supplies which eventually feeds the critical load (servers, storage, and networking equipment). In normal operation, the switch gear is just monitoring power quality.
If the utility power goes outside of acceptable quality parameters or simply fails, the switch gear waits a few seconds since, in the vast majority of the cases, the power will return before further action needs to be taken. If the power does not return after a predetermined number of seconds (usually less than 10), the switch gear will signal the backup generators to start. The generators start, run up to operating RPM, and are usually given a very short period to stabilize. Once the generator power is within acceptable parameters, the load is switched to the generator. During the few seconds required to switch to generator power, the UPS has been holding the critical load and the switch to generators is transparent. When the utility power returns and is stable, the load is switched back to utility and the generators are brought back down.
The utility failure sequence described above happens correctly almost every time. In fact, it occurs exactly as designed so frequently that most facilities will never see the fault mode we are looking at today. The rare failure mode that can cost $100m looks like this: when the utility power fails, the switch gear detects a voltage anomaly sufficiently large to indicate a high probability of a ground fault within the data center. A generator brought online into a direct short could be damaged. With expensive equipment possibly at risk, the switch gear locks out the generator. Five to ten minutes after that decision, the UPS will discharge and row after row of servers will start blinking out.
This same fault mode caused the 34-minute outage at the 2012 super bowl: The Power Failure Seen Around the World.
Backup generators run around 3/4 of million dollars so I understand the switch gear engineering decision to lockout and protect an expensive component. And, while I suspect that some customers would want it that way, I've never worked for one of those customers and the airline hit by this fault last summer certainly isn't one of them either.
There are likely many possible causes of a power anomaly of sufficient magnitude to cause switch gear lockout, but the two events I've been involved with were both caused by cars colliding with aluminum street light poles that subsequently fell across two phases of the utility power. Effectively an excellent conductor landed across two phases of a high voltage utility feed.
One of two times this happened, I was within driving distance of the data center and everyone I was with was getting massive numbers of alerts warning of a discharging UPS. We sped to the ailing facility and arrived just as servers were starting to go down as the UPSs discharged. With the help of the switch gear manufacturer and going through the event logs, we were able to determine what happened. What surprised me is the switch gear manufacturer was unwilling to make the change to eliminate this lockout condition even if we were willing to accept all equipment damage that resulted from that decision.
What happens if the generator is brought into the load rather than locking out? In the vast majority of the situations and in 100% those I've looked at, the fault is outside of the building and so the lockout has no value. If there was a ground fault in the facility, the impacted branch circuit breaker would open and the rest of the facility would continue to operate on generator and the servers downstream of the open breaker would switch to secondary power and also continue to operate normally. No customer impact. If the fault was much higher in the power distribution system and without breaker protection or the breaker failed to open, I suspect a generator might take damage but I would rather put just under $1m at risk than be guaranteed that the load will be dropped. If just one customer could lose $100m, saving the generator just doesn't feel like the right priority.
I'm lucky enough to work at a high-scale operator where custom engineering to avoid even a rare fault still makes excellent economic sense so we solved this particular fault mode some years back. In our approach, we implemented custom control firmware such that we can continue to multi-source industry switch gear but it is our firmware that makes the load transfer decisions and, consequently, we don't lockout.]]>
Laying fiber optic cables with repeaters along the ocean floor raises super-interesting technical challenges. I recently visited the CS Responder, a trans-ocean cable-laying ship operated by TE Connectivity.
TE Connectivity is $13.3B global technology company that specializes in communication cable, connectors, sensors, and electronic components. Their subsidiary TE SubCom manufactures, lays and maintains undersea cable. TE SubCom has a base in Baltimore and invited us to tour the facility and one of their cable-laying ships. The visit included an overview of their cable-laying business and solutions, demonstrations on how undersea cables are made, and best of all, a tour of the CS Responder.
In the first picture below our guide on the tour, Jim Herron, is on the left and our host David Robles on the right. Jim started the day with an overview of the TE SubCom cable-laying business and the many challenges faced in the process, ranging from permitting to adverse weather to the job of actually laying the cable itself. A sister ship to the CS Responder, the Tyco Resolute (now CS Resolute) was featured on Discovery Channel Canada's Mighty Ships documentary series. This episode, although somewhat dramatized, shows many of the challenges that cable-layers face and how they are managed. The Smithsonian Channel airs the series in the US. A trailer is on the Smithsonian Channel web site and the full episode is available on Amazon Video.
Our first stop was the shore-based splicing training school. Brian Isenstadt showed us how splicers are trained to reliably connect tiny fiber optic links where the links are low-loss and sealed from water well enough to last 20+ years. The second photo below shows the stainless steel cartridge that maintains the mechanical strength of the cable and provides space for the up to eight individually-spliced tiny fiber-optic pairs.
The first picture below shows the instrument that first aligns the optics and then fuses them together into a single unbroken fiber optic path. The screen in the second picture shows the two fiber ends being brought together into alignment to ensure a low signal-loss connection.
The stainless steel splice cartridge is next permanently encased in a waterproof thermally-injected sealant. In the second picture below, the sealing material is inspected carefully for impurities, bubbles, or other flaws using X-ray.
Repeaters are one of the biggest challenges to undersea cable longevity. High quality optical signals will only propagate for 60 to 80 km within high signal-to-noise standards. So, to cross an entire ocean, repeaters have to be used to boost the signal strength. These repeaters have to be able to operate reliably for 20+ years at depths beyond 3 miles. And they have to be strong enough to make it highly unlikely the cable will be torn apart at the repeater by weather, fishing activity, or other disturbances.
These repeaters are electrically powered with direct current (DC) and the power distribution system used is super interesting. Each repeater is in series like an old-fashioned Christmas tree light string. This allows only a single conductor to be used to power all repeaters, greatly decreasing the cost of the cable. The downside of single-conductor DC powering is that shore side Power Feed Equipment (PFE) has to provide sufficient voltage to drive the potentially multi-thousand mile cable and all the cable repeaters along the ocean floor. To achieve the needed levels of redundancy, the PFE on one cable end supplies positive voltage and the PFE on the other ends supplies a roughly equal negative voltage. The combined difference of potential or voltage could be as high as 10,000 volts, although each repeater only sees a tiny portion of that difference of potential since they are all wired in series.
The two most common undersea cable failure modes are 1) PFE failure and 2) undersea cable power shunts to ground. Both failures have similar recovery mechanisms. If a PFE fails, the PFE on the other end doubles its voltage level to maintain the same difference of potential at each repeater allowing the cable to continue to operate without interruption until the PFE can be repaired. In the second fault mode, there is an insulation failure in the cable perhaps caused by fishing activity or weather action. This will allow the voltage at the insulator failure to drop to the ground plane neutral of the sea water. The recovery mode for power shunts to ground is for each PFE to adjust the inserted voltage such that the PFE on each end powers all repeaters between that PFE and the shunt-to-ground fault. In this operating mode, the cable will again continue to operate correctly even with the ground fault somewhere along the cable length. And, as long as the shunt fault is repaired before there is a second fault of either type, the cable will continue to operate without negative customer impact.
Below are cutaway repeaters showing their construction.
We toured the shore-side storage facilities on our way to the Responder. Left below are customer-owned fiber-optic cables in large metal cylinders used for storage and transport. On the right are large hydraulic cylinders needed at sea for the cable-laying process.
The Responder's two captains (two shifts), Daniel Kenealy and Chris Gabrielson, took us through the ship starting with the bridge. Below Capt. Kenealy is explaining each bridge station and their use. Specialized stations for different ship operating modes are located throughout the bridge. When steaming underway, the central bridge stations are used to operate the ship. During docking the ship can be maneuvered by the captain directly via the wing control station at the far left and right side of the bridge. Kenealy explained he prefers instead to get to a location with a better view and gently ease up to the dock via radio commands given to a helmsman.
From the aft end of the bridge, an excellent view can be had of the working deck over which cable is laid, the ROV (remote operated vehicle) is launched and recovered, and the cable-burying sea plow is deployed and recovered.
This ship is DP2-capable, meaning it can maintain its position with high degrees of precision in rough seas and with redundancy. The dynamic positioning system's computer will apply thruster or propeller force to hold the boat in an exact GPS-sensed location. This technology is commonly used in the oil exploration business where it is often necessary to keep a boat in an exact position in bad weather and rough seas where any deviation can do substantial damage or even put workers at risk. Because this is a life-critical system and potentially with high-value equipment at risk, it must operate without exception through computer faults or equipment failures.
The DP2 system on the Responder primarily is used to hold position against wind and current in order to place the cable precisely as the ship slowly inches forward laying cable at speeds that can sometimes sink below 1 kt and seldom are far above 3 kts. The dynamic positioning system also is used for remote-operated-vehicle submarine operations and when doing some non-cable related contracts where the ship is near oil rigs or related-equipment.
A row of cable repeaters is in the first picture below. The subsequent picture is new rope used for raising and lowering cable.
The first picture below is looking across the three massive cable storage areas in the Responder. If you look carefully, you can see a cable spooling out on the left from the rear storage area and heading out across the two forward in-ship storage areas. The crew was in the process of delivering some cable moved from their Bermuda storage facility to the Baltimore depot. The second picture is looking down into the rearmost of the three cable storage areas on the boat. The cable being offloaded is visible at the top of the photo, feeding up from the bottom of the storage area where it is neatly spiraled (click any photo for a larger view).
Below are pictures of the remote operated vehicle (ROV) used on the Responder and the control center where operators pilot the ROV. This vehicle is used to inspect cable lay positions, move cables to safer locations, and for non-cable related contracts. Sealing the electronics used in the ROV against the seawater outside is a real challenge. This system runs the electronics in an oil bath sealed against seawater intrusion.
When cables need to be branched or any repair needing splicing, the cable may need to be cut, captured and lifted onto deck for service using a cutting grapnel.
The following two pictures are of the massive EB Sea Stallion sea plow that is dragged behind the ship when burying cable. This beast can bury cable up to 3 meters by dragging a plow section that cuts through the sea bottom as it pulls forward and the cable is fed in through the blade to the bottom of the trench. The sea plow needs to be precisely controlled and dragged at just the right speed to allow complete burial depending upon sea bottom topography. Considerable skill is required to maintain just the right tension on the plow and the communications cable feeding into it.
The control room for the EB Sea Stallion sea plow is in a shipping container mounted on deck. The plow is visible towards the stern of the ship on the left of the second picture below.
The next three photos show the cable management gear towards the stern of the ship, aft of the three cable storage areas. The cable itself is fed out through a row of rubber tires that grab the cable through friction. Each of these rubber tires is powered by a large electric motor that allows the cable to feed out smoothly at exactly the speed needed without putting the cable under excess load, creating heat through friction, or putting unnecessary mechanical forces that could damage the cable.
Before getting into the details, most industries look far simpler than they really are. In this case, how complex can it be to run a fiber optic cable across a body of water? But the deeper you dig, the more the challenges mount.
When running undersea cables, the ends are almost always in different countries, each having their own permitting process and regulations. Cables need to be as light and inexpensive as possible but, when not protected by extreme depths, they need to armored against damage from anchoring, trawling, and severe weather events. When near shore, cables need to be buried in the sea bottom as deep as 3m to further minimize the damage risk.
Signal integrity can only be maintained by repeaters every 60 to 80 km. These repeaters need to be electrically powered and they need to continue to operate through both power failures and cable insulator faults. Repeaters need to operate reliably for 20+ years even when operating more than 3 miles below the sea surface. Cables need to be spliced together without signal loss while maintaining both mechanical strength and resistance to water intrusion.
TE SubCom controls all the technology from the cable design and manufacture, the repeater design, all the way through to the cable laying and service ships, allowing TE Connectivity to deliver multi-decade solutions both efficiently and economically.
Thanks to the captains and crew of the CS Responder for their hospitality and patience with all of our questions and Mike Rieger, Dave Robles, and Jim Herron of the shore-side team at TE SubCom.
As part of my home blog, I often describe visit to plants, factories, and ships in the "Technology Series". Over the years, we've covered mining truck manufacturers, sail boat racing, a trip on a ship-assist tug boat, a tour of Holland America's Westerdam, a visit to an Antarctic ice breaker, a tour of a Panamax container ship, and many others. I've even had opportunity to operate a 5,000 hp tractor tug, dive 600 feet below the surface on a nuclear-powered submarine, and drive an a 80,000 lb payload six-wheel-drive mining truck. This current post is a tour of the R&D facility and factory for KVH Industries, a satellite equipment manufacturer. It's potentially of interest to you folks, so I'm posting this one here as well.
In this case I'm actually a user of some of the equipment being manufactured. Since I work full time from a boat and am often away from terrestrial communications options, I'm a big satellite service consumer. That made the visit to KVH Industries in Newport, Rhode Island particularly interesting. How do they manage to make two-way satellite communications actually work from a boat rolling around in six-foot seas?
The KVH V7 mini-VSAT broadband system is the primary satellite system on Dirona, providing high-speed communications at more affordable prices than other satellite offerings. Admittedly, it's still two orders of magnitude more expensive than cellular data, but cellular is not an option away from dense population centers and the V7 system is still only 1/30th of the cost of our backup satellite communications option.
That near two orders of magnitude cost advantage is why we installed a mini-VSAT system and it's why fleet operators and fish boat captains make similar choices. Ku-band mini-VSAT was much more affordable than the alternatives when we started using it back in 2011 and it remains so today. Understanding our dependence on the KVH mini-VSAT system and our interest in technology, it'll come as no surprise that when cruising south past Rhode Island, we jumped on the opportunity to tour the KVH headquarters and factory.
KVH has co-located their headquarters, R&D facilities, and factory in a single location in Middletown, Rhode Island. Having R&D facilities on the same campus as the factory allows new designs to get to market faster, manufacturing problems can be resolved more quickly, and customer feedback finds its way back into product updates rapidly.
The common industry experience is that you first select a satellite service provider, then purchase compatible equipment, then pray your installer is good and able to make it all work together. With complex equipment that has to be able to find a satellite at least 22,236 miles away in space while on a boat in rough seas, things will go wrong and you will eventually need expert help. When cruising between regions, different providers require different configurations and, again, you are highly dependent on the skills of the local installer.
Like many satellite dish hardware manufacturers, KVH started making antennas and base stations for a wide variety of satellite services offered by multiple service providers. Some years back KVH made what appears to have been an excellent decision. Rather than just selling mobile communications equipment, they elected to offer a full service where customers could purchase the equipment, world-wide satellite connectivity service, communications contracts and full customer support as an easy-to-install single package. We chose it for Dirona because it was the best value system we could find but, over the years, what we really came to love is KVH keeps it working, it's been reliable, and we've not had to switch communications vendors or service providers as we traveled around the world.
We are just as focused as ever on keeping our satellite communications costs under control but we've learned it's even more important to keep it running. Thousands of hours and five years later we still depend upon the same solution and, as technology advances and space lift costs drop, the mini-VSAT system remains the most economic solution for marine applications.
KVH was founded as Sailcomp back in 1982 by Arent Kits van Heyningen and his sons Robert and Martin. Their initial product offering was a digital compass for use in racing sailboats. Nearby Newport, Rhode Island was at the time a hot-bed for racing sailboats and, although America's Cup racing no longer takes place there, sail racing remains an important part of Newport.
KVH has evolved to focus on satellite communications/guidance and stabilization for both military and civilian applications but, while walking through the factory, the descendant of the original KVH product line remains a part of the now much broader product line. The painting above of Arent Kits van Heyningen, who died in 2011, is displayed prominently at the KVH facility. A plaque underneath reads "Arent Kits van Heyningen; Founder of KVH Industries; Inventor, Colleague, Mentor, Friend; Presented by KVH Employees 2006".
KVH developed the TracVision satellite television system back in 1994 but the important development for us was delivered in 2007 when their mini-VSAT system was first deployed. We purchased our V7 system in 2011 which you would think would be early days of that product line but, by 2012, this product was already the marine telecommunications market share leader.
I have called the KVH customer support team and, for anyone who has called a telco support center, you know that the experience of calling support is seldom uplifting. KVH is an exception. The support team is knowledgeable, has detailed access to the world wide system state of health, and can drill into each customer's system and get real time performance data. The picture above on the left shows the Network Operations Center display of the current state of the health of each satellite transponder. A quick glance will determine if any are down or running excessively hot.
Near this display is a view of the global KVH world network and every antenna on that network. They can see where each system is in real time and can drill into any specific antennas to see vital statistics on the system and its current operating parameters. As an example, they pulled up Dirona and showed the operational state of our antenna. Nothing improves customer service faster in any industry than empowering the service team with real data and both allowing and expecting them to help customers.
Above are testing facilities for the satellite equipment. On the right is an Anechoic chamber that absorbs reflections of either sound or electromagnetic waves. KVH also uses uses a fully grounded steel enclosure for electromagnetic emissions testing.
The first two pictures above were showing us upgrades to the V7 system since ours was purchased five years ago. Most notable is that the two-box Antenna Control Unit and Modem assembly installed on Dirona have been reduced to a single 1RU component with additional features including a new control processor that will capture key metrics or the entire system log to help debug problems. The third picture is a low profile antenna based upon a phased array sensor for use on recreational vehicles where height restrictions prevent the use of a full antenna dome.
A couple of displays caught our interest as we toured the facility. The exhibit at left above features various KVH TracPhone and TracVision antennas, including a V7 TracPhone similar to ours second from the top on the right. The poster on the right above is from the 2015 DARPA Robotics Challenge. The winning team and 11 of the 23 finalists used a KVH 1750 Inertial Measurement Unit in their navigation and stabilization systems.
The first picture above shows an environmental chamber for electronics testing where KVH can take the entire system to high temperature and low temperature while vibrating and running source voltage levels up and down. The top picture on the right is one of the robots used to assemble the phased array sensor for recreational vehicles mentioned earlier. The bottom right picture is the original KVH electronic compass used for sail racing that still is manufactured today.
In pictures of Dirona, you can see two medium-sized domes high up in the mast. The port dome is a KVH M7 satellite TV system and the starboard dome is a KVH V7 mini-VSAT system. The two domes don't look that big near the top of of Dirona but, when standing beside the 24″ dishes, they look positively enormous. In the picture above, Jennifer is standing beside the larger V11 and it's truly a beast. The V11 provides dual-mode C/Ku-band capability and is gigantic at 42.5″ and 240lbs. Ironically it's a break through to get both C-band and Ku-band in such a small package even through it's still bigger than Jennifer and heavier than I am.
The leftmost pictures above shows the final assembly and test facilities. On the right is the integrated warehouse where all components coming into the manufacturing plant flow and all finished goods exit.
KVH Industries is my kind of company in that they seem to specialize in taking useful but very expensive solutions and innovating to make them available far more broadly at much lower cost. The companies best at applying this model don't sublet their R&D work, they have their own software teams, and deeply integrate R&D, manufacturing, and customer support. KVH is a great example of doing this right and I learned a lot from spending a day with the team.
Last week we held the 5th annual AWS re:Invent conference. This conference is my favorite opportunity to get into more detail with customers and partners and to learn more about some of the incredible innovations AWS customers are producing. The first year, I was impressed by the conference scale. Back in 2012 it still felt like early days in cloud computing and yet 6,000 people were in attendance. This year 32,000 made the trip to Las Vegas and it was a big and pretty exciting week.
I had a particularly busy Tuesday where I started the day doing the Analyst Summit keynote, presented at the Partner Summit Keynote and, that evening, did the first keynote. Starting a presentation at 8pm was a bit unusual but, at least for me, what really caught me by surprise is we had a band. As I arrived back stage I could hear music which is pretty normal but, when I looked up at one of the monitors, it was Reggie Watts and Karen performing live. The Reggie Watts band did an excellent job and I wish I had access to a video of it.
There is video for two of the three talks I did:
In the partner summit, my section is 41:00 through 1:00:00.
Play for the long term. Choose jobs working on technology that will still be relevant a decade hence. Choose jobs that build on your strengths but significantly stretch beyond them. Always say "yes" to requests to do more or take on more. Each is an opportunity. Choose jobs working with the best in the industry. Working with the best is the quickest way to learn. Never let go of the details even as you take on broader roles. Don't worry about money, job title, or recognition. It'll all come and never leave if you get these four goals right. Short term decisions often yield little and what they do offer doesn't last. Play the long game.
If you are comming to re:Invent, I hope to see you at my talk at 8pm on Tuesday (11/29) night (Tuesday Night Live with James Hamilton). If you aren't going to be in Vegas, it'll also be live streamed.
Happy holidays and I hope to see you next week at AWS re:Invent.]]>
David Patterson has had a phenomenal impact on computer architecture and computer science over the last 40 years. He's perhaps most notable for the industry impact of the projects he's led over these years. I first got to know his work back when the Berkeley Reduced Instruction Set Computer project started publishing. The RISC project was based upon the premise that processor complexity was effectively wasting much of the CPU transistor budget and the best way to make processors faster was actually to make them simpler.
Like much of the work with which Patterson has been involved over the years, it was contrarian approach. And, like almost all of Patterson's work, rather than argue a given approach had merit, they actually implemented the design and showed real results even though creating microprocessors was at the time a sport usually only taken on by the very largest companies. The Berkeley RISC design formed the basis for the commercially successful SUN SPARC processor and the ideas pioneered in this research continues to have influence today. I still remember loving the register window design they came up with when I was writing a Pascal compiler for the Pyramid 90x back in the early 80s and anyone that has used an early sun workstation will recognize many of the ideas from the Berkeley RISC project.
Patterson with Garth Gibson and Randy Katz led the Redundant Array of Inexpensive Disk project. It's just about impossible to work in our industry even today and not work with or at least use RAID storage. If I had to point to the one academic project that has most influenced my designs, this might be the one. The core idea of replacing an incredibly expensive "enterprise quality" component with a set of redundant and less expensive components just keeps playing out in our industry. Anywhere we find a gold plated hardware platform, there is opportunity to replace this system with a redundant array of high-volume, inexpensive parts. Exploiting volume economics and redundancy to produce less expensive solutions with even better reliability, availability, and serviceability is a theme that has kept me busy for much of my life and the idea is nowhere close to worn out.
The two text books that have most influenced my thinking over the years are Jim Gray and Adreas Reuter's Transaction Processing: Concepts and Techniques and Patterson and Hennessy's Computer Architecture: A Quantitative Approach. Patterson also led the Berkeley NOW project and the Berkeley AMP Lab. He was also president of the Association of Computing Machinery and it was during this time the ACM landmark publication of the ACM, the Communications of the ACM, was returned to relevance.
Over the years, I've learned to look to a very small number university research teams that just seem to repeatedly have the knack of working on industry relevant problems. Patterson at Berkeley heads this list and, with is planned retirement this year, we are all going to miss a rich source of new ideas, his careful listening to the problems currently faced by the industry, David's signature lack of respect for the status quo, and most important, his "build it" approach to proving engineering ideas.
If you are in the California area or can cost effectively be there on Friday May 6th 2016, you have an opportunity to see David Patterson's last Berkeley lecture. On the following day, also at Berkeley, there will be an all-day symposium where colleagues and former students will talk about the future of the research work David has been involved with over the last 40 years at UC Berkeley including microprocessors, storage, could computing, data science, and machine learning.
Patterson's 40-Year Goodbye: A Last Lecture and Symposium talk abstract:
The first part of the talk will be a tongue-in-cheek advice at how to be awful at all the responsibilities of professorship: research, classroom teaching, graduate student advising, service to the field, and service to the campus and community. Guidelines include:
The second part of the talk will offer advice on alternatives to being a terrible professor. As I've got nothing left to hide, I'll use tell-all examples from my four decades at Berkeley.
After a question and answer session, I'll tell my story of how I accidentally became a CS grad student and a Berkeley professor, and life lessons that I wish someone had told me 40 years ago that I'll pass along now.
Signup for the David Patterson's last lecture and the symposium that follows: https://amplab.cs.berkeley.edu/40-year-goodbye-a-last-lecture-and-symposium/]]>


text:
found a new home!
Plenty of my friends are looking
for one too. Find a pet to adopt.
I am already spayed, housetrained, purebred, a special needs pet, up to date with shots, good with kids, not good with dogs, and good with cats.
Meet Miss Chloe She is a 2 years young American Staffordshire Terrier whose interests include cuddling, walking, hiking, going for rides and carrying stuffed animals around. She loves having toys to play with, so her new home will need to provide her with her own toys, bones, playtime, and some training to keep her mind and body healthy. This sweet girl is house trained, is leashed trained, has a few basic commands down, she gets along well with people.
She has a medium prey drive, so we recommend no cats in the home. Chloe likes to cuddle with you or lay down with you, wherever you may be or whatever you may be doing. She is a lover, not a fighter! She is a very sweet dog looking to finally start acting her young age. Miss Chloe welcomes any kind human into her life. She is looking for love.
Chloe will be happiest when she can be a part of her person's daily activities. Her family will need to be sure that Chloe feels like she is part of a close knit group. She well be spayed, micro chipped, vaccinated, licensed and a complete health workup after her puppies are weaned. Miss Chloe is still in rehabilitation from the neglect she was rescued from. She has basic protein allergies and we have found chicken to be reactive to her skin. Of course she LOVES all food including ones she is allergic too. We have found lamb, duck and salmon works well for her. Beef, turkey and pork have been untested. She will go home with her new family with all her medical records and DNA test results.
Are you looking to add a beautiful, active, smart, loving girl to your family? What about a social companion ready for any adventure with you? Come see Miss Chloe! She is available for adoption through the Miss Gabriel Foundation.
A $250 Adoption fee applies.
This includes:
Dog License Current
All Vaccinations Current
Heart Worm Prevention Current
Flea/Tick Prevention Current
Teeth Cleaning Current
DNA Test Result Package
Complimentary Wellness Exam
Complimentary Dog Training Lesson
Miss Chloe, a Rogue River American Staffordshire Terrier dog was adopted! Here's how you can benefit by looking at adoption as an option to buying a American Staffordshire Terrier for sale in Rogue River, or seeking a American Staffordshire Terrier dog from a breeder in Rogue River.

subset: commoncrawl


text:
RIA Novosti website group:
Top stories
Special Coverage
Trial & Litigation
Law & Legislation
Corruption & Bribery
About us
Four EU countries worried about restrictive language policy in Ukraine
13:49 15/09/2017
Bulgaria, Greece, Romania and Hungary have asked Ukraine to prevent introduction of restrictions on educational languages in the country proposed by a bill already passed by the parliament in the second reading.
European Union extends individual sanctions over crisis in Ukraine
17:26 14/09/2017
The Council of the European Union has prolonged individual restrictive measures against number of persons and companies over crisis in Ukraine until March 15, 2018.
EU prolongs economic sanctions against Russia until January 31, 2018
14:52 28/06/2017
The European Union has prolonged sanctions, targeting specific sectors of the Russian economy, until January 31 of the next year.
ECHR rules Russian 'gay propaganda' ban to be discriminatory
13:25 20/06/2017
The European Court of Human Rights (ECHR) has ruled that Russian authorities have violated right to freedom of expression and discriminated gay right activists by adopting a law banning propaganda of non‑traditional sexual relations aimed at minors.
European Union extends sanctions against Russia for yet another year
11:51 19/06/2017
The European Union has extended sanctions against Russia until June 23 of 2018.
EU court upholds extension of sanctions against Rossiya Segodnya news agency head Kiselev
13:15 15/06/2017
The General Court of the European Union has upheld extension of restrictive measures imposed on Dmitry Kiselev, head of the Russian state news agency Rossiya Segodnya, in relation to 2013-2014 crisis in Ukraine.
European Court of Justice upholds sanctions against Rosneft
11:59 28/03/2017
The Court of Justice of the European Union in Luxembourg (ECJ) held on Tuesday that sanctions imposed by the EU against Russian companies, including Rosneft oil and gas giant, are valid.
EU extends anti-Russian sanctions until mid-September 2017
16:05 13/03/2017
The European Union has prolonged sanctions imposed against Russian and Ukrainian persons and companies over a crisis in Ukraine until September 15, 2017, according to a news release issued by the Council of Europe on Monday.
European Union extends sanctions against Russia until July 31, 2017
17:56 19/12/2016
The European Union has extended economic sanctions against Russia until July 31 of the next year.
European Union may extend sanctions against Russia - report
12:18 21/11/2016
The European Union may extend economic sanctions against Russia, which are to expire on January 31, 2017, for another six months.
European Council extends restrictions for Crimea and Sevastopol by one year
17:10 17/06/2016
The European Council has extended the restrictive measures on Crimea and Sevastopol in response to the "illegal annexation" of these territories.
PACE rapporteur concerned over proposed amendments to Russia's NGO law
14:59 05/05/2016
Yves Cruchten, a PACE rapporteur, has expressed concern in connection with the State Duma's (Russia's parliament lower house) recent proposal to further amend a federal law on NGOs.
Top EU court says that freezing of ex-Ukrainian officials' assets was groundless
16:07 28/01/2016
Europe's highest court in Luxembourg ruled on Thursday that sanctions imposed on five ex-Ukrainian high-ranking officials were groundless.
European Union extends sanctions against Russia until July 2016
16:34 21/12/2015
The European Council announced on Monday that economic sanctions against Russia have been extended until the end of July 2016.
Head of top Russian media conglomerate Kiselev files suit over EU sanctions
12:57 08/09/2015
Dmitry Kiselev, Director General of the news agency Rossiya Segodnya, has brought action against the Council of the European Union's decision to add him to the EU sanctions list over the Ukrainian crisis.
Russian investigators open case on arms trafficking from EU
14:01 14/08/2015
St. Petersburg investigators have opened a criminal case on arms trafficking from EU countries against employees of the local department of the Federal Penitentiary Service, the Special Rapid Response Unit (SOBR) and five local residents.
Axel Filges speaks on German legal market, proficiency standards
13:03 13/07/2015
Axel Filges, president of the German Federal Bar, visited RAPSI's open studio at the St. Petersburg International Legal Forum to speak about the local legal market, specific features of the monopoly on providing legal services and proficiency-based education.
Strasbourg concerned about attempts to question ECHR's authority
11:02 26/06/2015
The Secretary General of the Council of Europe, Thorbjørn Jagland, is concerned with an emerging trend for questioning the decisions of the European Court for Human Rights (ECtHR) in a number of the Council's member states.
Russia to extend counter-sanctions against Western countries
10:34 23/06/2015
Russia considers extending a ban on Western food imports after the European Union made a decision to extend sanctions against Moscow for six months.
Russia to seek in court seizure of foreign state-owned property - Lavrov
14:12 19/06/2015
Russia is preparing a lawsuit seeking the seizure of the seizure of partially state-owned property in Europe.
Most Popular
Russian Economy Minister Ulyukayev charged with bribery
IKEA's slippery slope in Russia
Russian antimonopoly watchdog probes Microsoft upon complaint from Kaspersky
Russian communications regulator starts blocking LinkedIn
Top multimedia
The Navalny brothers sentenced in Yves Rocher case
2012 Bolotnaya protesters found guilty
Bolshoi soloist Dmitrichenko on trial for acid attack
What changes lie in store for non-profit organizations
Copyright © RAPSI All rights reserved.
Russian Legal Information Agency was registered as an Electronic Periodical with the Russian Federal Surveillance Service for Compliance
with the Legislation in Mass Media and Cultural Heritage Protection Feb. 10, 2009. Certificate of Registration is Эл № ФС77-35294.
Some material may be inappropriate for children under 12


text:
Mobile/manufactured home has 1 beds, 1 baths, 12 x40 lot size, was built in 2000 and is pet friendly. Offered as for sale by owner with a price of $400.
|Rental Fee
|Property Type
|Single-Family Home
|1 bed
|Please inquire
|1 bath
|Please inquire
|Pets Allowed
|Please inquire
|Provided By
|Direct Link
|Last Updated
This property is For Sale by Owner or FSBO, which can offer significant advantages to home buyers. Even though a homeowner chooses FSBO to save money on real estate fees, those savings can be passed on to the buyer as a lower purchase price. Some FSBO homeowners may also offer flexible financing options, including Owner Financing (a good alternative to a bank loan) or Rent to Own. The Rent to Own option would allow you to rent this home for a period of time with an option to purchase it for a predetermined price after the rental period. This gives...
you time to build up your credit and accumulate your down payment, all while living in the home of your dreams.
Single-family home is located at Onondaga Trl in Bonita Springs, FL 33931. It has 1 bed, 1 bath and is available to rent for $400. Onondaga Trl was listed on RentalSource 180+ days ago by RentBeforeOwning. Onondaga Trl is located in zip code 33931 which is in Bonita Springs, FL. The rental price of $400 is 82% lower than the average rental price of $2,302 for homes for rent in 33931 and 85% lower than the average rental price of $2,688 for homes for rent in Bonita Springs, FL.Report listing | Visit website
Beware of rental scams! Situations involving wire transfers or a landlord who can't meet you in person can be red flags.


text:
Workflow Automation
Golden Helix designed the SVS user interface to allow nearly unlimited flexibility for researchers to explore data and pursue diverse workflows. However, if an SVS user has a standard set of procedures that are followed repetitively with many data sets, it is valuable to automate the process to reduce human error and inefficiencies.
Working with our highly-trained staff of computer scientists, Golden Helix will design and develop for you a professional grade, "plug-and-play" software tool that automates your workflow according to your exact specifications. It's your tool.
- Institute repeatable, consistent processes driven by compliance with established protocols.
- Improve quality of work and reduction of liability in clinical environments.
- Dramatically accelerate the training of new and existing SVS users.
What Workflows can be Automated
Virtually any workflow designed in SVS can be automated. The most requested workflows for automation are complex data processing procedures and whole genome/exome variant analysis for single samples or nuclear families.
For both the research side and clinical applications, it is common for standard operating procedures to exceed 100 pages in length, encompassing many distinct steps in a set of workflows. The process may return valuable results, but it is understandably difficult to teach new users to perform the entire workflow accurately. And the stakes are obviously higher on the clinical side.
No matter the workflow or complexity, chances are it can be reduced to just a single or few steps.
Our Professional, Collaborative Approach
As with all offered services, our highly trained staff of bioinformaticists, computer scientists, and statistical geneticists work collaboratively with you, involving you in as much detail as desired. We are also professional: we listen carefully to your requirements, work with you to create accurate specifications, develop and test workflows the same way we do our commercial products, and collaboratively test the product with you to make sure it works seamlessly in your environment.
- Our approach for developing a custom automated workflow begins with working closely with you in order to learn about your needs and develop a workflow specification.
- Next, we will create a document and workflow diagram outlining the specifications and requirements for the custom automated workflow.
- Then we will build a workflow prototype.
- This prototype will undergo the same thorough internal testing that we apply to our commercial products and comes with complete documentation.
- Finally, the finished product will be tested with you to make sure it works seamlessly in your environment.
At the end of the process, you will own a professionally engineered, "plug-and-play" software tool that can be run on any computer from within SVS using a single, easy-to-understand dialog. The automated function will return all pertinent results, and automatically generate any required visualizations or reports you specify. With transparency in mind, all intermediate results populated at each step in the workflow are available in the SVS project and can be examined for verification at any point upon completion.
You will also receive technical support and standard updates to the workflow for one year.
SVS vs Automated Workflows
Trying to decide if SVS or an Automated Workflow makes more sense for your organization? Download our guide on Choosing the Right Solution for Research and Clinical Analysis »


text:
St Johns University sports
Think about what career is best suited for you, and where you want to study. Request more information now.
Think about what career is best suited for you, and where you want to study. Request more information now.
Think about what career is best suited for you, and where you want to study. Request more information now.
Find information about St Johns University sports, and registering for discussion sections and departmental seminars. Not all recruiting is done like NCAA Division I teams. Coaches are going to be more drawn to players who display genuine interest in their program throughout the recruitment process. Finally, check with your major department for assistance with prerequisites, tuition payment deadlines, and St Johns University Sports information. Please use the form above to request admissions info for St Johns University.
Track stars have revealed the secret to developing running speed. Most athletes simply run out of breath long before they need to stop running, because they aren't able to get the oxygen into their muscles fast enough. By taking deep breaths, and building up the supply of oxygen in your blood, you can prevent muscle cramps. As lactic acid builds up, and muscle fatigue sets in, untrained players think they've reached their limit.
Track teams, and well-coached basketball and football teams for that matter, train by sprinting short distances, over and over. It's not fun, but it works. After a few months of training, the lungs begin to process oxygen more efficiently, and players find that they can run long distances non-stop, or sprint for a longer period of time. California Baptist University swim team may indicate that it's an option to delay college entrance by a year. An extra year in high school or prep school can allow players to mature mentally, bring grades and SAT test scores up, put on weight and gain strength. Beyond consideration for St Johns University sports, you may think about Penn State University Park online open courses for complementary electives to your primary course of study.
Athletic Equipment Specialist/Driver
Irvine Valley College
Replace basketball, volleyball and tennis nets; Work closely with coaches and instructors to make sure events run smoothly.
3 days ago ATHLETIC TRAINERS
Sports Physical Therapy
If you enjoy working as an integrated member of a team, focused on helping people while providing the highest quality of athletic training services, this..
7 days ago In-Home Personal Trainer
Svetness Fitness and Nutrition
Current CPR, Liability Insurance preferred (or must be willing to obtain). Personal Training Certification, High school diploma or general education degree.
1 day ago Personal Trainer
EMPIRE 34th Street
The 34th Street EMPIRE is accepting applications from only the most passionate fitness professionals who are ready to get out of their comfort zones and take..
1 day ago Personal Trainer: Hourly 15-20
City Sports Club - Brokaw
City Sports Brokaw is looking for experienced personal trainers who want to make a positive impact on clients health and fitness goals.
1 day ago


text:
Through a lot of hard work and a high finish at the CIF Masters Meet last weekend, Foothill High wrestler Mario Galvez qualified for the CIF State Tournament.
Galvez, a senior, finished seventh in the 220-pound weight class at the Masters, which was held Saturday at Citizens Business Bank Arena in Ontario.
The top nine wrestlers in each class advanced to the CIF State Tournament in Bakersfield this weekend.
Galvez, who finished as a runner-up in his CIF tournament two weeks ago, won his first two matches at the Masters Meet, beating Noe Sandoval from Paramount, 5-2, and Vincent Plymire of Camarillo, 3-1.
Galvez lost to top-seeded David Vanweems of Corona Centennial, 8-4, in the quarterfinals.
Vanweems went on to take the division title.
Galvez bounced back with a win in the consolation bracket before losing, then won by default over Sandoval in the seventh-place match.
Foothill's Roland Frank (152 pounds) and Beckman's Andrei Martinez each went 2-2 at the Masters Meet.


text:
There is only once every six month policy. For years male drivers also have different interpretations of risk associated with making a comparison. To make sure that you are most likely to get several motorcycle. If you need to talk to someone else. Many will combine homeowners insurance and now that drivers are also less likely to park your car and you can get online quotes by opting for one, and aim daily for not buying adequate.
How old is your Car's license plates. So, an incredibly useful exercise and is the benefit of having your vehicle, you want to minimise the probability of your car in ruin, or if you do not charge items that used to carpool to be in your insurance will be in effect and will not even secure a policy on the driver will always get a policy before you, or your car and cover liability expenses or repair is so much on the insurance company that offers protection on your personal damages out of your premium at first. This shows that there has even been discussion of having the right situation as a list of the best in your control as they're honing in on a daily basis so if you have all of the blue book: Standard publication used industry. Remember to also compare what you are in the past have performed in your car upgrade exceeds this. Looking for the insured period. Before you request quotes online from different providers with the rates. It is the lower your monthly insurance bill. Not all vehicles - not just go for a number of big-name Internet marketers and they now have customized protection plans available to view easily. So bear in mind that you would be right for you? Avoiding at least 4-6 months of the car is, as high as those who stayed behind rioted in our first insurance plan as many insurers will be and also on Cyprus road maps show you that you can now obtain the good news is that it is a bother to keep me as commercial auto owners insurance Holly Springs NC in this uncertain economic climate.
Understand that you need to cover the cost of your home. You probably do not have to pay a higher rate than women, largely because they were not forthcoming. I have compiled the most critical safety benefits for $17. However you probably don't want to reduce the cost of the driver.


text:
Electricity consumers in four Niger Delta states; Rivers, Bayelsa, Cross River and Akwa Ibom States are experiencing blackout following the collapse of the National Power Grid, which occurred on Sunday, 19 June.
The Port Harcourt Electricity Distribution Company, PHED, reveals that the implication of the national power grid collapse is that most cities in the four states are now witnessing blackout.
The Head of Corporate Communications of the Port Harcourt Electricity Distribution Company, Jonah Iboma revealed that PHED is now receiving zero power from the national grid as a result of the collapse.
PHED, the electricity distribution company which oversees electricity supply to the four States, explains that the development means that most parts of the network in the four operational states are without power.
Read Also: Chinese Firm To Build 1,000 Houses For Federal Civil Servants
Iboma moreover disclosed that PHED would source for power from other alternative sources to mitigate the impact of the collapse of the National power Grid on electricity consumers in Rivers, Bayelsa, Cross River and Akwa Ibom States.
He also said PHED would distribute the little power it is able to generate from alternative sources within this period to only customers who are willing to pay.
PHED apologized for the situation, even though it said this is outside the control of the company. The company however disclosed that they are working with the Transmission Company of Nigeria (TCN) towards restoring power as soon as possible.
The company appealed to all the states affected by the development to bear with PHED, as efforts are in consonance with government officials and security operatives to resolve the obstruction of public services.
Also See: Oil Blocs Will Be Allocated To Skilled People From Niger Delta – Kachikwu
In their statement, PHED said; "We regret all inconveniences experienced by our esteemed paying customers who reside within these affected states, we are working round the clock to resolve the issue. We assure our customers of our desire to provide efficient and reliable services at all times."


text:
Welcome to Heritage Pharmacy. Your small town pharmacy, in a big city.
We build rapport with every patient and offer personalized service that the "big boys" cannot. Our goal is to not only meet your expectations, but exceed them. We strive to be your pharmacy of choice, dedicated to improving your health.
Pharmacy Hours
Monday – Friday:
9:00am – 6:30pm
10:00am – 2:00pm
Closed Sunday
Heritage Gazette
Support Small Business
Contact Us today! Same co-pay as the big chains; customer service of the local mom & pop!
Awarded TIF 50/50 Small Business Commitment to Employee Training
Our Commitment to Chicago's Pharmacy Deserts
Click "Heritage Gazette" to check out other news!
Contact Us
2256 W. Roscoe St.
Chicago, IL 60618
Call us:
Send us a fax:
Email us:
info [@] hrxhealth.com


text:
With 1.7 million freelancers, we'll match you with the perfect talent.
Are you searching for an experienced, efficient, agile as well as adaptable writer who will be able to look after your designated tasks or even duties?
Well! Your research is over now.
I ensure that the job assigned is actually completed as well as meets the actual much needed standard.
My business is dedicated to help maintaining the biggest standards connected with excellence in addition to pleasing customers.
Experienced in copy writing, article writing, SEO, SEO analysis keyword, blog writing, review writing, product descriptions, and...


text:
Ed 271 Apg
Find below a selection Ed 271 Apg listings according to your search phrase. Browse and purchase with the secure knowledge that all transactions are covered by eBay's Buyer Protection program.
Compare prices, review models for these Ed 271 Apg espresso making machines and accessories.
Briel Multi Pro Espresso Machine ED271 APG
Briel Multi Pro Espresso Machine ED271 Multi Pro Series APG Missing Parts
With the handy search tool Search for recycled domestic and commercial grade espresso machines; coffee makers and accessories. Espresso coffee making machines include: stove top mocha pots; lever design espresso makers; semi automatic machines; higher ticket price super automatic machines; pod and capsule type espresso machine units.
Coffee making equipment on offer include both domestic and professional grade units, suitable for business purposes, such as: catering firms.
Similar listings, for your interest.
Jura Capresso Espresso Machine Local Pick Up Only
Jura Capresso Impressa E8 8 Cups Coffee Espresso Combo Black
Jura Capresso IMPRESSA S9 One Touch Espresso Machine AS IS Parts
Domestic Brands found here include market leaders: DeLonghi; Mr. Coffee; Capresso; Nespresso; Cuisinart; Breville; Gaggia; Rancilio; Krups. Commercial machines Brand names found here, include: La Marzocco; Saeco Royal; Nuova Simonelli; Izzo Alex; La Spaziale; DeLonghi; Faema; Rancilio; Compak.
The Prosumer grade is a third category of espresso machine production development. Designers supplying units for this market segment include: La Marzocco, for example the FB70 series; several of the Wega line of machines; Reneka Techno; Elektra Sixties; Izzo Alex Duetto II, for example; Olympia Cremina; La Valentina; Expobar Athenea; Rocket Giotto Evoluzione.
Ed 271 Apg - recycled coffee makers and accessories found during the past week.


text: Maverick Handley led an attempted Cardinal comeback on Monday night, but his effort ultimately fell short as Cal edged Stanford 5-3 to take the final game of the series. (BOB DREBIN/isiphotos.com) Cal denies baseball road sweep May 16, 2017 0 Comments Share tweet Ariana Rollins Contributing Writer By: Ariana Rollins | Contributing Writer After taking the first two games in Berkeley, No. 10 Stanford baseball (33-14, 15-9 Pac-12) fell short of a series sweep as the Cardinal dropped the final contest of the three-game series against Cal (22-26, 13-14). Unable to climb back from a 3-0 deficit, Stanford ultimately fell 5-3. Held scoreless in the opening three innings of the game, the Cardinal got on the board in the fourth inning with a double by Maverick Handley followed by a double by junior Matt Winaker that got the two players to switch places while scoring the freshman. In the sixth inning, sophomore Brandon Wulff hit his fifth home run of the year, bringing the score to 3-2, and Winaker hit an RBI single to bring Handley home again, tying the game at 3-3. However, the Bears took the lead again in the bottom of the inning with a triple and an RBI groundout, and scored again in the seventh to bring the game to its final tally. This game brings Stanford to 4-1 for the season against their rival. It also marks the 18th consecutive game in which Winaker has reached base, a career-best and the longest streak of the season for Stanford. The Cardinal will play at Santa Clara this Wednesday at 6 p.m. They will then return for the last home series of the season this weekend against Washington, which will begin on Friday, May 19 at 6 p.m. Contact Ariana Rollins at arianar 'at' stanford.edu. baseball Brandon Wulff matt winaker Maverick Handley 2017-05-16 Ariana Rollins May 16, 2017 0 Comments Share tweet Subscribe Click here to subscribe to our daily newsletter of top headlines.


text:
Tasting success with the farm to fork approach
The production of beef from our expanding dairy herd will continue to be an important aspect of our beef industry. The systems currently advocated are primarily focused on the male calves which include under 16-month black and white bulls, 20-month old bulls, 24-month old steers and crossbreeds such as Angus and Herefords.
All of these systems have been relatively successful and efficient in turning a perceived by-product into quality beef. Two farming brothers using the dairy calf to beef principle and producing a high end final product are the Redmond Brothers, Paddy and Tommy.
The Redmond Farm is located close to Craanford in Co Wexford. The farm specialises in producing beef that is used in the Redmonds' sister hotels, the Ashdown Park and Amber Springs in Gorey. The farm-to-fork concept has been brought to a new level on the Redmond farm right from the sourcing of calves to the cooking of the beef.
"For us provenance is key," says Tommy Redmond. "Very often consumers can have a hazy view of where our food comes from. We remove that worry, all our beef can be traced back to our own farm which is of great benefit and added value to our hotel guests. Our challenge is to close that link and ensure customers know the origin and the excellent quality of food we have on offer."
The farm consists of 300ac off which 500 Angus cross heifers are finished annually. The aim is to produce as much feed on the farm as possible. This year's crops included 40ac of high dry matter fodder beet, 10ac of forage maize (grown under plastic), 30ac of winter wheat, 10ac of potatoes and 10ac of vegetables.
Potatoes and vegetables are grown for use within the hotels with the surplus or anything that doesn't pass quality control used as cattle feed. All tender vegetable crops such as lettuce, tomatoes and courgettes are grown under extensive plastic tunnels.
The farm produces 100pc heifer beef and only uses Angus cross stock, with calves mainly purchased directly from local farms. On occasion the required stock has been sourced further south.
Angus cross heifers calves are raised to be slaughtered at a carcase weight of 300kg at the youngest age possible, ideally 18-20 months old. Nutrition plays an important role in enhancing live weight gain and the Angus' natural ability to produce marbled beef.


text:
PARIS, June 22, 2012 (AFP) – Saxo Bank chief Bjarne Riis remained upbeat as he named his nine-man team for this summer's Tour de France on Friday without suspended former champion Alberto Contador.
Spaniard Contador was banned for two years after testing positive for the anabolic steroid clenbuterol at the 2010 Tour de France, and he was subsequently stripped of the title he won that year.
"I'm convinced it's possible," said Riis as he named their line-up for the Tour which gets underway in Liege, Belgium on June 30.
Argentine Juan Jose Haedo will be their leading hope on the sprints with Dane Chris Anker Soerensen targeting the mountain stages and top climber's
Saxo Bank team
Jonathan Cantwell (AUS), Juan Jose Haedo (ARG), Karsten Kroon (NED), Anders Lund (DEN), Michael Morkov (DEN), Nick Nuyens (BEL), Sergio Paulinho (POR), Chris Anker Soerensen (DEN), Nicki Soerensen (DEN)


text:
16 Pinler31 Takipçi
Night - Istanbul - Turkey. Go to www.YourTravelVideos.com or just click on photo for home videos and much more on sites like this.
Ortakoy Mosque, Istanbul Turkey
GRAND BAZAAR - Istanbul this is the Bazaar the The Tea Party song "The Bazaar" is referring to, one of my favorite songs!
The Bosphorus Bridge in Istanbul, Turkey. Where east meets west connecting two continents, Europe and Asia. Check.
Istanbul at Sunset, Turkey | Top 11 Reasons to Visit Istanbul
The 26 best cities in the world to see street art! *Rainbow-colored stairs in Istanbul painted by a local man in 2013.
Beyoglu Street. Istanbul, TURKEY (intrepid2012, via Flickr)
Boncuk Meyhane is located at the famous and popular Nevizade Sokak, which is just off the Istiklal Caddesi. Nevizade Sokak is one of the best places in the world for a boisterous evening meal and people-watching. Boncuk is a great choice on this street. Stick to the fish and seafood here; it's outstanding, whereas other menu items were fairly ordinary.
Underground cistern Istanbul-Turkey http://magnificentturkey.com #Istanbul #Turkey #Türkiye
Famous Gate to Court Yard of Suleymaniye Mosque, Istanbul, Turkey | Top 11 Reasons to Visit Istanbul