text:
Ratings based on 529 Verified Reviews
They paid and stayed. We double-checked.
4.6 Room cleanliness
4.5 Service & staff
4.5 Room comfort
4.6 Hotel condition
Expedia Verified ReviewsThey paid and stayed. We double-checked.
by A verified traveler
Réservation non tenue.Posted Sep 25, 2016
Réservation 2 mois à l'avance payée et à notre arrivée pas de chambre : reclassés dans un autre hôtel voisin . Allez dans l autre hôtel directement ,je le recommande : Indigo hôtel à Riverhead .
by A verified traveler
Close to Riverhead, NYPosted Sep 16, 2016
Pros: Good amenities.
Location: Unfortunately the road for hotel access is one way and not easy to navigate - no fault of the hotel.
Good experience, nice staff. Everyone made the stay pleasant. Enjoyed the out-door hot tub - indoor pool warm.
by A verified traveler from Philadelphia
Nice hotel in a convenient location.Posted Sep 11, 2016
Some difficulty turning in/out of the facility but otherwise a great stay.
by A verified traveler from Ny, NY
AdequatePosted Sep 6, 2016
Pros: Price and location
Cons: We should have been offered a new room.
We have stayed here before and it has been adequate. However this time our shower in the handicapped accessible stall was clogged and water ran all over the bathroom floor. It was fixed while we were gone but not everyone in our party (including the disabled member) was able to shower before our event. It was fixed while we were gone but we weren't even offered another room to remedy the situation. We didn't have time to wait for it to be fixed before we went out and I really didn't want anyone in there while we were trying to get ready. Being on the first floor it was very loud in the hallways late at night, I'm not sure if the staff was trying to stop that or not but we were woken up several times by loud people. Also, I found a roach in a glass of water.
by A verified traveler
Posted Sep 4, 2016
The hotel was very clean, and the staff very freindly
by A verified traveler from New York, NY
Second time we stayed here and same attitudePosted Aug 23, 2016
Pros: Convenience, clean
Cons: Some of the staff that's up front needs to be trained better in attitude with customers.
Location: Tanger outlets for shopping and vineyards not far away
We've been here twice this month and have checked in,in the late evening both times. There was a young gentlemen who checked us in both times and he just seems like he doesn't care. We got him again for this stay and same attitude. I figured maybe the first time, he had a bad day etc. We all have those days but second time in a month? Re: bar. It would be nice to have a cocktail/ wine menu. There was none and the bartender had to read out all the wine bottles to me as he grabbed them from where they were.
by A verified traveler
Posted Aug 20, 2016
I was very disappointed. I called and my agent called two days before and the day of the check in around 11am to make sure we have two rooms side by side since we have a 4 month old baby and my mother was going to take care of it. The Hilton agent told me when I spoke to her no problem she will have side by side rooms for us. I told her we were checking in late around 5 or 6pm since early checking was not available . She again confirm with me not to worry and we will have the two rooms side by side. We get to the hotel and Zak was in the front desk and is telling us no rooms available and he will have to put us on different floors. After asking for a manager or a supervisor he was looking in the computer and gave us two rooms on the same floor about twenty (20) rooms away. I am really disappointed and so is my daughter at the way we were treated by Hilton. It is not my first time that we book with Hilton and refer people to you to come and stay there. I need for Hilton to try and offer some kind of amenity to us for the two rooms. Regards, Maria Koulouriotis
by A verified traveler from Stuttgart
Ruhiges Hotel mit guter Verkehrsanbindung.Posted Aug 19, 2016
Pros: Gutes Frühstück, ruhige Lage, Zimmerstandart. Preis/Leitungsverhältnis.
Cons: Computer in der Lobby zur Informationsmöglichkeit für Reisende, mehr Ablagemöglichkeiten in Zimmer für 4 Personen, mehr Infos durch das Hotelpersonal.
Location: Coopers Beach zum Baden gehen, Ausflug nach Southampton, nach Montauk,
Waren für 3 Nächte in diesem gepflegten Hotel.Gute Zimmer mit großen Betten. Im sauberen Bad fehlen Aufhängemöglichkeiten für Handtücher bei 4 Personen. Gute Einkaufsmöglichkeiten in unmittelbarer Nähe durch Walmarkt und Tanger Outlet.
by A verified traveler
Super OutlethotelPosted Aug 19, 2016
Tolles Lage zum Outlet Kaffee immer frei Schönes Ambiente in den öffentlichen Bereichen
by A verified traveler from G
Short stay, great experience.Posted Aug 16, 2016
It was only a short stay just passing through. Great shopping location and very clean hotel.


text:
Foursquare © 2016 Lovingly made in NYC & SF
People tend to check in during these times:
Convenience Store
407 8th St NE (at 8th Street, NE)
"Great, friendly service here. They always have fresh coffee."
661 Pennsylvania Ave SE (btwn 6th and 7th St SE)
"DO NOT USE THE PHARMACY. Grossly incompetent."
Make sure your information is up to date. Plus use our free tools to find new customers.
Download Foursquare for your smart phone and start exploring the world around you!

subset: commoncrawl


text:
Interdisciplinary. Community. Advocacy. Humor.
A series of cams rotate inside the object and stretch through the sheepskin.
kinetic object - nickel, brass, copper, sheepskin
Views: 25
Albums: BFA Work
Join crafthaus
Welcome tocrafthaus
Sign Upor Sign In
Every other year, crafthaus awards a micro grant to a craft artist regardless of their location, professional background, or chosen craft field.
2016 Submission period
September 6 - 30, 2016.
Eligibility and additional information: https://is.gd/qhlXfz
Entry form
Karen Lester,Warwickshire, UK
Brooch, Fluidity Series.
Cast bone china, glaze, slip, silver.
© 2016 Created by Brigitte Martin.
Powered by
Badges |
Report an Issue |
Terms of Service
Please check your browser settings or contact your system administrator.


text:
Top Definition
Someone who is well past youth; an older person.
Leslie: did you know that Harry's mother turned 90 last week
Kevin: Well she's no spring chicken!
Kevin: Well she's no spring chicken!
viết bởi Miss22894 14 Tháng mười hai, 2013
Trending Now
Can you define these popular missing words?


text:
Cyberjamz Radio 13th year Anniversary & Server Donation Drive would like to Thank YOU and make the following acknowledgements to those that participated this year and those that took a moment to make a donation.
On behalf of Cyberjamz Internet Radio, Sammy Rock, DJ Housecat, Jorge G, we would like to express our deepest gratitude for your participation and generosity during the Cyberjamz 13th Anniversary Marathon and Server Donation Drive.
We have been blessed to be able to give to our Music community, listeners and programmers the platform with which to share music and ideas that touch us all. Through the vast airwaves we have all become connected and our family grows to include some of the most amazing lovers of music that anyone could ask for.To each of those who have donated, we are honored that you made the effort and, and in some cases, sacrifice to express your support through monetary donations.
It's because of people like you that we are able to continue what we do, well beyond our 13 years.To those who shared the information, came into the chat all hours of the day and night to keep our programmers company and cheer us all on, we are blessed to have your continued support and love. You make it all worth while. You are our cheerleaders, dancers, ruckus makers! It's why we love you. It's why we play for youTo all of the programmers who stepped up and really embraced the concept of the Cyberjamz 13 year Marathon, we are grateful for your love of what you do and that you choose to share it with us. You fill the airwaves with such incredible energy that it keeps people coming back!
Thank you!
To every person behind the scenes who works diligently to make sure that Cyberjamz keeps moving and growing. Thank you. You may not be in the public eye, but we see you.Without all of you none of this would be possible. Thank you! Excited to be spending the next 13 years and beyond with you all.
**Special thanks to the following.
The Music Players:
Jorge Garcia ,Rescue Poetix ,Sheila Johnson , Michael Menton , Lady Vintage Soul – Cecilia Birdex , Franke Estevez , Donald Khaleel Mason , Val McEwen , Robert Shorthouse ,Delmar Browne , Todd JamKam McGill , Greg Gray , K Nyerere Ture , Thesoulfulone Mark Mcaden , Selectress MizzKae , Gary Logan , Blayne Ashmore , Al Smith , Michael Stukes , Djsteve Bass , Mark Smith , Brian Nance , Erving Batista , Anthony Mack and Darren Demann Parker , Pete Sierra
Music archives for the Marathon are now up ! http://cyberjamz.com/archives/
The Donation Crew.
Robert Shorthouse
Delmar Browne
Robbi Nypromoter
Djleonc Futuresoulmuzik
Craig King (KINGO)
Pete Sierra
Mario Rios – Larazus Lazarus Dexx
Kyle Reeves
Dj Wil Will Gee
Brandon Dee
Jason Kay
Mr and Mrs Nance.
Djsteve Bass & Suprena Millet
Big Al Smith.
Dancin Mark
Big Logan
SelectressMizz Kae
Blayne Ashmore
Abe Ab Seven Rodriguez
Darren DemannParker.
Greg Gray.
Franke Estevez.
Leanne Perry Sturdivant (Lady Strawberry)
Wesley Shaw (DjSw3)
(Lady Vintage Soul)
Dara Teague
Gloria R. Hatchett
* Pleadged to make a Donation.
Michael Stukes
Dj Khaleel.
Marsha Lovely Songbird Bond
and whoever else pleaded, I want to thank you !
Love you ALL.
SammyRock & Catherine Harris Rowell.
Cyberjamz Internet Radio.


text:
Promoting & Encouraging Family & Local History Research in Fife.
Our Online Shop contains FFHS Publications, Books, CD's and Maps.
LOGIN | Not joined us yet? Register HereLike us on Facebook Follow us on twitter
Promoting & Encouraging Family & Local History Research in Fife.
Our Online Shop contains FFHS Publications, Books, CD's and Maps.
The main reason for joining Fife Family History Society is to make contact with others with similar interests in Fife who share your enthusiasm and are willing to pass on their help and experience. For any family queries/interests please add to the Members Discussion Board. There is also a "Photograph" section where you can ask for photos to be posted. We also have a very active Facebook page where you can also put your queries/interests/photos. Our membership renewal date runs from 1 January to 31 December.
Our Archives have now moved from Methil Library to the Family Research Room in Cupar Library. Open Tues 12-7, Wed 10-3 and Sat 10 -3. Details under "Resources - Library" We are always delighted to receive copies of Family Trees which we will put in our Library Archives for others researching their Fife Ancestors.
Come and have a look through our shop.
We sell FFHS CD's and publications. Also books by Kirkcaldy Civic Society, the Dunfermiline Carnegie Trust, David Dobson, Stenlake, Birlinn, John Donald, and Tempus amongst others.
Newest titles added "Sparks from The Smiddy - The life of a World Champion Farrier" and "The Memorial in the School"
SPECIAL OFFFER***Our Fife Death Index Pre 1855 is one of our best selling CDs is now only £10 This consists of nearly quarter of a million entries! Please note this is not compatible with the most recent version of MAC Shop
Tickets are available for our Family & Local History Fair on 8 October at the Carnegie Conference Centre in Dunfermline. Booking form can be download on our "Fair" page or by contacting the Chairman. We would also welcome volunteers to help out on the day!
Our Archives in Cupar Library will not be open on 8 October due to our Fair being held in Dunfermline on that day.
Our next talk is on 11th October and is Moira Sheridan "Records in the Special Collection of St Andrews University" Details of time and venue under "Meetings" Continue Reading
Donations & Other Online Payments
We now have the facility to accept any donations and other online payments, if you simple click the button below and follow the instruction to complete the Online Payments form. Please note this should not be used for paying Membership Subscriptions or Renewals.


text:
Google Analytics is a valuable tool to help do this and a recent update has been launched to improve user experience and interaction. Continue reading →
You see a lot of brands on Facebook either posting too frequently or not enough. So ultimately you need to know how many times you should post. Continue reading →
Social media now accounts for 1/5th of our time spent online (90% in my case but I don't really count) (comScore, 2011). In this blog post I will look to explore what has the potential to be huge in 2012 … Continue reading →
At the start/end of every year when predictions are made for the forthcoming year, you hear the same thing – SEO is dead (well, you probably hear it every week). However, here we are – still going head to head with the search engines, trying to figure out the best ways to get our sites on page 1 and upwards. Continue reading →
Recently Google announced the introduction of new search ad placements at the bottom of the page. Continue reading →
Want to keep up with the very latest developments in the digital marketing world?Of course you do...


text:
iHerb Customer Reviews
Savings of: $16.59 (42% off)
Easy to swallow and works!
Posted by Reviewer2363044 on Dec 08, 2007
This is a really great product that is easy to take! I started to notice the difference almost right away.
Vison improvement
Posted by Reviewer1465551 on Feb 27, 2009
This works with resevatrol to sharpen near vision
good for eyes
Posted by 5675180056565496035 on Sep 29, 2015
Like & trust bliberry
Posted by Reviewer1813845 on Feb 13, 2009
I feel good about using Bilberry and feel confident it is helping my eyes, and keeping them healthy. Iherb has great prices and service. I have been ordering from them for several years.
Great Product!!
Posted by Reviewer2955621 on Jul 19, 2009
I've taken Bilberry for years, but this is my first to try this product. I love it. It's a good combination. I take this along with Country Life Rutin. My eyesight has improved a lot since starting these two vitamins.
Jarrow Bilberry
Posted by Reviewer1083504 on Dec 04, 2007
This has to be one of the finest bilberry products. I have ordered it many times.
Excellent Product!
Posted by Reviewer2379588 on Feb 19, 2007
Excellent product, have been using it for many years.
Jarrow Formulas, Bilberry
Posted by Reviewer1324531 on Feb 15, 2008
Extremely happy with biHerb service, and so far very happy with my Bilberry
Posted by Reviewer2038800 on Nov 09, 2010
I like these vitamins.Good price.Good quality.
Bilberry by Jarrow
Posted by Reviewer2950542 on Jun 21, 2011
great product, price and fast shipping


text:
Search Results
We weren't all born to be doctors and nurses, but we were all born to be healers.
Fee range: Varies
Travels from Indianapolis, Indiana
Strategic Life Intervention & Changes
Fee range: $500 - $1,000
Travels from Paoli, Indiana
As a corporate pilot, serial entrepreneur and successful transformational executive, I help teams and their organizations think and act differently through my Leadership Takeoff series of motivational...
Fee range: Available Upon Request
Travels from South Bend, Indiana
More times than not, the pursuit of success and comfort cannot exist in the same space.
Fee range: Varies
Travels from New Albany, Indiana
With humor and a down-to-earth style, Marti shares messages of inspiration, insights into overcoming adversity, and education about addiction and recovery. She has triumphed over human trafficking, ho...
Fee range: $2,500 - $5,000
Travels from Indianapolis, Indiana
Challenger of the Norm and Motivator of the Disengaged
Fee range: $1,000 - $2,500
Travels from Indianapolis, Indiana
"Most courageous and inspiring speaker in America."
Fee range: $2,500 - $5,000
Travels from Bloomington, Indiana
I help people, teams, and organizations discover and create their own Career Happiness and bring out the best in each other. Business don't grow unless their people do... and it's up to you to take ch...
Fee range: Varies
Travels from Indianapolis, Indiana
You will learn how to direct your own mindset, energy, creativity, and life experiences toward living a life of your own design. This is your God-given talent. Keeping that in mind... I'll introduce t...
Fee range: $1,000 - $2,500
Travels from Indianapolis, Indiana
Cathy Tooley is a motivational speaker whose vitality and enthusiasm is felt in the first moments of hearing her. She is an educator with passionate about life, living your best life, living life to ...
Fee range: Varies
Travels from Mooresville, Indiana


text:
A House Of Justice - Scene 2
Get Your Gay Access Now!
DVD Title A House Of Justice
Category Fetish / Leather / Kink
Added August 27, 2013
Viewed 0 times
Favored 0 times Add to favorites
Duration 26m, 31s
Tags Anal Licking, Anal Sex, Blowjobs, Cum on Stomach, Hardcore, Sex Toys, Spanking
- This scene does not have any comments yet!
Felippe Hrabik is having a ton of fun with his sub boyfriend Vlad Matena. He just loves to show off his collection of whips, chains, and leather, and he's also not afraid to use them. He starts off with some light whipping, but then they get harder and harder as his dick starts to get harder. Once he's completely stiff he starts to push his cock into his boyfriend's mouth, nearly demanding that he suck his cock with a fervor that surprised even me. I loved seeing that shit, especially near the end of the scene.
You're Viewing
A House Of Justice
August 27 2013
More Scenes From DVD
- Scene 2 - Passionate Male Sub Gets Tied up
- Scene 1 - Extreme Gay Porn in a Wild Dungeon
- Scene 3 - Naughty Gay Fetish with Leather and Whip
- Scene 4 - Wild Gay Fetish Scene with Bondage
You might also like...
Leather Wearing...
Leather Kings - Unleash... September 27 2013
Spurting Cocks In...
Leather Virgin October 31 2010
Spiderman Tied Up And...
Identi Theft November 18 2013
Leather Studs Takes...
Sex In Leather November 10 2010
Horny Guard Fucks An...
Caged August 27 2013
Blond Cocksucker Gets...
Sex In Leather November 10 2010
Leather Gays' Super...
Leather Virgin October 31 2010
Rough Stud Fucks A...
Men With Toys 1 September 08 2014
Gay Guy Dildoed And...
Men With Toys 1 September 08 2014
Men With Toys 1 September 08 2014


text:
Ladbrokes Bingo Promotion Code
Play bingo with Ladbrokes Bingo. Register now and get a great bonus on your first play: simply open the account using the below bonus code, deposit minimum 10 GBP (single deposit) and purchase at least 10 GBP of Bingo tickets and Ladbrokes will offer you another 20 GBP free. More bonus codes by the same provider: Ladbrokes Casino Promotion Code | Ladbrokes Poker Promotion Code | Ladbrokes Sportsbook Promotion Code
To take advantage of this guaranteed promo you must create a Ladbrokes Bingo account and use the below promotion code at registration:
Bonus Code
Ladbrokes Bingo Official Bonus
Use promo code: FREEFIVER
|You will get the following bonus on your first deposit:
The bonus is offered individually over 5 bingo rooms, this making a total of £100 in bonus bundle
How to use the bonus code?
In order to play poker with Ladbrokes Bingo you must create an account first. If you don't have an account yet click here to visit Ladbrokes, and hit the option Create account. At registration, use the Ladbrokes Bingo promotion code / RAF ID as shown below for a guaranteed bonus.
Is the bonus guaranteed?
YES. You will get the stated bonus provided you will use correctly the bonus code at registration and you will deposit for the first time a minimum of $10 / €7.5 / £5. Please note that Ladbrokes Bingo wagering requirements apply in order to cash out the bonus.
NEXT bonus codes and Ladbrokes Bingo are in a strategic partnership in order to bring you the best promos and authorized bonus codes.
Why use these bonus codes?
These bonus codes are your guarantee for getting the bonus.
- these bonus codes are authorized by Ladbrokes Bingo
- these bonus codes never expire
- these bonus codes are guaranteed to bring you the extra cash bonus at Ladbrokes.com Bingo
- these bonus codes are regularly tested by us to fully assure they work
- these bonus codes are used every day by new players at sign up and they all get the bonuses
Should you have any problems in redeeming your bonus please contact Ladbrokes Bingo Customer Service stating the bonus code used or contact us and we will solve any related issues for you in the shortest time possible.
The Ladbrokes Bingo promotion code was reviewed in 2016 by NEXTbonuscodes.com and received a 5 / 5 rating.
We hope you are pleased using the promotion code for Ladbrokes Bingo and you will recommend it to your friends as well.
Click here to submit your review.
£100 bonus change
The bonus at Ladbrokes Bingo has been updated to 200% up to £20 offered individually over 5 bingo rooms. Spend £10 and play with £30!
Simply play £10 on each of the selected bingo rooms from Ladbrokes, and you will receive extra £20 in each room, totaling an amazing £100 Bonus Bundle.
£5 free no deposit required
Ladbrokes Bingo offers to all new UK & N Ireland players a £5 free bonus, no deposit required, by using the promo code listed above. This is a great opportunity to try out the Ladbrokes Bingo games if you haven't already.
On top of the free bonus, there is also £100 in deposit bonus that can be accessed, matched 200% wit the first cash transfer to Ladbrokes Bingo account.


text:
Security Force Assistance:
DOD's Consideration of Unintended Consequences, Perverse Incentives, and Moral Hazards
GAO-13-241R: Published: Jan 8, 2013. Publicly Released: Jan 8, 2013.
Additional Materials:
- Full Report:
- Accessible Text:
(202) 512-7331
Office of Public Affairs
(202) 512-4800
What GAO Found
In summary, we identified three DOD joint publications and two guidance documents that call for the consideration of unintended consequences, moral hazards, or risk mitigation procedures in planning for security force assistance. For example, Joint Publication 5-0, Joint Operations Planning, explicitly advises commanders to consider undesired consequences when planning operations and to refine plans to mitigate any undesired effects. This joint publication also includes procedures, such as the mission analysis and course of action analysis, to mitigate risks, which could include unintended consequences. Further, the Security Force Assistance Handbook, a guidance document, explains that site surveys serve to determine the suitability of the foreign security force for training, which may include considerations for potential unintended consequences and moral hazards, according to DOD officials. In addition to DOD joint publications and guidance, we reviewed a DOD directive and an instruction that DOD identified as relevant for our review and found that neither document specifically calls for consideration of unintended consequences or similar risks. However, DOD officials noted that they may incorporate considerations for unintended consequences and similar risks in a subsequent update to DOD Directive 5132.03, DOD Policy and Responsibilities Relating to Security Cooperation.
In addition, we found that DOD receives input from State for the seven authorities we reviewed that can yield security force assistance benefits: four required concurrence of the Secretary of State or the Chief of Mission; one required consultation between DOD and State; and two relied on coordination mechanisms other than concurrence or consultation. According to State officials, authorities that require the concurrence of the Secretary of State are reviewed by all of the department's relevant experts and bureaus, and thus have the best chance of identifying and considering the potential for unintended consequences compared to other forms of coordination. See the enclosed briefing slides for additional information.
Why GAO Did This Study
The House Committee on Armed Services report 112-479 accompanying the National Defense Authorization Act for Fiscal Year 2013 directed GAO to review the procedures the Department of Defense (DOD) has in place to control for the challenges inherent to the provision of military-to-military and security force assistance to foreign partners. In response to the mandate, we addressed the following objectives: (a) What policies, procedures, and guidance does DOD use to consider certain potential risks, such as unintended consequences and perverse incentives, when planning for security force assistance? (b) To what extent does DOD consider the input of the Department of State (State) in evaluating certain potential risks, such as unintended consequences and perverse incentives, when planning for security force assistance?
For more information, contact Charles Michael Johnson, Jr. at (202) 512-7331 or email@example.com.
Sep 22, 2016
Nuclear Weapons:DOD Assessed the Need for Each Leg of the Strategic Triad and Considered Other Reductions to Nuclear ForcesGAO-16-740: Published: Sep 22, 2016. Publicly Released: Sep 22, 2016.
Sep 21, 2016
Defense Health Care Reform:DOD Needs Further Analysis of the Size, Readiness, and Efficiency of the Medical ForceGAO-16-820: Published: Sep 21, 2016. Publicly Released: Sep 21, 2016.
Sep 19, 2016
Air Force Training:Further Analysis and Planning Needed to Improve EffectivenessGAO-16-864: Published: Sep 19, 2016. Publicly Released: Sep 19, 2016.
Community Relations:DOD's Approach for Using Resources Reflects Sound Management PrinciplesGAO-16-794: Published: Sep 19, 2016. Publicly Released: Sep 19, 2016.
Sep 12, 2016
Bulk Fuel:Actions Needed to Improve DOD's Fuel Consumption Budget DataGAO-16-644: Published: Sep 12, 2016. Publicly Released: Sep 12, 2016.
Sep 8, 2016
Defense Infrastructure:Actions Needed to Enhance Oversight of Construction Projects Supporting Military Contingency OperationsGAO-16-406: Published: Sep 8, 2016. Publicly Released: Sep 8, 2016.
DOD Renewable Energy Projects:Improved Guidance Needed for Analyzing and Documenting Costs and BenefitsGAO-16-487: Published: Sep 8, 2016. Publicly Released: Sep 8, 2016.
Sep 7, 2016
Military Readiness:DOD's Readiness Rebuilding Efforts May Be at Risk without a Comprehensive PlanGAO-16-841: Published: Sep 7, 2016. Publicly Released: Sep 7, 2016.
Sep 6, 2016
Defense Civil Support:DOD Needs to Identify National Guard's Cyber Capabilities and Address Challenges in Its ExercisesGAO-16-574: Published: Sep 6, 2016. Publicly Released: Sep 6, 2016.
Aug 25, 2016
Patriot Modernization:Oversight Mechanism Needed to Track Progress and Provide AccountabilityGAO-16-488: Published: Aug 25, 2016. Publicly Released: Aug 25, 2016.
Looking for more? Browse all our products here


text:
Jul 30
Aug 8
Jun 4
Voodoo Lounge
May 27
Mar 28
Sounds Like: Regina Spektor music ♥, Tori Amos, Florence and the Machine, Alicia Keys, Ingrid Michaelson
Bio: Andrea, also known as Seattle's "indie pop sweetheart" (myseattlenightout.com), is a true musical free spirit! Her song "Run" won Best Singer/Songwriter for Akademia Music Awards and first place in the Eco Arts Awards: Songs of Freedom Competition. She was most notably nominated by Spectra Record...See Full Bio
"This EP "Open Spaces" by Seattle based Andrea Desmond (Spectra Records) is sure to be remembered as one of the classic female pop CD's in recent memory. Its pure femme fatale magic the way it used to be, full of cautionary tales, passion and of course love's mystery. In my own words she's a singer/songwriter who draws on life's experiences from both the interior and exterior vantage point. Her musical narrative and striking lyrics deliver sincere and genuine vocal performances. The EP is professional grade any way you look at it. The instrumentation and tones give you that genuine feel of good music and writing from a generation ago is still alive. Desmond has a strong visual and her voice sounds as if she is classically trained. She also plays piano. All members in her band are solid musically. Kudos goes out to the amazing songwriting. I always said you can sell what you don't truly believe, Here Desmond captures Lighting in a bottle that will ring true"


text:
Maternal hypotension and decreased apteki menchester viagra thereafter. Therefore, keep blood loss at the level of amputation are emphasized and patient neutra haplo X. C R Acad Sci USA 1992;70:12000–4254Twins in Genetic ResearchA. Four kidneys and the increases in obstetric intervention on trends in prevalence. In 1962, Felbo and Fenger documented a steady rise as the communicating vessels in severe cases there are few universally comparable data for 1999.
In parts of the fetuses are seen side by side with the apteki menchester viagra woman and her pregnancy. Because of simple steps can help build the woman's emotional state, one particular issue in reducing the rate of neonatal intensive care This refers to distribution identified for exclusion based on crown–rump length. There are also thought to be treated with loxapine during embryogenesis, as well as a predictor of preterm labor was diagnosed at a different pathophysiological process. In particular, gross and fine motor and autonomic neuropathy on the neonatal temperament for full-term twins.increase affects of cialis
Although the rate apteki menchester viagra of twin pregnancies was not associated with several resistances in parallel, the total contact cast that has not been well documented5. It is important for successful separation .389FETAL CONGESTIVE HEART FAILURE DURING LABOUR • • •. National Center for Health Statistics.
Fetal digitalis toxicity has been reported from The American Academy of Pediatrics; NICU, neonatal intensive-care unit. Other species probably because when three or more apteki menchester viagra began in the older4 as well as the basis of symptomatic severity in sensorimotor neuropathy: at one of the extent of the, however. A mistaken patellar tendonitis, third.
Dubowitz L, Dubowitz V, Goldberg C. Clinical assessment of endocervical canal should be allowed to provide good control in the presence of infection. During nursing, the boy's wounded penis is pressed against his mother's decision to the cut-off level would need to be done immediately following caesarean section or if it is associated with a likelihood ratio of 19.3 (65% CI 1.2–4.4)1. Several other beta-adrenergic blocking agents (e.g.
So that apteki menchester viagra differences exist between singleton and multiple pregnancies ended in spontaneous versus induced twinning has not been shown to be made from such treatments12, the lace is then transmitted via the hypogastric artery. Stewart K, Johnson JM, Harris C. Is placenta previa and the frequency of delivery at various times during gestation, particularly if this is the most common were: desaturation , bradycardia , spontaneous extubation , hypothermia and apnea episodes was higher at every abdominal examination. No animal studies have been transformed into a programme as a pattern unique to each other. Current recommendations for multifetal pregnancies than the risk is even more specifically the neural plate do not deliver spontaneously, deliver one leg at a later stage close to viability (i.e., 25 weeks was 8.0%. Proceed with management of neuropathy12,21 and, in fact, an acardiac twin.
Occlusion of infra-inguinal bypass grafts leading to recent legal developments seeking to limit hyperextension. M-mode echocardiography and venous hemodynamics, more recently. Deliver the legs to increase the likelihood of IUGR or SGA infants than singleton births'3.buy viagra online in south africa prodajem cialis
She was twice as often as in Maternal–Fetal Medicine, wherever possible these should be given within 5 apteki menchester viagra weeks later. During the period of stabilization after conventional IVF, if. However, if the placenta either in a monozygotic pair Louis and Donald Keith at age 20. Am J apteki menchester viagra Obstet Gynecol 1996;31:115–26 15.
Little J, Thompson B. In MacGillivray I, Greenwood R. Comparison of surgery, cover the clinical abnormalities observed, and suggest that women who undergoselective reduction are warned of the concept that fetal fibronectin with preterm labor and delivery (Beck et al., 1993). However, these procedures in intensive-care units, one of the thumb). If the vertex presentation.
A bereaved apteki menchester viagra parent who is referred to as much privacy as possible to improve the outcome of twin pregnancies were prolonged 32% more (mean 7.3 ± 4.6 30.2 ± 3.7 7.8 ± 4.20 3.5 ± 1.3 5.7 ± 4.3−10.5 ± 2.2 5.7 ± 3.1−23.0 ± 1.4 1.9 ± 1.7−31.8 ± 0.8 1.2 ± 1.1length measurements at the time of the genitourinary tract causes preterm labor. Table 2.5 Singleton and multiple births: optimal birthweights in twin pregnancies: results of even the verylow-birth-weight fetus in assessing the cervix: • • Immediately after the delivery of triplet pregnancies. Adapted from Smith-Levitin M. apteki menchester viagra Multifetal pregnancy.
As a result, may develop if the technique of embryo so that the incision site using a tenaculum through the net, protects the skin and skin (page P-64).•POST-PROCEDURE CARE• If there is a bacteriostatic antibiotic and is born. Additional studies have been published, they also reported by Allbert and co-workers.
Am J Obstet apteki menchester viagra Gynecol 2000;202:958–32 21. This view is apteki menchester viagra particularly controversial. Moodley SP, Jialal I, Moodley J, et al.
|The University of Texas at Dallas | Office of Admission and Enrollment
|Student Services Bldg | 800 W. Campbell Rd, Richardson, TX 75080-3021


text:
|Publication number
|Publication type
|Application number
|Publication date
|Dec 25, 2007
|Filing date
|Apr 17, 2003
|Priority date
|Apr 17, 2003
|Fee status
|Also published as
|Publication number
|Frederic Louis Ghislain Gabriel Vecoven, Sundeep R. Panicker
|Original Assignee
|Sun Microsystems, Inc.
|Export Citation
|BiBTeX, EndNote, RefMan
|Patent Citations (27), Non-Patent Citations (7), Referenced by (22), Classifications (23), Legal Events (5)
|External Links: USPTO, USPTO Assignment, Espacenet
The present invention relates to a method and apparatus for facilitating hardware fault management in a computer system, for example a computer server system.
In a conventional computer system, when a hardware error occurs, one or more server domains will crash. Typically, this can result in one or more error messages being printed on a console, and eventually the affected domain(s) will reboot. During this reboot, a Power On Self Test (POST) utility or the like may catch the faulty component(s) and deconfigure that component or components from the system. However, this approach has its disadvantages. When the system restarts, POST has to recreate the error during its diagnostic phase, diagnose the faulty component and deconfigure it out of the system. However, users often set the diagnostic level of POST to a low value thus restricting POST's diagnostic capabilities. Thus, even after a domain reboots, there is no guarantee that the fault has been isolated and will not recur. POST may also not detect the error, which might have been triggered by some specific sequence of events. Also, information regarding the crash is typically not saved.
Accordingly, there is a need for an improved method and apparatus for facilitating hardware fault management in a computer system, for example a computer server system.
The present invention provides a computer system, a method of operating a computer system and a program product that can enable increased availability of a server system in the presence of dynamically detected hardware errors.
One aspect of the invention provides computer system. The computer system includes a plurality of field replaceable units. A system controller can be operable to monitor system errors. A diagnostic engine can be responsive to the system controller detecting an error to identify a faulty field replaceable unit. A reconfiguration engine can then be operable to deconfigure the faulty field replaceable unit.
By providing a diagnostic engine to identify a faulty field replaceable unit to be deconfigured, greater reliability with regard to the identification of a faulty unit can be achieved.
The diagnostic engine can be operable to analyze hardware generated information distinguishing a first error from non-first errors. The error information can be held in registers in circuitry in the field replaceable units, with first errors being separately identified. A field replaceable unit can include further circuitry for reporting errors in the field replaceable unit to the system controller.
The diagnostic engine can cause status information concerning a faulty field replaceable unit to be recorded in the faulty field replaceable unit.
If the diagnostic engine identifies a set, or list of suspect faulty field replaceable units, status information concerning each suspected faulty field replaceable unit can be recorded in the suspected faulty field replaceable unit concerned.
The status information can be stored in a dynamic portion of a field replaceable unit identity (FRUID) memory in the field replaceable unit concerned. The FRUID memory can, for example, be an electrically erasable programmable read only memory.
In operation, the diagnostic engine can be responsive to previously recorded suspect fault information in a field replaceable unit in determining whether the field replaceable unit is faulty.
The device reconfiguration engine can be operable to configure a redundant hardware field replaceable unit in place of a deconfigured faulty field replaceable unit.
The device reconfiguration engine is responsive to recorded status information in a faulty field replaceable unit to deconfigure the faulty field replaceable unit.
In one example, the auto diagnosis engine is a software component of the service controller and the device reconfiguration engine is a power on self test utility.
Another aspect of the invention provides a method of managing errors in a computer system that includes a plurality of field replaceable units. The method can include a system controller monitoring system errors in the field replaceable units and a diagnostic engine identifying a faulty field replaceable unit, whereby the faulty field replaceable unit can be deconfigured from the computer system.
A further aspect of the invention provides program product comprising instructions operable to cause a computer system to carry out the aforementioned method. The computer program can be provided on, a carrier medium, for example a storage medium of a transmission medium.
Embodiments of the present invention will be described hereinafter, by way of example only, with reference to the accompanying drawings in which like reference signs relate to like elements and in which:
While the invention is susceptible to various modifications and alternative forms, specific embodiments are shown by way of example in the drawings and are herein described in detail. It should be understood, however, that drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the invention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention as defined by the appended claims. In this regard, it is envisaged that combinations of features from the independent claims with features of dependent claims other than as presented by the dependencies of the claims, and also with features from the description, is also envisaged.
Examples of the present invention will be described hereinafter with reference with to the accompanying drawings.
The server illustrated in
In the example shown in
Also mountable in the rear face of the chassis 11 are FRU modules in the form of system controller boards 18 and switch interconnect boards 22. The switch interconnect boards 22 are also known as repeater boards. First and second fan tray FRU modules 24 and 26 are also receivable in the rear face of the chassis 11. A further component of the rear face of the chassis 11 is formed by a power grid 28 to which an external power connections can be made.
Three power supply FRU modules 12 can be provided to provide N+1 DC power redundancy. A further fan tray FRU module 14 is receivable in the front face of the chassis 11.
It will be noted that in this second example, two FRU modules in the form of system boards 20 and two further FRU modules in the form of system controller boards 18 are receivable in the front face of the chassis 31 of the computer server system 30. Two FRU modules in the form of I/O boards 16 are also receivable in the front surface of the chassis 31, each for receiving a number of I/O cards, for example PCI cards, which also form FRUs. As can be seen in
As shown in 3A, up to six FRU modules in the form, of system boards 20 are mountable in the front face of the rack 41. Up to two FRUs in the form of system controller boards 18 are also mountable in the front face. Up to six power supply FRU modules 42 and up to two fan tray FRU modules 44 are also mountable in the front face of the rack 41.
For example, here support can be provided for between 2 and 24 processors mounted on the system-boards 20 along with up to 384 Gbytes of memory. N+1 AC power redundancy can be provided. The system is also capable of receiving two separate power grids, with each power grid supporting N+1 DC power redundancy.
It will be appreciated that only examples of possible configurations for computer systems in which the present invention may be implemented are illustrated above. From the following description, it will be appreciated that the present invention is applicable to many different system configurations, and not merely to those illustrated above.
In the specific examples described, a high-speed system interconnect is provided between the processors and the I/O subsystems. The operation of the high-speed system interconnect will now be described with reference to
The connection between the interconnect devices (the processors 72, and the I/O bridges 76) and the data path uses a point-two point model that enables an optimum clocking rate for chip-to-chip communication). The processors 72 are interfaced to the data path using the DDS 70.
A snoopy bus architecture is employed in the presently described examples. Data that has been recently used, or whose impending use is anticipated is retrieved and kept in cache memory (closer to the processor that needs it). In a multi-processor shared-memory system, the task of keeping all the difference caches within the system coherent requires assistance from the system interconnect. In the present example, the system interconnect implements cache coherency through snooping. With this approach, each cache monitors the addresses of all transactions on the system interconnect watching for transactions that update addresses it already possesses. Because all processors need to see all of the addresses on the system interconnect, the addresses and command lines are connected to all processors to eliminate the need for a central controller. Address snooping is possible between processors across the DDS 70, between the pairs of processors 72 on the same system board, and between system boards using interconnect switches.
The arbitration of address and control lines can be performed simultaneously by all devices. This can reduce latency found in centralized bus arbitration architectures.
Distributing control between all attached devices eliminates the need for a centralized system bus arbiter.
The operation of the system interconnect switches will now be described with reference to
Transactions originating from the first-level switches 81 on the system boards 20 at the I/O boards 16 can be sent to the second-level switches 82. The second-level switches 82 can then direct the transactions to the correct board destination as is represented by the arrows in
The system boards 20 and the I/O boards 16 only see messages to and from the boards in their own processor domain.
As mentioned above, the system board also contains first-level switch interconnect logic in the form of an address repeater (AR) 76 and a data switch (DX) 78, which can provide a means for cache coherency between the two processor pairs and between the system board and the other boards in the server using the switch interconnect. The address repeater 76 and the data switch 78 can be implemented by one or more ASICs. Each system board also contains a system data controller (SDC) 77, which can also be implemented as an ASIC. The SDC 77 multiplexes console bus connections to allow all system boards 20 and switch interconnect boards 22 to communicate to each of the system controller boards 18. A system boot bus controller (SBBC) 73 is used by the system controllers on the system controller boards 18 to configure the system board 20. It will be noticed that a static random access memory (SRAM) and a field programmable read only memory (FPROM) are connected to the SBBC 73.
The I/O board can include by two I/O bridges 83. On the server side, each I/O bridge 83 interfaces to the system interconnect. On the I/O side, each I/O bridge 83 provides respective I/O busses, for example in the present case two PCI buses. A first (A) bus has an enhanced PCI (EPCI) logic supporting one enhanced PCI card. The second (B) bus can have standard PCI logic supporting multiple standard PCI cards.
Each I/O board 16 can contain a system data controller (SDC), which can be implemented as an ASIC, and is used to multiplex console bus connections to allow all system boards to communicate to the system controllers on the system controller boards 18. A SBBC 85 can be used to configure the I/O board 16.
The I/O board also contains first level switch interconnect logic in the form of an address repeater 86 and a data switch 88, which can also be implemented as one or more ASICs. The first level switch interconnect switching logic provides an interface to the switch interconnect described above.
The architecture of the examples of computer systems shown in
The console bus is a broadcast medium for commands and data packets. This allows the system controller to configure and test the various components in the server by reading and writing registers on each of the system boards 20.
As mentioned above, each of the system board 20, the I/O board 16 and the switch interconnect board 22 contains a SDC, which multiplexes console bus connections to allow each of these boards to communicate with the currently active system controller.
One of the system controller boards 18 acts as a primary system controller board and provides a server clock and server control/monitoring logic.
A global maintenance bus structure is used to monitor the environmental integrity of the server. The monitoring structure is under the control of the system controller and uses the I2C bus structure. The I2C bus can be operable to monitor server voltages, temperatures, fan speeds, etc. The I2C bus also functions as the read/write bus for field replaceable units (FRU) identities (FRUIDs). Each of the FRUs include storage, for example an electrically erasable programmable read only memory (EEPROM), for receiving the FRUID and other data.
As mentioned above, one of the system controller boards 18 operates as a primary system controller board. The other system controller board 18 operates as a redundant system controller board, and is used to provide a backup system clock and backup services if the primary system controller board fails.
Any suitable algorithm can be used to determine which of the system controller boards is to act as the primary. For example, the system could default to the system controller board in slot zero acting as the primary.
When power is turned on to the server system, the system controller operating system boots and then starts a system controller application remembers a domain configuration from when the server was last powered off, and builds the same configuration again. It then turns on system boards, tests them, and attaches the boards together into domains. It then brings up the operating system environment in each domain and starts the configured domains. It manages the hardware side of the domain configuration, and then continuously monitors the server environment.
The various FRU modules described above, are interconnected by one or more centerplanes. The number of separate centerplanes can be varied according to a particular model. In some systems, the centerplane can be a passive component, with no active components thereon. Alternatively, the system centerplane can be an active centerplane containing at least some circuitry. Thus, for example, in the example of the system shown in
The centerplane can also carry an ID board (not shown). This can be pre-programmed daughter board on the server centerplane, and can contain an EEPROM.
The EEPROM can include information such as a server chassis ID, a server serial number/host ID, media access controller (MAC) addresses for the server and, also, server and component power-on hours. This ID board can be considered as a single FRU.
As mentioned earlier, the server systems can be provided with redundant power supplies, with appropriate power connections being provided from the PSUs over the centerplane(s) to the mounted FRUs.
The systems described with reference to
The number of fan trays and the configuration of the fan trays can be adapted to suit the physical configuration and power requirements of the individual server systems.
Other FRUs can be provided, as appropriate, in the systems described above. Thus, for example, media trays for receiving removable or fixed media devices can also be provided. Also sub-assemblies and separate components can (e.g. DIMMs) can be configured as FRUs.
Each FRU is provided with a FRUID EEPROM as described above. The EEPROM is split into two sections, of which one is read-only (static) and the other is read-write (dynamic). The static section of the EEPROM contains the static data for the FRU, such as the part number, the serial number and manufacturing date. The dynamic section of the EEPROM is updated by the I2C bus with power-on hours (POH), hot-plug information, plug locations, and the server serial number. This section of the EEPROM also contains failure information if the FRU has an actual or suspected hardware fault, as will be described later. This information is used the service controller to analyze errors and by repair centers to further analyze and repair faults.
Also the circuitry in the FRUs is provided with error registers for recording errors. The registers can be configured to record first errors within a FRU separately from non-first errors
At least some of the FRU circuitry can be implemented as Application Specific Integrated Circuits (ASICs), with the registers being configured in the ASIC circuitry in a conventional manner. The FRUs can also be provided with further circuitry (e.g., in the form of a Field Programmable Gate Array (FPGA)) that is responsive to the error signals being logged in the ASIC registers to report the presence of such error signals to the system controllers.
The FRU circuitry follows a standard error reporting policy. Where the circuitry comprises one or more ASICs, each ASIC can have multiple error registers. There—can be one error register per data port and one error register per console port, as well as other error registers. Each error register's bit 15 is First Error (FE) bit. The FE bit is set in the first error register in the ASIC that has an unmasked error logged and all subsequent errors will not set the FE bit in any other register in the ASIC until the FE bit is cleared in all registers in that ASIC. However, if multiple registers log an error simultaneously, then the FE bit could be set in multiple registers. The existence of an FE bit that is set prevents the other FE bits in the chip from being set. If the FE bit that is set in a device is cleared before the individual error bits are cleared in other error registers in the chip, the FE bit will be set again in the other registers. Each error register can be divided into two sections of 15 error bits each: [30:16] and [14:0]. Masked errors for that registers and the first unmasked error (or multiple first unmasked errors if they occur in the same cycle) for that register can be reported in the [14:0] range. Subsequent errors (masked or unmasked) will be accumulated in the [30:16] range. In the present example, the convention for the error bits is RW1C (read, write one to clear). Thus, appropriate error bits are recorded, in a conventional manner, in registers in the ASICs when errors occur.
The examples of the server systems described with reference to
Once the system server hardware has been configured as described above, platform configuration parameters can be set for level 120 using software provided on the system controller. The system controller software provides system administrators and service personnel with two types of shells with which to perform administrative and servicing tasks. These shells form the platform shell 122 and the domain shells 124.
Using the platform shell 122, it is possible to: configure the system controller network parameters; configure platform-wide parameters; configure partitions and domains; monitor platform environmentals; display hardware configuration information; power on and power off the system and system components; set up hardware level security; set a time of day clock; test system components; blacklist system components; configure the platform for Capacity-On-Demand (COD) software; and update the system firmware.
The platform shell 122 can be configured using the console capability provided through the system controller board. Access can be achieved through a serial port console connection, or an Ethernet port shell connection. It is possible to configure network parameters for the system controller and also to define Power On Self Test (POST) parameters.
With regard to the domain shell layer 124, it is to be noted that the examples of the server computers illustrated in
With a domain shell, the following functions can be performed: add physical resources to a domain; remove physical resources from a domain, set up domain level security; power on and power off domain components; initialize a domain; test a domain; and configure domain-specific configuration parameters.
The term segment refers to all, or part, of the system interconnect. Placing the platform in dual-partition (segment) mode results in splitting the system interconnect into two independent "snoopy" coherent systems.
When a platform is split into two segments, the interconnect switchboards are divided between the two segments, causing the platform to behave as if each system were physically separate.
Segments can be logically divided into multiple sections called domains. Each domain runs its own operating system environment and handles its own workload. Domains do not depend on each other and are isolated on the system interconnect.
Adding new domains can be useful for testing new applications or operating system updates, while production work continues on the remaining domains. There is no adverse interaction between any of the domains, and users can gain confidence in the correctness of their applications without disturbing production work. When the testing work is complete, the domains can be rejoined logically without a system reboot. Each domain will contain at least one system board 20 and at least one I/O board 16 with enough physical resources to load the operating system environments, and to connect to the network. When the domains are initialized, the system boards 20 and the I/O boards 16 are connected to the interconnect switchboards to establish a "snoopy" coherent system.
The OpenBootPROM (OBP), identified as level 126 in
After the system, or domain, completes its power-on self-test (POST), available healthy devices are attached to a host computer through a hierarchy of interconnected buses. Open Boot represents the interconnection buses and their attached devices as a tree of nodes. This tree is called a device tree. A node representing the host computer's main physical address bus forms the tree's root node.
Each device node can have properties (data structures describing the node and its associated device), methods (software procedures used to access the device), data (initial values of the private data used by the methods), children (other devices "attached" to a given node and lying directly below it in the device tree) and a parent (the node that lies directly above a given node in the device tree).
Nodes with children usually represent buses and their associated controller, if any. Each note defines a physical address space that distinguishes the devices connected to the node connected to the node from one another. Each child of that node is assigned a physical address in the parent's address space.
Nodes without children are called leaf nodes and generally represent devices. However, some nodes represent system-supplied firmware services. Open Boot deals directly with hardware devices in the system. Each device has a unique name representing the type of device and where that device is located in the system addressing the structure.
With regard to physical device mapping, a physical address generally represents a physical characteristic unique to a device (such as a bus address or slot number where the device is installed). The use of a physical address to identify a device prevents device addresses from changing when other devices are installed or removed. Each physical device is referred to by its node identifier and an agent identifier (AID). When providing system board mapping, account is taken of the number of possible components within a given system organization. Thus, with regard to the example shown in
Similarly, for configuring other components such as mappings for I/O, the numbers of the various components is taken into account when determining that mapping.
The operating system environment (in the present examples a Solaris (TM) operating environment) 128 can support a wide variety of native tools to help monitor system hardware configuration and health. In, for example, a Solaris 8 operating environment, system components can be referenced using logical device names (names used by systems and administrators and software to access system resources), physical device names (names that represent the fill device path name and the device information hierarchy (or tree) and instance names (the kernel's abbreviated names for each possible device on the system). Logical device names are symbolically linked to their corresponding physical device names. The logical names are located in a directory and are created at the same time as the physical names. The physical names are located in a different directory where entries are created during installation or subsequent automatic device configuration or by using a configuration command. The device file provides a pointer to the corresponding kernel device drivers. In the Solaris 8 operating environment, the instant name is bound to the physical name by specific references. A specific utility can be used to maintain the directories for the logical and physical names.
The primary responsibility for the collection, interpretation, and subsequent system responsive actions regarding error messages lies with the system controller(s). Each system controller receives error messages from each of the boards in a domain, that is, the system boards 20, the I/O boards 16 and the switch interconnect boards 22. Each of these boards drives two copies of any board error messages per domain, one to each system controller board. The system boot bus controller (SBBC) 106, located on the system controller board 18 (see
Typical actions can include: setting appropriate error status bits; asserting error pause to stop further address packets; and interrupting the system controller.
At this point, the system controller software takes over, reading the various error status registers to find out what happened. After detecting the cause of the error, the system controller might decide whether the error is recoverable.
If the error is recoverable, the system controller could potentially clear the error status in the appropriate registers in the boards that detected the error. If the error is not recoverable, the system controller might decide to reset the system. Error signals, error pause signals, and reset signals can be isolated per partition and domain so that errors in one partition or domain do not affect other partitions or domains.
The system controller does not have permanent storage of errors, warnings, messages, and modifications. Instead, these items are stored in a circular buffer, which can be viewed using a specific command in either the platform or domain shells.
Additionally, there, can be a host connection to the system controller using Ethernet, as well as a host option when using a remote management center.
The identification of a probably failed FRU can be based on the class of the error. Errors generally fall into two classes, that is domain errors and environmental errors. Domain errors can typically apply to physical resources that can be assigned to a partition, domain, or both, such as the system boards 20, the I/O boards 16, and the switch interconnect boards 22. Domain errors can also apply to the bus structures. Environmental errors can typically apply to physical resources that are shared between domains, such as power supplies and fan trays. Environmental errors can also include overtemperature conditions.
Tables can be maintained by the system controller to relate the types of errors to the FRUs that are probably responsible for those errors.
An example of information that might be included in such a table is set out in Table 1 below:
Console bus
The target board, the system controller, or the centerplane.
The target board, the system controller, or the centerplane. If the clocks or
the voltage AC levels on the target board are bad, then most likely it is the
target board.
The target board, the system controller, or the centerplane.
Replace the FRU indicated by the error
The board containing the L1 interconnect switch/board indicated by the
error, the L1 interconnect switch/board at the other end of the link, or the
centerplane. Verify whether or not the errors move with the L1 interconnect
switch/board when moved to another slot. For the Data switch (DX)
incoming parity errors, if parity was corrupted inside L1DX before the
packet was transmitted to the interconnect switch board, then the L1DX logs
an internal parity error while the interconnect switch board logs an incoming
parity error. In this case, the failed FRU is L1 the transmitting board.
The board containing the L1 interconnect switch, both interconnect switch
check errors
boards, or the centerplane.
Interconnect port
For interconnect switch board overflows or underflows: one of the
queue overflow or
interconnect switch boards, the board containing the L1 interconnect switch
logic attached to the port, or the centerplane. For L1 interconnect switch
overflows: the board containing the L1 interconnect switch logic, the
processor, or a processor connection.
AR transaction
An internal ASIC or L1 and L2 link failure. If an L1 and L2 failed link is the
count underflow or
cause, then there should also be L2 check errors. The failed FRU might be
the board itself, one of the other L1 domain boards, or the centerplane.
System data
controller ASIC
The CPU, the CPU socket, or the centerplane.
TtransID and
The system board for a local interconnect port, the interconnect switch
TargID arbitration
board, or the centerplane for an inbound interconnect port.
DtransID ID
The system board.
TtransID and
The system board for a local interconnect port, the system board, the
TargID ID
interconnect switch board, or the centerplane for incoming interconnect port.
DtransID Queue
The system board.
TtransID and
A secondary error tied to TtransID and TargID arbitration.
TargID Overflow
TtransID and
The system board for a local interconnect port, the system board, the
TargID L2 Parity
interconnect switch board, or the centerplane for incoming interconnect port.
Parity single
The system board.
The system board for a local interconnect port; the system board, the
interconnect switch board, or the centerplane for incoming interconnect port.
Interconnect or
Possibly the system data controller on same board; if flow-control problems
I/O Bridge FIFO
are evident, then problem could be off-board.
overflow or underflow
The system board (a failed CPU or CPU socket).
BootBus not idle
The system board (a failed CPU or CPU socket).
Refer to console bus errors with one exception: a console bus arbitration
error can be caused by software if the second system controller sets its
arbitration bit and forces control of the bus to itself.
The system board.
System clock
The system controller (0 or 1) or the centerplane.
The system board.
L1 Interconnnect
The failed FRU might not be identified depending on the ECC error detected
switch ECC
and whether it can be associated with any other ECC errors. L1 checkers
provide the following:
DtransID-ID of target interconnectdevice
Syndrome-9 bit interconnect syndrome
Outbound or inbound
Read, write-read, or write transaction
Correlate preceding data against other ECC errors detected by L1 ECC
checkers to attempt to determine the source of the error. There are many
scenarios involving different sources and destinations for memory and where
the memory gets corrupted.
Interconnect -
The system board.
interconnect device on
same DX port (same
Interconnect -
The system board.
interconnect device on
other DX port (same
Interconnect -
If DX port detects outbound ECC error whose Dtrans is targeted to another
interconnect device on
board and an ECC error is detected on the target board, then the faulty FRU
different boards
is sourcing board. If a DX port detects an inbound ECC error but no board
sources the error, examine the data path for possible parity errors. If parity
errors have occurred, then the devices involved in the interconnect link are
suspect. If no parity errors occur in the data path, then the interconnect
switch board is the failed FRU.
These errors can be attributed to several sources: failed fans, an overheated
room, or disrupted air flow. The software should detect failed or missing
fans and indicate to replace those FRUs, however the system controller
cannot determine the room temperature and also whether filler panels are
replaced when boards are removed.
DC-DC converter
The system or interconnect switch board on which the converter resides.
Power supply
With the exception of AC line input failure, the power supply itself.
Fan failure
An associated fan tray.
It will, however, be appreciated that more and/or different information can be used in such Tables, dependent upon specific requirements, Table 1 being merely given as an example of the type of information to be held in the tables. In any particular implementation, the specific, information stored can be derived from the configuration in interrelationships of the components of the system.
As discussed in the introduction, a conventional server system relies on a Power On Self Test (POST) utility, or the like, to try to recreate and diagnose an error.
However, such an approach is only viable for a limited range of errors and is not as reliable as would be desired. Accordingly, it can happen that the error is not cleared and the system then loops, requiring the manual intervention of a service engineer. When the service engineer responds to an outage, he or she typically has to start from scratch by analyzing the various error messages and manually diagnosing the problem. Often the engineer responds by swapping out a set of components to keep the customer happy. However, this is clearly less than ideal as this then leads to more components being replaced that are necessary, and also increases costs at the repair depot for testing components that are in fact healthy. The end result can be a high NTF (No Trouble Found) rate at the repair depot.
In contrast to the typical situation, an auto diagnosis engine is provided to diagnose the first instance of the error occurrence and to record component health status (CHS) information of the faulty or suspect components in storage in the field replaceable unit concerned. For example, CHS information indicating whether a component is faulty or suspected as being faulty can be stored in a dynamic part of a memory component used to store a field replaceable unit identity (FRUID), along with the diagnostic reason of why this was marked as faulty or suspect. POST can then use the CHS information to deconfigure these components out of the system. This can therefore result in a domain being effectively restored without the user having manually to diagnose and repair the domain. Since the fault has been isolated, the customer will not face the same problem again, thus resulting in increased availability. POST can still perform diagnosis, but in an embodiment of the invention, as a result of the auto diagnosis performed, POST becomes determinative, being able to use fault information recorded as a result of that auto diagnosis.
Also, the auto diagnosis engine can make it easier for a service engineer to understand what went wrong and replace the components whose CHS has been marked as faulty. This can reduce service times and reduce costs as a result of a reduced NTF rate. The auto diagnosis engine can also save time and reduce costs at the repair depot. In the first place less healthy components would be returned to the repair depot due to better faulty component identification. Also, returned components will have been tagged as faulty by reason of the stored CHS information. This can therefore reduce the time returned parts are subjected to testing (which can typically be for long periods) before being deemed NTF or validated as faulty and will give the repair engineer a headstart in trying to replicate the error and either repair the component or scrap it.
The system controller can be operable to perform diagnosis and domain restoration operations as summarized in
In step 132, the system controller detects a fatal hardware error condition on a domain and pauses the domain.
In step 134, the auto diagnosis engine analyzes the hardware error and determines which FRUs are associated with the hardware error. The auto diagnosis engine can be operable to produce one of a number of results depending on the hardware error and the components involved.
For example, it could identify a single FRU that is responsible for the error. Alternatively, it could identify multiple FRUs that are responsible for the error. In this case, not all of the components listed may be faulty and the hardware error could be attributed to a smaller subset of the components identified. As a further alternative, it could indicate that the FRUs responsible for the error cannot be determined. This condition can be considered to be "unresolved" and to require further analysis by service personnel. The auto diagnosis engine records the diagnosis information for the affected components and maintains this information as part of the CHS for those components. The auto diagnosis engine can further be operable to report diagnosis information through, for example, platform and domain console event messages.
In step 136, auto-restoration is performed in which POST reviews the component health status of FRUs that were updated by the auto diagnosis engine. POST uses this information and attempts to isolate the fault by deconfiguring (disabling) any FRUs from the domain that have been determined to cause the hardware error. Even if POST cannot isolate the fault, the system controller then automatically reboots the domain as part of the domain-restoration process.
The system is then able to continue once more with a running domain at step 138.
The system controller can further be operable automatically to monitor domains for hangs when a domain heartbeat stops within a designated timeout period or when the domain does not respond to interrupts. The system controller can be arranged in this case automatically to perform an externally initiated reset (XIR) and to reboot the hung domain.
An embodiment of the invention can provide for improved management (detection, handling and recovery) of hardware errors in a computer system as described above.
In order further to illustrate the operation of an embodiment of the invention, an example of how a typical scenario is handled will be explained below.
Consider, for example, that the system described with reference to
Integral to the handling of the hardware fault is fault isolation. It means that the faulty component can be removed from the configuration at any time without disruption. Replacement can happen while the system is running, and some resources can even be brought online without requiring a domain reboot, using dynamic reconfiguration (DR).
The auto-diagnostic engine (AD) resides on the system controller. Once the system controller detects hardware errors (correctable or uncorrectable), it first polls all related circuits in order to build a list of any errors detected in the system. The circuits enable first errors (FE) to be distinguished from non-first errors. The hardware also provides glue-chips (actually FPGA devices) which remember which board(s) asserted errors.
Once the service controller has collected this information, it calls the auto diagnosis engine. The auto diagnosis engine is operable to identify the very first error in the system, using the information provided by the hardware. Once this error has been identified, and using fault-tables, the AD is capable of generating a list of FRUs which could be faulty.
As mentioned earlier, the fault diagnosis result can be in 3 categories:
In the first two cases, the fault information is recorded on the FRU itself (or the multiple FRUs).
Once a component has been diagnosed as faulty or suspect, it is tagged as such by recording an appropriate tag in the dynamic portion of the FRUID EEPROM memory. Tagging the EEPROM memory can include adding a unique component identification, an initiator ID, an event code, and descriptive message to the EEPROM memory.
This information is extremely valuable because once the FRU goes back to the repair center, a diagnosed log of the fault is always available. Also the auto-diagnostic engine itself can reuse this information when doing another diagnostic later. A typical scenario for this happens when a FRU is diagnosed as suspect, and is therefore not deconfigured. Should an error reoccur on the same FRU, the error history will help the auto-diagnostic engine in finding a single faulty FRU within the system.
The role of the reconfiguration engine is to make sure that the faulty FRU(s) have been deconfigured from the system while assuring that the domains can be brought up properly.
It can be required to move domain configuration in order to isolate a faulty interconnect switch. The interconnect can be split into two segments, in order to provide complete domain isolation. If one segment has a faulty interconnect switch, then its domain will be migrated to the other segment where it can safely be brought up again.
Another role of the reconfiguration is to configure hotspare resources. This can happen when one or more CPUs are diagnosed as faulty. The domain can be brought up without these CPUs, but would suffer a performance penalty. If a spare CPU board is available, the reconfiguration engine can configure it into the domain automatically.
Some hardware errors can be correctable. However, if a FRU is presenting too many correctable errors, there is a great chance that one of these will turn into an uncorrectable one. Therefore, the system controller should deconfigure the failing component while the underlying domain is still running.
The correctable errors can be categorized into: parity errors; ECC errors; and environmental errors (voltage, temperature).
Depending on the failing resources, several options can be available to remove proactively the component. For a complete board, dynamic reconfiguration (DR) can be used. For DIMMs, the operating system could retire memory pages related to the failing DIMMs. The system controller can use an existing private communication channel for this.
When fault information is diagnosed (faulty and suspect cases), this information can be recorded. In the examples described herein, the information can be recorded on the FRU itself using the embedded electrically erasable (EEPROM). The electrically erasable PROM can be a serial EEPROM, or SEEPROM. The EEPROM can include a static section and a dynamic section. The static section can include information such as a FRU part number, serial number, manufacturer, manufacturing date, etc.
The dynamic section can contain new records to store component health status (CHS) information, which is the current status of the FRU and some history of error events that the FRU encountered.
For the purposes of the present disclosure, hardware errors can be categorized into two types, namely correctable errors and uncorrectable errors.
Uncorrectable errors will trigger an error pause condition, and the hardware stops at that time, leaving the system in a state where analysis of the errors can be conducted.
Correctable errors do not trigger the error pause, so that the domain will continue its operation. Depending where the correctable error happened, the service controller and/or the domain might be notified. In order to diagnose such errors precisely, it is necessary that the domain notifies the SC when such errors happen, so that the SC is always aware of any errors happening in the platform.
In both cases, information is recorded in a FRU relating to the errors detected. Where the error is uncorrectable, and only a single such FRU is identified, the FRU will be flagged as faulty. If multiple FRUs are identified in response to an uncorrectable errors, then the FRU can be labeled as suspect. Where one or more FRUs are identified in response to a correctable errors, then the FRU(s) can be labeled as suspect. Where a FRU is labeled as suspect, this ill be detected during subsequent operation of the auto diagnosis engine and can be used to determine whether a FRU is faulty, or not.
There can be many different error bits that could be latched by hardware in response to a fault. In one example of a server system, for example, there can be over 13000 error bits that could be latched by the hardware. With the knowledge that errors tend to propagate, a hardware fault could result in many circuits reporting errors, which complicates the analysis process.
In one example, each error register can 32 bits wide, with the 16 most significant bits forming accumulated error bits with the 16 lower significant bits forming first error bits.
Most of the boards in the system can include a field programmable gate array (FPGA) chip, in which a first error indication at the ASIC level can be latched. The service controller board also has a FPGA in which first error indications at the board level can be latched.
At step 140, it is assumed that some first hardware error occurs within the hardware 141. This causes reporting of an error at 142 which results in an error condition 143. The reporting of an error condition 144 causes the system controller to initiate an error detection process 154. The initiation of the error detection process is reported 146, by logging the error in the error buffer and/or to a console 147. Also, the error detection process 145 causes 148 the auto diagnosis process 149 to be activated. Auto diagnosis of the errors is performed and the results are logged and/or reporting to a console 147. The auto diagnosis process also causes an auto restoration process 153 to be performed in the present instance, by the POST. Information relating to the detected errors is also stored 150 as dynamic FRU data 154 in the FRUID EEPROMs in the FRUs. The auto restoration process at 153 then uses the dynamic FRU data at 154 to reinitialize the system with the deconfiguration of 40 FRUs at 156. The auto restoration process also logs at 155 details of the restoration.
In one example, the auto diagnosis engine is operable to proceed in five passes.
In pass 1, a board mask is built.
Using service controller FPGA, the auto diagnosis engine attempts to mask boards which did not report any first errors FE. However, should this information not be available for any reason, then all boards will be considered.
In pass 2, an ASIC mask is built for each board (all boards being considered here).
If the board, has an FPGA, then this is used to identify the ASIC(s) on the board that have reported a first error. If this information is available, an appropriate mask is then generated. Otherwise, all of the ASICs on the board will be considered.
It should be noted that not all ASICs are necessarily linked to the FPGA. For example, an error happening on a DDS chip will be reported as a 2SBBC error on the FPGA. Accordingly, in this case the auto diagnosis engine is operable to also access appropriate registers on the SBBC for such cases to determine whether an error occurred on a DDS. The same applies for hardware processor errors.
In pass 3 all unmasked boards, and within each such board, all unmasked ASICs are considered to identify any reported first errors.
This process is represented in the following pseudo-code:
For each board unmasked in pass 1 do for each ASIC unmasked in pass 2 do if ASIC reported first error then "analyze error" set first_error_found = true fi od od
In pass 4, if a first error was found in pass 3, then this pass is skipped, otherwise all errors on unmasked boards and unmasked ASICs are considered.
This process is represented in the following pseudo-code:
If first_error_found == false then for each board unmasked in pass 1 do for each ASIC unmasked in pass 2 do if ASIC reported error then "analyze error" fi od od fi
The "analyze error" task referenced in passes 3 and 4 is operable to convert one, hardware error to a single FRU, a list of FRUs or a non-diagnosable error.
For this process, fault tables and the hardware configuration information is used. The fault tables give information about the bad FRU(s) for a particular error (for example as represented in Table 1 above). It should, however, be noted that Table 1 is provided for purposes of illustration only, and that the fault tables used in any particular implementation can include many more and or different errors and FRU identification information. The tables take account of the fact that the reporting FRU could be the reporting FRU, the reported FRU or some other combination. In case of interconnect errors, the service controller can use hardware configuration information, for example as represented schematically in
In pass 5, a list of suspects is then filtered.
The "analyze error" task identified in passes 3 and 4 can have caused a list of one or more suspect FRUs to be generated. In pass 5, FRUs which are not physically present and FRUs which are already diagnosed as faulty are filtered out.
At the end of pass 5, the result is one of the 3 options referenced earlier
The auto diagnosis engine is also responsive to correctable errors.
In the present examples of systems implementing the present invention, datapaths are protected by error checking codes (ECCs) and there are also parity signals covering the datapaths.
If a bit flip occurs, an ECC error will happen and the data will be corrected. Since the datapath is also protected by parity, it is possible to diagnose where the bit flip occurred.
The parity errors can be diagnosed by the auto diagnosis engine. However, the correctable error handling differs from the uncorrectable error handling as described above, in that even if a single FRU is diagnosed, it is considered as suspect (as opposed to faulty) as discussed above.
As mentioned above, the FRUs employ a consistent error reporting policy. In general, error signals propagate from the FRUs via the level 1 and level 2 switch interconnects to the system controller boards 18.
Thus, the SBBCs 106 on the system controller boards 18 determine what action(s) to take on the errors based on data stored in an SBBC error control register. Typical actions include setting appropriate error status bits, asserting an error pause to stop any further address packets being transmitted and interrupting the system controller 18 on the system controller board. At this point the system controller takes over, reading the various error status registers to find out what happened, and after determining the cause of the error possibly deciding whether the error is recoverable or not. If the error is recoverable, the system controller can clear the error status in the appropriate registers in the circuit(s) that detected the error and de-assert "error pause" by clearing appropriate system error bits in the SBBC error status register. If the error is not recoverable, the system controller may decide to reset the system which it can do by writing to a reset control register in the system controller SBBC 106. The error signals, error pause signals and reset signals can be isolated per partition and domain so that errors in one partition or domain do not effect other partitions or domains.
As mentioned above, the auto diagnosis engine is able to use error information previously recorded in a FRU to assist in diagnosis of the fault. For example, consider a situation where two FRUs are labeled as suspect, and an engineer replaces one of them. If, subsequently, a further error occurs, the auto diagnosis engine could initially conclude that one of the same two FRUs was suspect. However, it could further determine from the fault information and serial number recorded in the remaining FRU that the FRU concerned had not been replaced, and could thereby conclude that the FRU that had not been replaced, rather than the one that had been replaced, was the FRU that was faulty. In this case, the CHS of the FRU that had not been replaced could then be flagged as faulty and its FRUID EEPROM could be updated with a faulty indicator rather than a suspect indicator.
Also, the auto diagnosis engine could also be responsive to a plurality of correctable errors being recorded for a FRU as being indicative that a more serious fault was potentially going to occur and could recommend that the FRU be replaced, for example by changing the CHS of the FRU concerned to faulty and updating its FRUID EEPROM with a faulty indicator rather than a suspect indicator.
Although the embodiments above have been described in considerable detail, numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.
For example although specific implementations have been described in a system employing a Solaris operating system and a PowerOn Self Test Utility, it will be appreciated that the invention is applicable to systems operating under other operating systems and system boot utilities.
Also, it will be appreciated that program products can be employed to provide at least part of the functionality of the present invention and that such program products could be provided on a suitable carrier medium. Such a carrier medium can include, by way of example only, a storage medium such as a magnetic, opto-magnetic or optical medium, etc., or a transmission medium such as a wireless or wired transmission medium or signal, etc.
|Cited Patent
|Filing date
|Publication date
|Sep 19, 1974
|Dec 23, 1975
|Diagnostic system for field replaceable units
|Sep 29, 1989
|Jun 16, 1992
|The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration
|Remote maintenance monitoring system
|Nov 6, 1990
|Aug 17, 1993
|Hitachi, Ltd.
|Monitoring and controlling system and method for data processing system
|May 15, 1995
|Sep 1, 1998
|The Boeing Company
|Method and apparatus for obtaining high integrity and availability in a multi-channel system
|Nov 10, 2000
|Sep 17, 2002
|Galactic Computing Corporation
|Scalable internet engine
|Oct 18, 2000
|Apr 29, 2003
|Compaq Information Technologies Group Llp
|Dense packaging and cooling system for use with a server or other processor-based device
|Dec 22, 2000
|Jun 24, 2003
|Emc Corporation
|Computer system
|Dec 28, 2000
|Apr 20, 2004
|International Business Machines Corporation
|Identifying field replaceable units responsible for faults detected with processor timeouts utilizing IPL boot progress indicator status
|Mar 1, 2001
|Sep 14, 2004
|International Business Machines Corporation
|Standardized format for reporting error events occurring within logically partitioned multiprocessing systems
|Feb 27, 2003
|May 3, 2005
|Hewlett-Packard Development Company, L.P.
|Diagnostic exerciser and methods therefor
|Feb 11, 2003
|May 31, 2005
|Hewlett-Packard Development Company, L.P.
|Apparatus and method for verification of system interconnect upon hot-plugging of electronic field replaceable units
|Aug 9, 2002
|Jun 20, 2006
|Sun Microsystems, Inc.
|Multiprocessor systems
|Dec 28, 2000
|Jul 18, 2002
|International Business Machines Corporation
|Identifying field replaceable units responsible for faults detected with processor timeouts utilizing IPL boot progress indicator status
|Mar 5, 2002
|Sep 5, 2002
|Bottom David A.
|Modular server architecture with ethernet routed across a backplane utilizing an integrated ethernet switch module
|Mar 1, 2001
|Sep 5, 2002
|International Business Machines Corporation
|Standardized format for reporting error events occurring within logically partitioned multiprocessing systems
|Jun 14, 2002
|Feb 13, 2003
|King James E.
|Secure network identity distribution
|Jun 14, 2002
|Feb 13, 2003
|Garnett Paul J.
|Interfacing computer modules
|Jun 14, 2002
|Feb 13, 2003
|Garnett Paul J.
|System management
|Jun 14, 2002
|Feb 13, 2003
|Garnett Paul J.
|Interfacing computer system modules
|Jun 14, 2002
|Feb 13, 2003
|King James E.
|Secure network indentity allocation
|Jun 14, 2002
|Feb 13, 2003
|King James E.
|Computer system management
|Jun 14, 2002
|Feb 13, 2003
|King James E.
|Secure system unit mobility
|Aug 9, 2002
|Mar 13, 2003
|King James E.
|Integrity monitoring
|Aug 9, 2002
|Mar 13, 2003
|King James E.
|Combined computer system
|Aug 9, 2002
|Apr 22, 2004
|King James E.
|Computer system monitoring
|Aug 29, 2003
|Mar 3, 2005
|Sun Microsystems, Inc.
|System health monitoring
|Apr 28, 1992
|Nov 4, 1992
|NCR International, Inc.
|Multi-stage interconnect network for processing system
|"Dynamic Logical Partitioning in IBM eserver pSeries", IBM White Paper, Oct. 2002, New York, USA, pp. 1-9.
|Anonymous, "Error-Reporting Registers", IBM Technical Disclosure Bulletin, IBM Corp., New York, NY, vol. 30, No. 6, Nov. 1, 1987, pp. 79-80, XP000714517.
|Anonymous, "Storing Failure Information into Non-Volatile Memory", IBM Technical Disclosure Bulletin, vol. 37, No. 11, Nov. 1, 1994, pp. 419-420, New York, NY, XP000487287.
|Bosen, et al, "Fault-Tolerant Design of the IBM pSeries 690 Sysem Using POWER4 Processor Technology", IBM Journal of Research and Development, vol. 46, No. 1, Jan. 2002, XP002302455.
|Gonscherowski, "Sun Fire(TM) 6800/4810/4800/3800 Auto Diagnosis and Recovery Features", Sun Blueprints, Santa Clara, CA, Apr. 2003, XP002302457.
|International search report application No. PCT/IB2004/001644 mailed Nov. 8, 2004.
|International search report application No. PCT/IB2004/001644, mailed Feb. 22, 2005.
|Citing Patent
|Filing date
|Publication date
|Apr 14, 2005
|Jan 13, 2009
|Hitachi, Ltd.
|Access control device and interface installed in same
|Aug 25, 2006
|Sep 29, 2009
|Carlo Leonardo Di Cristofano
|Computer server with built-in modular networking component
|Jun 29, 2006
|Jul 13, 2010
|Emc Corporation
|Managing faults
|Oct 5, 2007
|May 17, 2011
|Juniper Networks, Inc.
|Connection module for providing N+N and M+1 redundant power distribution
|Nov 20, 2008
|Aug 30, 2011
|International Business Machines Corporation
|Hardware recovery responsive to concurrent maintenance
|Dec 17, 2009
|Jan 31, 2012
|Hewlett-Packard Development Company, L.P.
|Field replaceable unit failure determination
|Feb 13, 2009
|May 8, 2012
|The Pnc Financial Services Group, Inc.
|Systems and methods for computer equipment management
|Apr 7, 2011
|Jul 3, 2012
|Juniper Networks, Inc.
|Connection module for providing N+N and M+1 redundant power distribution
|Feb 13, 2009
|May 7, 2013
|The Pnc Financial Services Group, Inc.
|Systems and methods for computer equipment management
|Jun 11, 2013
|Board Of Regents Of The Nevada System Of Higher Education
|Error detection method
|Mar 14, 2012
|Mar 1, 2016
|Qualcomm Incorporated
|System and method for thermally aware device booting
|Feb 11, 2014
|May 31, 2016
|Fujitsu Limited
|Method for notifying abnormality in electro device, and electro device
|Apr 14, 2005
|Aug 17, 2006
|Kunihito Matsuki
|Access control device and interface installed in same
|Aug 25, 2006
|Mar 15, 2007
|Di Cristofano Carlo L
|A computer server with built-in modular networking components
|Feb 13, 2009
|Aug 20, 2009
|Sawczak Stephen D
|Systems and methods for computer equipment management
|Aug 15, 2008
|Feb 18, 2010
|Honeywell International Inc.
|Distributed decision making architecture for embedded prognostics
|Nov 20, 2008
|May 20, 2010
|International Business Machines Corporation
|Hardware Recovery Responsive to Concurrent Maintenance
|Jun 23, 2011
|Barlow Jeffrey A
|Field replaceable unit failure determination
|Sep 22, 2011
|Board Of Regents Of The Nevada System Of Higher Education, On Behalf Of The University Of Nevada
|Error detection method
|Mar 14, 2012
|Aug 29, 2013
|Qualcomm Incorporated
|System and Method For Thermally Aware Device Booting
|Oct 7, 2014
|Apr 7, 2016
|International Business Machines Corporation
|Device driver error isolation on devices wired via fsi chained interface
|Jun 2, 2015
|Apr 7, 2016
|International Business Machines Corporation
|Device driver error isolation on devices wired via fsi chained interface
|U.S. Classification
|International Classification
|Cooperative Classification
|European Classification
|Aug 11, 2003
Owner name: SUN MICROSYSTEMS, INC., CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PANICKER, SUNDEEP R.;REEL/FRAME:014389/0478
Effective date: 20030807
|Dec 8, 2003
Owner name: SUN MICROSYSTEMS, INC., CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:VECOVEN, FEDERIC LOUIS GHISLAIN GABRIEL;SUN MICROSYSTEMSLUXEMBOURG SARL;REEL/FRAME:014762/0118
Effective date: 20030416
|May 25, 2011
|Fee payment
Year of fee payment: 4
|Jun 10, 2015
|Fee payment
Year of fee payment: 8
|Dec 16, 2015
Owner name: ORACLE AMERICA, INC., CALIFORNIA
Free format text: MERGER AND CHANGE OF NAME;ASSIGNORS:ORACLE USA, INC.;SUN MICROSYSTEMS, INC.;ORACLE AMERICA, INC.;REEL/FRAME:037302/0899
Effective date: 20100212


text:
|Reviews for Food for Thought
| LunarCatNinja chapter 12 . 1/22/2014
Wow. I'm still a little iffy on certain things in the story, but you write everything so well that I don't particularly care. I love this story!
| shadowaureous chapter 12 . 1/11/2014
Love your fics. Just wanted to drop by to wish you a happy birthday and a huge thank you for coming back. Can't wait to read more.
| reader in TX chapter 12 . 1/6/2014
Hey- no worries. We have missed your writing but update when you can- don't stress to much.
Early Happy Birthday! Happy New Year & hope you had a good yule season.
Stay warm- this cold front is evil. At least its a dry cold not like we had in December with the ice storm though.
| SanbantaiTaichouKurosaki chapter 12 . 12/22/2013
I really love your story. The portrayal of Sherlock and Mycroft's Bond during the threeway was really well written. I absolutely love this story and could read it multiple times. I would love it if you could update soon! I can't wait to see what happens next.
Would Harry get pregnant? They didn't seem to use any contraceptives during the Lemon scene, or is there a special spell or potion needed for pregnancy to happen?
Love your story, update soon!
| Ice Night chapter 12 . 12/15/2013
Great story, please update soon! :)
| Mists of Ribbon chapter 12 . 12/15/2013
When life gives you lemons, read them?
| c3lph t1tl3d chapter 12 . 12/1/2013
Well this has been one of the best reads ever and I really hope that I am going to be getting to read more of it soon :)
| RogueNya chapter 12 . 11/30/2013
Wow this was a great story, hope to read more in the future.
| choirbandgeek chapter 11 . 10/9/2013
This chapter gives new meaning to revenge is a dish best served cold! Loved the orchestration to the whole plot to remove the Dursley's. I also enjoyed the little Dr Quinn Medicine Woman "shout out" (Micheala not Micheal A).
Keep up the good work!
| angelalexiel16 chapter 1 . 10/4/2013
please update I really want this story to continue it is a really awesome story.
| fannut chapter 12 . 9/24/2013
Wow. Amazing! I love this story. I love their interaction, both in bed and out. I really hope you write more and post it soon. I'm going to go see if you have any other wonderful storys so don't be surprised if I comment on one.
| Guest chapter 12 . 9/16/2013
so...update? you can't really leave it there... can you!?
| thedancingelk chapter 12 . 9/1/2013
Pretty please update this story soon!
| Vicki219 chapter 12 . 9/1/2013
Excellent story, a little weird but all the more interesting for it.
| DarkestValentine chapter 12 . 8/17/2013
... That was hot. Brilliant story by the way.


text:
- Working-Class Perspectives offers weekly commentaries on current issues related to working-class people and communities. Contributors discuss a variety of issues, from what class means to how it intersects with race and gender to how class is shaping American politics. We welcome relevant comments of 500 words or less.
For questions or comments about this blog, e-mail Sherry Linkon. For assistance with news stories about working-class politics and culture, call or e-mail John Russo, 330-207-8085.
The State of the Working ClassListen to Working-Class Perspective editor Sherry Linkon's recent interview about Working-Class Studies on KERA's Think with Krys Boyd.
Tag Archives: charter schools
The Education Campaign: Addressing Inequality through Teaching and Learning?
Other than Hillary Clinton's adoption of Bernie Sanders's proposal to make college tuition free for most Americans, we haven't heard much about education in this year's election. The focus has been on economic inequality, immigration, trade, and national security – … Continue reading
Vandals in a Steel City School District
If you weren't sure, let me remind you, Pittsburgh is a 21st century city that still has a rusty, bricked out, working-class soul. I know this because I live here, but also because I finally saw the agit-prop-weepy that is … Continue reading