text:
See more than 50 low-fat menus that are a joy to eat.
Recipe of the Day
This is a popular meat dish from Southern Norway. Lamb and cabbage are layered and stewed with peppercorns. Serve with boiled potatoes that have been sprinkled with parsley.
Similar Recipes | More Daily Recipes
See a great trick for making truly delicious low-fat turkey burgers.
Low-Fat Recipe Collections6
Low-Fat Desserts
Low-Fat Main Dishes
Low-Fat Appetizers
Low-Fat Side Dishes
Low-Fat Soups and Stews
Low-Fat Salads
Staff Picks
Roasted Vegetable Medley
Seared Scallops with Tropical Salsa
Gary's Turkey Burritos
The Ultimate Chili
Easy Grilled Chicken
Jay's Signature Pizza Crust
Best Spanish Rice
Almost No Fat Banana Bread
Low-Fat Recipes
Quinoa and Black Beans
Fluffy French Toast
Homemade Black Bean Veggie Burgers
Amazing Pork Tenderloin in the Slow Cooker
Szechwan Shrimp
Sarah's Applesauce
Baked Honey Mustard Chicken
Refried Beans Without the Refry
Pretzel Turtles
Chocolate Crinkles II
Slow Cooker Turkey Breast
D's Famous Salsa
Adrienne's Cucumber Salad
Slow Cooker Cilantro Lime Chicken
Blackened Chicken
Black Beans and Rice
Mexican Rice III
Red Lentil Curry
Broccoli Beef I
Split Pea Soup
Choose from dozens of top-rated roast turkey recipes, from stuffed to deep-fried.
We're counting down from now until Thanksgiving with a great pie every day. Join the fun.
Delicious recipes, party ideas, and cooking tips! Get a year of Allrecipes magazine for $5!
See all you can do with this centuries-old super grain.
See how to can your own fruits and vegetables so you can enjoy them year-round.
Watch how to prepare and cook this extremely versatile vegetable.


text:
Lara Logan
Judith Matloff on Lara Logan and Safety On the Beat: a CJR Podcast
By The Editors Feb 17, 2011 at 04:15 PM
This week, we heard the horrible news of the assault of Lara Logan, CBS News's chief foreign correspondent, in Cairo's... More
Lara Logan, Foreign Correspondents, and Sexual Abuse
By Liz Cox Barrett Feb 17, 2011 at 11:42 AM
On Tuesday came the chilling news from CBS News that chief foreign correspondent Lara Logan, while reporting a 60 Minutes... More
Safety Tips for Female Correspondents
How to minimize the risk of sexual assault while on the job
By Judith Matloff Feb 22, 2011 at 04:21 PM
The attack on Lara Logan of CBS was a worst-case scenario for many female reporters. Yet gropes and unwanted advances... More
Why won't journalists ask Bill Cosby the tough questions? - Sexual assault charges are hardly ever mentioned to the TV star
Knoxville's alt-weekly wasn't losing money. It got shut down anyway - Scripps pulls the plug on Metro Pulse in favor of an entertainment supplement for the local daily
Chuck Todd's Obama book says more about the author than it does about the president - 'The Stranger' underscores a broader problem with the way we cover politics
Beware labeling Pope Francis a liberal - Political boxes like liberal/conservative and evolutionist/creationist miss his real significance
The Virginian-Pilot produces a breakthrough investigation amid layoffs - Reporter John Holland discusses how the story came together, newsroom cutbacks, and colleagues who cover for each other
Why won't journalists ask Bill Cosby the tough questions? (17)
Serial creators don't know what will happen to Adnan Syed (13)
Knoxville's alt-weekly wasn't losing money. It got shut down anyway (10)
My take: The end of an era in Tampa (10)
Email blasts from CJR writers and editors
Which media outlets use 'illegal immigrant'? (Fusion)
"[I]n spite of all the good reasons not to use the phrase, it is still very easy to find in the US press, even in headlines"
Jack Shafer on losing his job (Capital New York)
"Right now, my immediate plan is to go to work as a lay therapist at The Intercept to bring the healing there so John Cook and Matt Taibbi can return. I have great interpersonal skills."
On outlets that continue to call unknown drone victims 'militants' (The Intercept)
"Like the US drone program itself, this deceitful media practice continues unabated"
UNITY's convention is no more (All Digitocracy)
"The organization's board of directors decided that UNITY will no longer host the quadrennial conferences for which it had become known"
Greg Marx discusses democracy and news with Tom Rosenstiel of the American Press Institute
CJR's Guide to Online News Startups
ACEsTooHigh.com – Reporting on the science, education, and policy surrounding childhood trauma
Who Owns What
The Business of Digital Journalism
A report from the Columbia University Graduate School of Journalism
Questions and exercises for journalism students.
Should all journalists be on Twitter? – Reasons to take up or forgo the 140-character platform
Is it ever okay to name victims? – Editors question the policy of omitting information in sexual assault and domestic violence cases

subset: commoncrawl


text:
Find records of Rosa Molina Salaz
Find birth, death, marriage, military, and other records of Rosa A Molina Salaz at Ancestry.com.Search for records
Biography & Family History Add details
Rosa A Molina Salaz (1937 - 2003) was born on November 3, 1937. She was born into the Molina Salaz family.
There are no known marriages or children on file for Rosa. Rosa died on August 4, 2003 at 65 years of age.
Rosa A Molina Salaz's last known residence is at Stamford, Fairfield County, Connecticut.
The data on this page comes from the official record for Rosa A Molina Salaz in the SSDI (United States Social Security Master Death Index). What is the SSDI?
This genealogy of Rosa A Molina Salaz's past was created by AncientFaces users like you Contribute what you know to connect with the people who know her. Some things we would like to know about Rosa's genealogy:
- Where was Rosa born?
- Where did Rosa die?
- Who was Rosas family?
- What did Rosa look like? Add some photos of their life.
Family Tree & Genealogy Add / Edit People
You can add or remove people from this genealogy by clicking here.
History of Rosa A Molina Salaz Add details
- Given name
- Rosa
- Surnames
- Molina family history, Salaz family history
- Last Known Residence
- Stamford, Fairfield County, Connecticut
- Born
- Death
- Gender
- Female
Do you know more about Rosa A Molina Salaz?
You can add details to this page and share Rosa's story.
Obituary Add details
There is no information regarding the obituary records of Rosa A Molina Salaz. You can add obituary information about Rosa A Molina Salaz by clicking "Add Details".
Discuss Rosa
Share your memories below with the rest of Rosa's family. We'll notify anyone else who is searching for Rosa of your message.
Find records of Rosa Molina Salaz
Find birth, death, marriage, military, and other records of Rosa A Molina Salaz at Ancestry.com.Search for records
Possible Relatives
- Charles Cloutier
- ·
- Francois Cloutier
- ·
- Genevieve Moreau Cloutier
- ·
- Genevieve Cloutier Verreau
- ·
- Louise Cloutier Cauchon
- ·
- Jean Busque
- ·
- Paul Verreau
- ·
- Antoine Dorval
- ·
- Antoine Ignace Nadeau
- ·
- Genevieve Crepeau Dorval
- ·
- Romaine C Raine
- ·
- Joseph Berube
- ·
- Pierre Berube
- ·
- Augustin Goupille
- ·
- Joseph-Marie Cloutier
- ·
- Agnes N
- ·
- Genevieve-Eleonore Cloutier Belisle
- ·
- Marie-Joseph Cloutier Boucher
- ·
- Marie-Charlotte Lessard Cloutier
- ·
- Jacques Cloutier
- ·


text:
Loading video...
DeadlineNow: Hollywood's Heavy Hitters Celebrate Jane Fonda At AFI Gala (Video)
Flash Mob
2014 Governors Awards Party Pics
June 6
The 6th annual Governors Awards black-tie dinner was a glamorous event on…
Deadline's The Contenders 2014
June 6
Deadline Hollywood's 4th annual The Contenders event drew a sold-out crowd…
Party Pics: 'Interstellar' European Premiere
June 6
She is a traitor whose best effort was Barbarella. She portrayed herself. A dumb bimbo.
It must burn up cons that Ms. Fonda gets awards while Condi Rice is shut out of college commencement speaker gigs. Heh.
Comments are closed.


text:
Cricket wrap: Third Test, Day Three
Paul Kennedy wraps up all the action on day three of the third Test between Australia and South Africa.
Source: ABC News | Duration: 1min 23sec
Topics: cricket, perth-6000, wa, australia
By Guy Stayner
Victorians overwhelmingly support voluntary euthanasia for terminally ill patients, according to Vote Compass results.
By Anne Barker
Vientiane has been dubbed the world's most relaxed capital, but also has some of the world's most dangerous roads.
Ien Ang has called three countries home and reflects on belonging, cultural identity and finding love online.
Talk About It's Del Irani takes a look at seven pressures of modern parenthood, from breast feeding to work-life balance.


text:
Sign in
Sign Up
lowest rate guarantee
5F., No.95-1, Songjiang Rd.
No.11, Sec. 1, Heping W. Rd.
1F., No.209, Sec. 1, Civic Blvd.(Taipei Bus Station, next to the driveway by MRT Y3 exit)
1, Chung Shan N.Rd., Sec.4,
5F., No.72, Sec. 1, Zhongxiao W. Rd.
Great reasons to book with Agoda
Over 500,000 hotels worldwide – more added every day
Trusted by over 17,000,000 satisfied customers
Best Price Guarantee
8 million+ hotel reviews by Agoda customers
With availability in over 55,000 cities worldwide
agoda special offers
including last minute offers(even same day!)free nightsfree upgrades!
Customer Support
Fair deal, free Internet, 3min...
0.54 Km.
0.62 Km.
Room is Clean, Great Location (5...
0.63 Km.
Helpful staff. Easy access....
0.66 Km.
The boss, Italian breakfast and...
0.67 Km.
Good location Good service Nice...
0.69 Km.
Cleanliness, convenient location...
0.73 Km.
Location is good and hotel is...
0.74 Km.
Good service, good location and...
0.75 Km.
best location; friendly staff;...
0.76 Km.
Grand &Beautiful at Resort...
nothing, umm.. the only advantage...
0.79 Km.
the hotel is nice, as well the...
0.8 Km.
Clean and tidy room with...
0.81 Km.
Great location, spacious room,...
0.82 Km.
Tub is fine, easy to access to mrt...
0.84 Km.
location Convenience,Clean,friendly...
0.9 Km.
Simple breakfast, friendly staff...
0.96 Km.
Good value, spacious bed and...
staff performance, location nice...
0.97 Km.
4.5 / 5
over 80,000 reviews
of users recommend us
Go To Top
All material herein © 2005 – 2014 Agoda Company Pte. Ltd., All Rights Reserved.
AGODA ® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of


text:
Salem selectmen sign off on new truck for Public Works
On Monday, selectmen approved spending just over $17,000 on a new pickup truck for the department. If there is not money left over in the 2014 budget for the purchase, the board allowed the use of the water and sewer department's unassigned fund balance for the purchase.
Selectmen unanimously approved using the unassigned fund balance for the purchase, if needed, but did ask if it would have been possible to determine that a new truck was needed during the last budget process.
Russell said it wasn't known that the truck was in that bad of shape until it went in for the state inspection. If the department had thought the truck wouldn't be able to pass inspection, Russell said there would have been a request for a new truck in the department's budget.
Keller asked if it would be possible for the town to inspect vehicles prior to the beginning of the town budget process.
"The inspection is done by the state in March, and to have us go through and inspect these vehicles a second time during the year is time consuming from a labor standpoint," said Hickey.
Press Releases
Public Information Meeting on New Shellfish Area Closures – Dec. 2
Last Bowhunter-only Hunter Education Course Offered
Hike Safe Cards to Be Available for 2015
Sustainability Commission Recommendations for NH Fish & Game
The Granite State Rail Trail vision takes shape
Most Read
After the election is over, the signs remain
School's out for voters
Most Comments
Hillary stumps for Democrats in Nashua


text:
- Sauerkraut Is Most Googled Thanksgiving Recipe In Maryland
- Smart Woman: Black Friday and Cyber Monday Tips
- Toyota, Tire Maker Toyo Issue Safety Recalls
- Is Apple about to kill off its iPhone 5C?
- Meet the Keenan family, who are happily stuck in the 1950s
- US Air Force's A-10 Warthogs and Reaper drones to blast ISIS from the skies
- 13-month-old baby girl dies after suffering suspected cardiac arrest at nursery in Edinburgh
- Rochdale couple swindled up to £1million from the NHS to fund their luxury lifestyle
- Freeze! Police arrest mafia godfather's son after discovering secret tunnel accessed through a fake fridge
- Take That snub Spotify by streaming new album III exclusively on Google
Royal Family Considers Legal Action Over Topless Pictures of Kate
Related Articles
- Magazine Fined for Topless Kate Middleton Photos
- Irish lace firm responsible for Kate Middleton's wedding dress create gown for Prince William's baby
- Now Hospital Which Treated Kate Denies Hoax Call Radio Station Contacted Managers or Press Oficers
- Kate Middleton's Second Cousin Shows Off Family Goods
- Kate Middleton Glamorous in Grey During Glitzy Event
More from World News
- He'll be Vlad once it's over: Tense meetings, awkward handshakes and shunted to the edge of family photos - it's been a G20 to forget for Putin
- Ukraine rejects 'fake' photos of MH17 downing
- 'Syrian hero boy' is a FAKE: Footage of youngster dodging sniper fire to rescue girl was actually shot on Gladiator film set by Norwegian director using professional actors
- Angela Merkel poses for photos at Brisbane's Brewski Bar
- Photos kept in family for 47 years show Che Guevara after he was killed by the Bolivian army in 1967
A French magazine has stoked controversy by publishing pictures of Catherine, the Duchess of Cambridge, topless while on vacation with Prince William, prompting the threat of legal action from palace officials over what they blasted as a "grotesque" invasion of privacy.
The pictures were taken while the couple was staying at a private chateau in Provence, in southern France, according to Closer magazine.
A St. James's Palace spokesman said the pair were "hugely saddened to learn that a French publication and a photographer have invaded their privacy in such a grotesque and totally unjustifiable manner.
"The incident is reminiscent of the worst excesses of the press and paparazzi during the life of Diana, Princess of Wales, and all the more upsetting to The Duke and Duchess for being so.
"Their Royal Highnesses had every expectation of privacy in the remote house. It is unthinkable that anyone should take such photographs, let alone publish them," the palace spokesman said.
Officials are consulting with lawyers on what options may be available to the couple, he said.
The pair remain focused on their tour on behalf of Queen Elizabeth II, he added.
William, who is second in line to the throne, and his wife are in Malaysia as they carry out a nine-day Southeast Asian tour of Commonwealth nations, which started in Singapore.
A royal source told CNN the Duchess of Cambridge is "upset" with the French magazine.
The photographs, which are grainy and appear to have been taken with a long lens, cannot be accessed on the website of Closer in France, which shows only the magazine's front cover with heavily blurred-out images.
St. James's Palace officials told CNN they "believe a red line has been crossed" and are consulting with French lawyers.
The photographs' publication comes only three weeks after the British royal family was caught up in a media furor over images of William's younger brother, Prince Harry, partying naked in his Las Vegas hotel room.
Those photographs were widely circulated online but were published in only one UK tabloid, The Sun. The press Complaints Commission, the UK press watchdog, received about 3,800 complaints from the public but said it was inappropriate for it to take any action in the absence of a formal complaint from the palace.
Laurence Pieau, editor-in-chief of Closer in France, posted several comments on Twitter late Thursday promoting the release of the topless pictures of Catherine.
One said: "The world exclusive of #closer tomorrow is Kate Middleton as you have never seen her ... and as you will never see her again."
Another read: "For kate middleton to have tan lines she would have to be wearing a swimsuit!"
Interviewed by CNN affiliate BFM-TV on Friday, Pieau defended the decision to publish the photographs, saying, "We were just doing our job."
The magazine was tipped off about the dates of the royal couple's visit to Provence and sent a photographer, she said. The pair were visible on a terrace from the road.
Pieau said that there had been no debate at the magazine over whether to publish the photos and that she believes they are not degrading, as "they are just like any other couple in love."
She described the reaction to the pictures' publication as "disproportionate" and slammed the British press as "complete hypocrites," since photos of Harry naked were published by The Sun.
Pieau said that there are additional photos that are even more revealing but that Closer is not publishing them for now. The magazine will wait to see if any complaint is made against it, she said.
"I think that what these photos really highlight is the issue of the royal couple's security," she added.
A string of negative comments from members of the public has already been posted on the French Closer magazine's Facebook page.
One, written in French, said: "Your behavior is unacceptable, you're as bad as the English tabloids. You have gone too far with the photos of Kate topless and it will cost you a lot I hope. Shameful."
The furor over the topless pictures of Catherine threaten to overshadow what has until now been a well-received tour in Asia, undertaken as part of diamond jubilee celebrations for the queen, William's grandmother.
The Duchess of Cambridge gave her first public speech in Malaysia's capital, Kuala Lumpur, on Thursday, at a hospice for terminally ill children and adults. She is the patron of a charity supporting children's hospices in East Anglia, England.
The royal couple will be in Malaysia until Sunday before heading to the Solomon Islands and Tuvalu.
The Duchess of Cambridge was very conservatively dressed as she visited a mosque in Kuala Lumpur on Friday.
There was also a security scare when a man lunged at Catherine and tried to give her a flower as she and William left a festival of Malaysian dancing in a park in the capital.
A man was arrested over the incident, a palace source said.
"The Duchess was getting into the car and the Duke was walking round to the other side of the car when an enthusiastic member of the public tried to give her a flower," he told CNN.
"He was very close, right up against the car. He was arrested by local police officers. Royal protection officers were not involved, though they had noticed him in the crowd earlier on. The Duke and Duchess were not in any danger at all at any point."
William and Harry have been very sensitive to issues around invasion of privacy since the death of their mother, Diana, in a car crash as she fled a crowd of photographers in Paris 15 years ago.
It is not yet clear what form any legal action by the Duke and Duchess of Cambridge against Closer magazine, run by an Italian business called Mondadori, might take.
French magazine VSD was fined 2,000 euros ($2,580) last week after it published a photo of Valerie Trierweiler, partner of French President Francois Hollande, in a bikini on its front cover, BFM-TV reports.
Trierweiler is also taking legal action against other French magazines that published images of her and Hollande in swimwear while on vacation, BFM-TV says.
The French Closer magazine is run by a different company from the publication of the same name in the United Kingdom.
Closer UK, published by Bauer Consumer Media, issued a statement distancing itself from the French magazine.
"Closer magazine UK would like to make it clear that the two publications make entirely independent editorial decisions. In this respect the comments made by the editor of the French edition which have reported in the media today do not reflect the opinions of Closer magazine UK.
"Closer magazine UK was not offered any pictures of this nature and certainly has no intention of publishing the photographs of the Duchess of Cambridge which have been published in France this morning."
The British media is under close scrutiny after revelations of phone hacking and other abuses. The conclusions of an independent judge-led inquiry, which may recommend greater restrictions on media freedoms, are expected by the end of the year.


text:
Plot:Team Plasma has the Unova region in a tight grip. Every single day, the mysterious N makes speeches to people, begging them to release their Pokemon. And every day, hundreds of foolish trainers abandon their close bonds with their Pokemon and release them. Unbeknowest to the trainers, Team Plasma has motivations behind this practice. If there are no trainers, then there is nobody to oppose them and stop them from targeting the two legendary dragons, Reshiram and Zekrom.
A hero can be found in the most unlikely of people. Forunately for the Unova region, they have a group of heroes to oppose Team Plasma. A group of kids setting out on their own journeys will discover much about themselves and about the bond with Pokemon as they fight to discover their path in life and take down Team Plasma.
This rpg is set mainly in the Unova region. When it starts, all of the trainers will be waking up on the morning that their journey begins. They will journey to Professor Juniper's lab in Nuvema Town to select one of the three Unova starters.
Grass Snake Pokemon
Moves:Tackle, Leer
Fire Pig Pokemon
Moves:Tackle, Tail Whip
Sea Otter Pokemon
Moves:Tackle, Tail Whip
Name:Amber Whetchton - Cartmanland
Personality:Amber is a very optimistic person, trying to see the good in everyone. Amber doesn't believe that anybody can be truly evil and will often try to help out people who she feels are confused or conflicted.
Appearance:Amber has red hair neatly tied into braids. She has freckles around her cheeks and nose. Amber typically wears a uniform from the Pokemon Trainer's School ; a long sleeve blouse with a vest on top and a plaid skirt with socks that go up to her knees and Mary Janes on her feet.
History:Amber was heavily sheltered by her parents as a child. Her family moved from Goldenrod City in Johto to Striation City in Unova. Her parents are overly neurotic and paranoid of everything. Forunately, Amber managed to persuade them to let her go on a journey.
Inventory:Pokedex, Pokeballs x5, Town Map, PokeGear
Hometown:Goldenrod City (Originally), Striaton City
Personality:Azura is a very friendly and curious baby Pokemon who enjoys bouncing around and cuddling with her trainer.
History:Azura was given to Amber by her grandparents, the Day Care Couple in Goldenrod City as an egg as a parting gift.
Moves:Splash, Charm, Scald
Wyrm (no last name known) - Charizardman
Appearance:Wyrm wears a hat, which covers his head, and a jacket, even if it is warm outside.
Personality:He loves being kind and caring, but can be a bit rambunctious at times.
History: He was born and raised in Opelucid City, and he moved to Striaton City when he was 13. He already had an odd colored Charmander from his parents for his tenth birthday. He got an Oshawott from Professor Juniper, and he has lived there for 5 years, until his friend Amber decides to go on a journey, and he figures he'd tag along.
Inventory: Pokedex, 5 PokeBalls, a PokeGear and a Town Map.
Hometown: Opelucid City, moved to Striaton City at 13
Charmander- Level 5
Oshawott- Level 5
Water Gun
Name: Lelouch
Personality: Lelouch is friendly towards other people and pokemon. Dilligent to a fault, he puts great effort into all that he does. Conversely, when he is not concentrating on something Lelouch is quite laid back.
Appearance: Lelouch has a long white beard and mustache. Oddly only looks about 1/2 of his actual age. He wears a gray robe, a pointy hat, a short red cape and uses a wooden cane to walk. (even though he doesn't need it)
History: When Lelouch was young, he hiked up Mt. Silver for exercise but could not find a way back down. For decades he survived in the wilderness, making friends with wild pokemon and also fighting off hostile pokemon. One day, Lelouch happened to bump into a trainer who gave him a map that enabled him to finally come down from the mountain. When he heard that people could actually TRAIN pokemon, Lelouch rushed over to Unova to become a trainer himself.
Inventory: C-Gear, Pokeballs x5
Hometown: Pewter city
Pokemon: Oshawott (Jak), Moves - Tackle, Tail whip, Water gun
Corey Weir
Appearance:He has short blond hair, and wears a standard outfit.
Personality:Corey is mean, he is mean to his Pokemon and others, especially Wyrm
Trainer Class:Pokemon Gym Challenger (Trainer basically)
Hometown:Striaton City
Moves:Pound, Growl, Water Gun
This thread is for OOC questions or comments so we won't clutter up the rpg. Cartmanland


text:
When a chick hasn,t had a bowel movement in awhile and a giant turd blocks your entry into her stink hole.
I wasn,t able to get no backdoor action from my girl because she had a serious case of cock block going on.
de J.J. Done 10 Decembrie 2003
A block made out of cocks.
Try to cut the infamous cock block.
de anus fie pace whole 27 Martie 2003
Tony Rodriguez
that Biotch is a cock block
de Butler 08 Ianuarie 2004
noun: a diaphragm
(adj; cockblocky, v; to cockblock see cockblocker, cockblockation)
(adj; cockblocky, v; to cockblock see cockblocker, cockblockation)
Betty Mae didn't want to get pregnant so she used a cockblock.
de Nigla Dave 01 Ianuarie 2005
its anything u want it to be really
an insult, exclamation of surprise..whatever
an insult, exclamation of surprise..whatever
u stuipd fuckin cockblock!
de maumau and yearno 10 Noiembrie 2003
(v.) The act committed by a woman during sex when the penis is too long for her to handle. Usually executed by the placing of her hands on his pelvis and either pushing away or creating a stopping point to prevent deeper penetration.
"I was trying to get deep inside of her while having sex, but she thru up the cock block dude." "She couldn't handle all this dick!"
de mabdallah 29 Iulie 2009
- ← anterior
- 1
- …
- 25
- 26
- următor →


text:
Search for jobs in Hampshire
Buying, Selling or Renting?
Find new & used cars for sale
Search our local business directory
View latest announcements online
Find 1000s of local products and services
Looking for a date in Hampshire?
Create Private and Trade ads
Find and book your next holiday with us
The county's police force has dedicated Country Watch officers working on rural issues
Jury told the complainant was calling for an ambulance when the defendant offered her a lift to hospital
Hampshire County Council said 257,000 people drink more than the safe recommended levels of alcohol each week
GnERGY visited by the parliamentary under secretary of state in the Department of Energy and Climate Change
Interim headteacher says 'green shoots' of improvements are being seen at special measures school
For the latest local news straight to your Twitter, Facebook feed or e-mail inbox.
For full newsroom contact details, click here. For advertising contacts, click here.
The Aldershot News & Mail Series and Star Courier are part of Trinity Mirror Southern, offering you unique access to our audience across the region online and in print.
We reach more than 64,000* unique users a month with dedicated coverage of Aldershot, Farnborough, Fleet, Yateley and the surrounding areas.
80% of our visitors**, who view 279,221* page impressions every month, are in the desirable ABC1 socio-economic groups.
To book an advert with the our advertising team call 01483 508700 or visit the Trinity Mirror Southern website for more information.
We are part of Trinity Mirror Southern, offering you unique access to our audience across the region online and in print.
Source: *Omniture (UK 6 month average Jan-Jun 2013). **JICREG 01/05/13
Editorial Director, Surrey Advertiser Group
Marnie has been Editorial Director of the Surrey and Hampshire titles for eight years. The titles were originally part of the Guardian Media Group's regional division before being sold to Trinity Mirror in 2010. Before that she worked at the BBC as a producer for 5 Live having moved to national radio from BBC Sussex and Surrey.


text:
A team led by Sandia National Laboratories scientists has created a tunable plasmonic crystal that could increase the bandwidth of high-speed communication networks and enhance high-speed electronics. The researchers accomplished the effect by making the plasma-containing crystal agile in transmitting terahertz light at varying frequencies.
DISA closed data center operations in Dayton, Ohio, and Chambersburg, Pennsylvania, and transferred the functions of these centers to other Defense Enterprise Computing centers on October 1.
Defense sector-specific solutions are driving growth in the cybersecurity market and will continue to do so through 2016. According to a Research and Markets report, the escalating amount of data stored in defense information systems and the increased number of cyberthreats are a major driver of this trend.
One of the U.S. Army's top priorities is ensuring the right information is available in any environment down to the lowest tactical level on the battlefield. Oshkosh Defense, recently demonstrated its command, control, communications and computers (C4) and systems integration capabilities to connect soldiers to the network on its MRAP All-Terrain Vehicle (M-ATV).
The U.S. Army National Guard has integrated the Company Intelligence Support Team (COIST) workstation with its counter-improvised explosive device (C-IED) training to improve operations in the field.
The Missile Defense Agency (MDA), U.S. Pacific Command, and personnel aboard the USS Lake Erie successfully conducted an operational flight test of the latest version of the Aegis Ballistic Missile Defense (BMD) System.
Massachusetts Institute of Technology (MIT) researchers have created prototypes of cube-shaped robots the size of a child's building block, which can climb over and around one another, leap through the air, roll across the ground and move while suspended upside down from metallic surfaces.
Industry and government personnel believe that event cancellations and travel restrictions are having a negative impact on innovation and collaboration.
The Defense Advanced Research Projects Agency (DARPA) is seeking ideas and technical proposals for how to best develop a fully reusable unmanned aircraft that would provide access to space faster, easier and at a lower cost than current satellite launch vehicles. According to Jess Sponable, manager of the Experimental Spaceplane (XS-1) program, the agency aims to build on proven technologies to create a reliable, cost-effective space delivery system that can be used to launch payloads into space, return to Earth and repeat the process the next day. Technical goals include the ability to fly 10 times in 10 days achieving speeds of more than Mach 10.
Current concepts call for a reusable first stage that would fly to hypersonic speeds at a suborbital altitude then one or more expendable upper stages would separate and deploy a satellite into low-Earth orbit. The aim is to achieve this at a cost of less than $5 million per flight for 3,000- to 5,000-pound payloads. "How it's configured, how it gets up and how it gets back are pretty much all on the table. We're looking for the most creative yet practical solutions possible," Sponable states.
DARPA has scheduled an XS-1 Proposers Day for October 7 and plans to hold one-on-one discussions with potential proposers on October 8. Registration for the event must be received by noon on October 1. Additional information is available via email and on the Federal Business Opportunities website.
Eight emerging cybersecurity technologies ready for transition into commercial products will be unveiled at the Mayflower Renaissance Hotel on October 9. The U.S. Department of Homeland Security (DHS) Science and Technology Directorate is hosting the event, which will feature intrusion detection, removable media protection, software assurance and malware forensics capabilities.
The Department of Energy's national laboratories developed the technologies that the DHS' Transition-to-Practice program will showcase during the Technology Demonstration for Investors, Integrators and IT Companies East event.
Cybersecurity professionals and technology investors from private industry will learn about these new technologies through presentations, demonstrations and discussions with the research teams that produced them. Attendees also will have an opportunity to schedule a private one-on-one discussions with the researchers to discuss opportunities for commercializing the technologies and areas of interest to drive further cybersecurity research.
Attendance is free, but registration must be received by October 7.


text:
remove these banner ads by becoming a JUB Supporter.
[COLOR=#000000][FONT=Times][COLOR=#464646][SIZE=5]...sunbuns99 November 27th, 2014 09:57 PM
[CENTER]...sunbuns99 November 22nd, 2014 09:24 PM
[COLOR=#000000][FONT=Times]...sunbuns99 November 22nd, 2014 09:23 PM
[COLOR=#000000][FONT=Times][B][S...sunbuns99 November 22nd, 2014 07:39 PM
As I sat across the table from him, I wondered what it was like to hear the secret confessions of those who felt guilty or sinful. I imagined that the confessions of the young were different than...sixthson November 22nd, 2014 01:54 PM
So which is the genuine one and which the photo-shopped? ... ...
You're 19 - if you're a little bit hairy now, that shit's gonna take over until by age 28 you'll be combing your earlobes anyway.
You'll 'fill
Thanks! Glad to be back! By the way, you didn't need to change the head on the first pic. Every source I've found that picture claims it is really RichardBearchaser1973 November 28th, 2014, 05:04 AM
WOW!!! What a monster cock!!! Love it. Thank you englishmale1! Got any more by any chance?orca1993 November 28th, 2014, 05:00 AM
November 28th, 2014, 04:44 AM
turn you upside right...
Remove these banner ads by becoming a JUB Supporter.
Visit our sister sites: Broke Straight Boys | CollegeDudes.com | CollegeBoyPhysicals.com | RocketTube
All models appearing on JustUsBoys.com were over 18 at the time of photography. The records for sexually explicit images required by U.S. 2257 are kept by the
individual producers of the images. The location of the records is available by clicking the Custodian of Records link at the bottom of each gallery page.
© 2012 JustUsBoys.com. The JustUsBoys.com name and logo are registered trademarks. Labeled with ICRA and RTA. Member of ASACP and The Free Speech Coalition.


text:
Welcome to Coin-O-Matic, the nation's oldest and most trusted vendor for commercial laundry equipment. Since 1921, we've provided
commercial washers and dryers for businesses and locations around the Chicagoland Midwestern Region with excellence in customer service and
Are you looking for coin laundry machines for sale?
We have a great selection of new and used machinesat competitive prices. Many institutions
benefit from the presence of coin operated laundry machines: locally-owned laundromats, apartment complexes, and student housing units. Providing a
commercial grade washer and dryer for residents and students is a great way to make life cleaner and more convenient for everyone. Contact us to purchase
any of our commercial laundry equipment today!
Coin-O-Matic has a proven reputation for quality. We pride ourselves on offering complete solutions for all our customers' needs. With multiple payment
options, you can choose a sound investment or a hassle-free lease. When you buy from Coin-O-Matic, you can rest assured that well help you with whatever
questions you have about your needs. If there are ever any problems with your commercial washer or dryer, our repair services can diagnose and fix your
machines cost effectively and quickly helping you get back in business. Call us today to see what we can do for you!
Dear Coin-O-Matic,
I really had no direction when considering starting a new coin operated laundry business, but knew I wanted to fill a commercial space I owned. I contacted various experts in the laundry industry and was impressed with the willingness and knowledge of Coin-O-Matic to work in partnership to satisfy my needs and wants. From the design to construction and finally upon opening it couldnt have run more smoothly. My recommendation would be Coin-O-Matic to any one! - Zak Siddiqui, Sycamore, Illinois, 8/15/2011


text:
My Sweet Valentine Surprises
My husband surprised me with notes all over the house and even in my truck this morning. There were Valentines on the bathroom TV, the mirror, in my purse, inside the cover of my Galaxy Tab and on the chocolate bar he delivered to the studios this morning. So very sweet!


text:
New to Typophile? Accounts are free, and easy to set up.
I need this font or something as similar as possible as a starting point to recreate this logo in AI. Looks like a classic font but I can't seem to find it using any of the online font identity tools. Thanks in advance!
INVEST PARTNERS: Trajan by Carol Twombly, Bold or artificially emboldened.


text:
Bilingual HR Generalist
Builds and maintains strong partnerships with managers throughout the organization in order to provide consultation and guidance on HR related matters.
Assists department managers with the training and development of team members Handles employee relations matters by ...
Nov 13 -
HR Generalist-- Talent Acquistion Specialist
Salary is 75-85k Plus Bonus
Location is Knoxville , TN area
Dynamic, strong growth, manufacturer has an immediate need for a HR Generalist-Talent Acquisition Specialist! Located in the Farragut (Knoxville), TN area!
This position supports the US Business Unit's ...
Director of Human Resources
Director Human Resources - Manufacturing
Our client is an automotive supplier, and they are looking to add a senior level Human Resources Director to their operations. This position will lead HR and Safety/EHS, including:
Employee relations
Plant EHS - Safety and ...
Nov 12 -
We'll be sending text message alerts for your subscription to


text:
DESCRIPTION: 1930 Ford A 2 door sedan. 350, 350, with all chrome Corvette rear, pro built tube chassis with i.f.s., 4 wheel power disc brakes, rack & pinion steering, tilt & tele, power windows, power antenna, pow...More Info
DESCRIPTION: 1928 Chevrolet Business coupe, complete body-off restoration about 5 years ago by Everett Adams, formerly of the Harrah's Auto Collection in NV. Beautiful little car & it runs great!
DESCRIPTION: 1929 Ford Model A sport coupe. V-6 Ford with T-5 manual transmission & 10 bolt rear with adjustable coil overs suspension, Super Bell front with disc brakes all on a custom boxed frame. Air conditioni...More Info
DESCRIPTION: 2007 Saturn Sky convertible. 2.4L 4 cylinder, 5 speed automatic, leather, 6 disc CD/MP3, Decklid spoiler, sunburst yelow with black top. California car with 9,200 miles. Like New!
DESCRIPTION: 1997 Ferrari F355 Spider. Beautiful Tour de France blue with tan leather interior and top boot. 6 speed, 29,700 miles. This Spider has just had a complete extensive mechanical service and is ready to ...More Info
DESCRIPTION: 1960 Chevrolet Impala 2-door hardtop. 348 4 BBL, t-h 350 transmission, power steering, power brakes, beautiful fawn beige over suntan copper paint with copper interior. Outstanding fit & trim! Origina...More Info
DESCRIPTION: 1965 Chevrolet Corvette coupe. 327, 350 h.p., 4 speed, telescopic steering column, day/night rear view mirror, silver pearl with silver interior, numbers matching car that has basically not been resto...More Info
DESCRIPTION: 1966 Ford Mustang GT fastback. Original California rust-free car built in San Jose with 76,680 miles. A code, 4 speed, Rally Pac, luxury black interior, style steel wheels & an older red paint job tha...More Info
DESCRIPTION: 1959 Ford Fairlane Galaxie 500 Skyliner retractible. 292 V-8 2 BBL, automatic, power steering, power brakes, after-market under dash air conditioning, beautiful deep blue metallic paint with white top...More Info
DESCRIPTION: 1948 Buick sedanette. This beautiful two-tone Allen Dale green over Aztec green Model 56S has been in California its entire life & appears to have very low milage since new. It was restored a number o...More Info
DESCRIPTION: 1966 Ford Thunderbird Town Landau. 390, auto, power steering, power disc brakes, power windows, power locks, air conditioning, swing away steering column. Outstanding Emberglo paint with original blac...More Info
DESCRIPTION: 1962 Chevrolet Corvette. 340 horsepower, numbers matching, 4 speed, beautiful Roman red paint with black interior. Both tops are in excellent condition. Great looking, running & driving high horsepowe...More Info
DESCRIPTION: 1947 Lincoln Continental MK I convertible. 351 Ford V-8, auto, 9" rear, power steering, power disc brakes, air bag suspension, power windows, power top, tilt, air conditioner, stereo. Tan interior & t...More Info
DESCRIPTION: 1949 Ford pickup. All Ford drivetrain - 5.0 H.O. - roller cam, AOD transmission, posi rear end with Classic Engineering suspension, Mustang II i.f.s., power steering, power 4-wheel disc brakes, tilt w...More Info
DESCRIPTION: 1954 Chevrolet Bel Air convertible. LS-1 350 V-8, 4L60E trans, Corvette I.R.S., power rack & pinion, I.F.S., power 4-wheel disc brakes, air & heat, tilt, power windows & power top. Painted chassis wit...More Info
DESCRIPTION: 1970 Oldsmobile "Ninety-Eight" convertible. 455 V-8, auto & loaded with about every option you could order in 1970. The best part is it has 54,800 miles & still retains most of it's original Ebony bla...More Info
DESCRIPTION: 1930 Ford Model A coupe. Original drivetrain with updated Stromberg down draft carb, side mount, rumble seat with roll down rear window. Older restoration with recent mechanical work. Runs & drives be...More Info
DESCRIPTION: 1934 Ford 5-window coupe. 350 V-8 with cross ram 2 4BBL induction, 5-speed manual transmission, 8" rear with 4-bar suspension front & rear. Entire engine & drivetrain is comprised of polished aluminum...More Info
DESCRIPTION: 1947 Ford convertible. Older body-off restoration on this wonderfully preserved beauty. The flathead V-8 starts quickly, is whisper quiet & performs beautifully. Radio, heater, power top & vintage tur...More Info


text:
Articles & Reviews
News Archives
Display Drivers
Graphics Cards
Operating Systems
Phoronix Premium
Ad-Free Browsing
Support Phoronix
Single-Page Article Viewing
Oracle Has Yet To Clarify Solaris 11 Kernel Source
Published on 09 January 2012 02:14 PM EST
Written by
Michael Larabel
38 Comments
It was one month ago that Phoronix was the first to note
the Solaris 11 kernel source-code was leaked
onto the Internet via Torrent sites. One month later, Oracle still hasn't officially commented on the situation.
A user in the Phoronix Forums was
the first to point out
the source-code had been leaked for Solaris 11, which was
the latest Solaris OS finally released in November
after being in development for years and then closed-up by Oracle (when they
abandoned OpenSolaris
) following the Sun Microsystems acquisition.
After I pointed out on Phoronix itself the Solaris 11 kernel source-code appearing online, the news quickly worked its way around the Internet on other high-profile sites. However, even after appearing on Slashdot, Ars Technica, and other outlets, there still isn't any official commentary from Oracle about the kernel source-code being available, with some parts of it still being marked as CDDL licensed, which only makes the situation more confusing.
The source to this latest Solaris kernel can still be found via Torrents, but Oracle just isn't commenting -- or they might just not even care at all.
This though has led to some confusion among what's left of the open-source Solaris community. The OpenIndiana developers have been
discussing the situation
. Until Oracle officially blesses an open-source Solaris 11 kernel or provides an answer about this situation, they're staying as far away from this kernel source as possible. These developers obviously fear legal repercussions if this source leak was unauthorized and the legal-happy Oracle takes action.
Alan Coopersmith, the X.Org engineer now at Oracle who works on Solaris going back many years in the Sun Microsystems days, did
comment in the OpenIndiana thread
. However, his comments weren't really anything to fully clarify the situation and just talking about the CDDL, though his words do make this leak wasn't officially sanctioned. "Without any sort of Oracle statement that this is an intentional release, and with no links to it from an official Oracle site (including opensolaris.org), you have to assume it's unauthorized."
Additionally from Alan, "Have you checked every file in it for such markings? Preparing the source releases is not a simple process, and if someone unfamiliar with that process was to just grab it, they might well include such files, either by mistake or malicious intent."
The ones really at a loss here is just the Illumos / OpenIndiana communities that could benefit from the advancements made within the Solaris 11 kernel. At least for many, Solaris is becoming less relevant with Linux having a
ZFS file-system
port (in way of a FUSE implementation and an out-of-tree kernel module; Btrfs also serves as an open-source alternative to the Sun file-system), SystemTap as an alternative to DTrace, and
Oracle is supposedly bringing DTrace to Linux
in some form (these are all Solaris selling points at the moment).
The FreeBSD project has also picked-up and integrated key CDDL-licensed components into their operating system like ZFS and DTrace, but without carrying the baggage of Oracle. With Oracle continuing to push harder on Oracle Enterprise Linux (their Red Hat Enterprise Linux based distribution), hopefully Solaris 11 will be the last and finally mark the end of the SunOS days.
38 Comments
News Archives
Recent Oracle News
Oracle & Canonical Collaborate Over Their Competing Linux OSes On OpenStack
Oracle Linux 7 Released Today As Its RHEL7 Clone
Oracle Solaris 11.2 Now Available In Beta Form
Oracle Linux 6.5 vs. Oracle Linux 7.0 Beta Benchmarks
Oracle Finally Releases Java 8
Java JDK 8 Release Candidate Finally Arrives
Benchmarking Oracle's Unbreakable Enterprise Kernel
OpenSXCE 2013.05 Revives The Solaris Community
Popular News
Intel's Xeon Phi Is Being Sold For An Insanely Low Price Right Now
Mozilla's Servo Engine Is Crazy Fast Compared To Gecko
Microsoft To Open-Source .NET, Bring It Officially To Linux
NEMO-UX Shell Is A Futuristic, Multi-User Wayland Experience
Debian Init System Coupling Vote Results
FreeBSD Receives A Million Dollar Donation
LibreOffice 4.4 To Have A New OpenGL Back-End
Fedora 23~25 Might Switch Away From Firefox
About The Author
Michael Larabel is the principal author of Phoronix.com and founded the web-site in 2004 with a focus on enriching the Linux hardware experience and being the largest web-site devoted to Linux hardware reviews, particularly for products relevant to Linux gamers and enthusiasts but also commonly reviewing servers/workstations and embedded Linux devices. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics hardware drivers, and other topics. Michael is also the lead developer of the
Phoronix Test Suite
, and
automated testing software. He can be followed via
or contacted via
Latest Linux Hardware Reviews
Intel Xeon E5-1680 v3 & E5-2687W v3 Compared To The Core i7 5960X On Linux
Intel 120GB 530 Series SSD Linux Performance
Btrfs/EXT4/XFS/F2FS RAID 0/1/5/6/10 Linux Benchmarks On Four SSDs
AMD's Windows Catalyst Driver Remains Largely Faster Than Linux Drivers
Latest Linux Articles
NVIDIA vs. Nouveau Drivers With Linux 3.18 + Mesa 10.4-devel
Is The Open-Source NVIDIA Driver Fast Enough For Steam On Linux Gaming?
Linux 3.18 File-System Performance Minimally Changed But Possible Regressions
AMD Radeon Gallium3D Is Catching Up & Sometimes Beating Catalyst On Linux
Latest Linux News
Linux 3.18 Kernel: Not Much Change With Intel Haswell Performance
More File-System Tests Of The Linux 3.18 Kernel
Using NVIDIA's NVENC On Linux With FFmpeg
There's Talk Again About An "Open To The Core" Ubuntu Laptop
PowerVR SGX Driver Code Gets Leaked
V2 Of KDBUS Published For Linux Kernel Review
VirtualBox 4.3.20 Arrives, Still No Sign Of VirtualBox 4.4
Scientific Linux 6.6 vs. Scientific Linux 7.0 Benchmarks
Qualcomm Looks To Get Into The ARM Server Business
HHVM 3.4 Adds New Features, Support
Latest Forum Discussions
Roadmap to Catalyst 14.10 ?
Updated and Optimized Ubuntu Free Graphics Drivers
Cant get working Kaveri APU - A10-7850k
Debian Developer Resigns From The Systemd Maintainership Team
Script for Fan Speed Control
Debian Init System Coupling Vote Results
The Slides Announcing The New "AMDGPU" Kernel Driver
Ubuntu Developers Still Thinking What To Do About Adobe Flash Support
Legal Disclaimer & Disclosure
Contact Us
Copyright © 2004 - 2014 by
Phoronix Media
All trademarks used are properties of their respective owners. All rights reserved.
Powered by the Phoronix Content Management System (PHXCMS-9).


text:
Pre-Application Training
Developing and Writing Your ANA Application
The 2015 Funding Opportunity Announcements will be out soon and the Eastern Region Training and Technical Assistance Center, a resource of ANA, will be offering a Pre-Application Training at the end of the 2014 USET Annual Meeting. The training is designed to provide prospective ANA applicants with "nuts and bolts" information on the federal application process and the ANA Funding Opportunity Announcements.
Participants learn how to:
- Register and apply on grants.gov
- Understand the ANA Funding Opportunity Announcement Evaluation Criteria
Format, package, structure and submit an application for ANA funding consideration
The training sessions are offered free of charge to tribes and Native American, Native Alaskan, Native Hawaiian and Pacific Islander non-profit organizations. Each participant is responsible for paying their own travel costs to attend. Pre-application training participants should have a defined project idea.
Please limit your attendees to 3 people/organization. Also, independent grant writers must fax, (571) 323-2102, a signed letter from the Tribe or Native organization you'll be writing the grant for prior to the training in order to attend.


text:
Maria Sharapova Wimbledon Outfit 2013 [PHOTOS]; White Nike Dress Unveiled on Twitter [VIDEO]
Maria Sharapova unveiled her new Wimbledon outfit on her Twitter account Tuesday.
Maria Sharapova unveiled her new Wimbledon outfit on her Twitter account Tuesday.
The outfit, follows the All England Club dress code, which requires white to be the majority color of all attire, Fox noted.
"Wimbledon is around the corner, and my new @niketennis dress is already here! What do you think about it?," she tweeted along with a picture.
The 26-year-old Russian tennis player lost in the final round of the 2013 French Open Tournament last weekend to Serena Williams.
"Eleven years," Williams said in French during the ceremony. "I think it's unbelievable. Now I have 16 Grand Slam titles. It's difficult for me to speak because I'm so excited."
Williams cruised through the game, winning in straight sets 6-4, 6-4 to win her second French Open victory and 16th Grand Slam title of her career. She completed 10 aces, including three in the championship game, and the last point of the game was her fastest serve of the day at 123 mph. She fell to her knees, screamed and buried her face in the clay.
Meanwhile, Sharapova took to her Twitter to express her appeciation to her fans.
"Thank you #Paris for a wonderful two weeks. Despite losing today, I cant wait to be back next year! #rolandgarros 2014!!" she tweeted.
Days after Sharapova's disappointing loss, she was spotted cheering on her new boyfriend, Grigor Dimitrov, at the Aegon Championship at the Queen's Club in West London, according to The Daily Mail.
The pair was seen last month cuddling on a Madrid street as Sharapova tugged on his arm and rested her head on his shoulder.
The couple's romance comes only months after the tennis star announced her split from fiance, Slovenia Sasha Vujacic.
At the time, Sharapova hoped that they could remain friends, but because Vujacic was unable to return home for 10 months because he was playing professional basketball in Turkey, it put a burden on their relationship, "making it 'extremely difficult."
Wimbledon kicks off June 24, 2013.
Photo Galleries
Kristen Stewart, Robert Pattinson 2014: 'Twilight' Actor & FKA Twigs Show Some Naughty PDA In New PHOTO [VIDEO]- 1 Photos
'Real Housewives of Beverly Hills' News: Kyle Richards & Kris Jenner Might Be New BFF's, Could The 'Kardashians' Star Join The Cast Of RHOBH? [PHOTOS, VIDEOS]- 1 Photos
Cristiano Ronaldo News: Dressing To Impress & Riding In A Rolls-Royce [PHOTO, VIDEO]- 1 Photos
'Party Down South' Cast: Taylor 'Lil Bit' Wright Dishes On Season 2 Drama, Baby Bit & How She's Now 'Stronger' And 'Wiser' [EXCLUSIVE]- 1 Photos


text:
A method of (and system for) recovering the state of a failed node in a distributed shared memory system, includes directing a flush of data from a failed node, and flushing the data from the failed node to a memory node.
1. A method of recovering the state of a failed node in a distributed shared memory system, comprising:
directing a flush of data from a failed node; and
flushing said data from said failed node to a memory node,
wherein said flushing includes performing a local flush on said failed node followed by a remote flush.
2. The method of
determining whether said flushing is complete.
3. The method of
determining all lines having a �Modified� status on said failed node; and
examining a directory of said memory node and ensuring that said all lines having said �Modified� status on said failed node have been written back to said memory node.
4. The method of
locally flushing data from said failed node.
5. The method of
remotely flushing said data from said failed node.
6. The method of
detecting a failed node.
7. The method of
8. The method of
9. The method of
10. A system for recovering a state of a failed node in a distributed shared memory system, comprising:
a controller for directing a flush of data from a failed node; and
a flush engine for flushing said data from said failed node to a memory node,
wherein said flush engine includes a local flush engine for performing a local flush on said failed node and a remote flush engine for performing, after said local flush a remote flush.
11. The system of
a judging unit for determining whether said flushing is complete.
12. The system of
a first unit for determining all lines having a �Modified� status on said failed node; and
a second unit for examining a directory of said memory node and ensuring that said all lines having said �Modified� status on said failed node have been written back to said memory node.
13. The system of
14. The system of
15. The system of
a detector for detecting a failed node.
16. The system of
17. The system of
18. The system of
19. A system for recovering a state of a failed node in a distributed shared memory system, comprising:
means for directing a flush of data from a failed node; and
means for flushing said data from said failed node to a memory node,
wherein said flushing means includes a local flush engine for performing a local flush on said failed node and a remote flush engine for performing, after said local flush, a remote flush.
20. A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform a method of recovering the state of a failed node in a distributed shared memory system, comprising:
directing a flush of data from a failed node; and
flushing said data from said failed node to a memory node,
wherein said flushing includes performing a local flush on said failed node followed by a remote flush.
21. The system of
22. The signal-bearing medium of
1. Field of the Invention
The present invention relates to a method and system for retrieving the state of a failed CPU, cache, and Memory node to a nonfaulty location for failure recovery purposes.
2. Description of the Related Art
Prior to the present invention, conventional systems have been unable to adequately deal with a failed computer processor, cache, or memory node.
CPU/cache nodes can fail due to permanent hardware faults, transient hardware faults, operator errors, environmental errors, and software faults. Whenever such a fault occurs in a CPU node, it can no longer participate as an operational member of the system, and the state locked in its cache hierarchy must either be retrieved or reconstructed. This process is a critical part of error recovery.
Moreover, the technology advances in cache and memory technology have recently created a new type of problem in attempting to retrieve the state that is locked in the caches of a failed CPU/cache node.
Some state-of-the-art systems utilize an L3 cache which is a new, relatively large cache (e.g., 1 Gbyte or larger). Because the L3 cache in this type of system is quite large (1 GByte or larger), a huge amount of state data is stored there at the time of any fault, and it is quite difficult to efficiently reconstruct this state with the conventional techniques.
Indeed, the conventional systems have not even addressed such a problem yet since the L3 cache is relatively new and has been incorporated only into relatively new architectures. Indeed, in the past, the conventional systems have dealt only with relatively small caches (e.g., on the order of 100 Kbytes to 1 Mbyte). Thus, when such a cache (1000 times smaller than the new L3 cache) failed, a relatively small amount of data was lost and the recovery to obtain such data was commensurately smaller (e.g., 1000 times smaller). Thus, with the new cache, a significant amount of data is lost and a significant amount of work and time is involved in attempting to recover such an L3 cache and reload it to the system.
Hereinbelow, a simplified description will be provided of the failure recovery protocol that is necessary when the present invention is not employed.
A conventional solution to failure recovery is oriented around the assumption that when a CPU/cache node fails, all of its nonvolatile state (including that in memory) is lost and must be restored from disk. To make this possible, at some prior point in time (and periodically thereafter) the application stores a full copy of its entire state on disk. Thereafter, as state updates are made during the course of normal execution, these updates are also logged to disk. Thus, the disk contains information adequate to reconstruct the state of the application up to and including the last state update.
In the event of a failure, the state of the system is reconstructed by first loading the last full copy of its state that was saved to disk, and then applying the logged state updates to bring the system state up to date.
If a shared memory processor (SMP) instance contains large amounts of memory (e.g., 64 GB or larger), then the time required to bring a copy into memory from disk and then apply the updates can be several minutes. In addition, the system state is only as recent as the last update to disk. More recent updates will have been lost since they were not logged to disk.
Thus, prior to the invention, there was no efficient and adequate way to recover a failed computer processor, cache, or memory node having a large state size.
In view of the foregoing problems, drawbacks, and disadvantages of the conventional methods, it is an object of the present invention to provide a structure and method for the rapid recovery of the state of a failed CPU/cache/memory node in a distributed shared memory system.
In a first aspect, a method of (and system for) recovering a state of a failed node in a distributed shared memory system, includes directing a flush of data from a failed node, and flushing the data from the failed node to a memory node.
In a second aspect, a system for recovering a state of a failed node in a distributed shared memory system, includes a controller for directing a flush of data from a failed node, and a flush engine for flushing the data from the failed node to a memory node.
In a third aspect, a signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform a method of recovering the state of a failed node in a distributed shared memory system, includes directing a flush of data from a failed node, and flushing the data from the failed node to a memory node.
With the unique and unobvious features and aspects of the invention, a method and system are provided which efficiently retrieve the state that is locked in the caches of a failed CPU/cache node, and especially a large size cache such as an L3 cache (e.g., 1 GByte or larger), such that the relatively large amount of state stored there at the time of the fault, can be efficiently reconstructed.
The foregoing and other purposes, aspects and advantages will be better understood from the following detailed description of a preferred embodiment of the invention with reference to the drawings, in which:
FIG. 1 illustrates an exemplary architecture 100 in which the structure and method of the present invention operate;
FIGS. 2 illustrates a processor node and more specifically a connection between a central processing unit (CPU/L2) node to/from a interconnect network shown in FIG. 1;
FIG. 3 illustrates a memory node and more specifically a connection between a memory node to/from the interconnect network shown in FIG. 1;
FIG. 4 illustrates a failed processor node and shows how to get data out of a cache of a failed processor node;
FIG. 5 illustrates a method 500 of data and error recovery in a failed node according to the present invention;
FIG. 6 illustrates an exemplary hardware/information handling system 600 for incorporating the present invention therein; and
FIG. 7 illustrates a signal bearing medium 700 (e.g., storage medium) for storing steps of a program of the method according to the present invention.
Referring now to the drawings, and more particularly to FIGS. 1-7, there are shown preferred embodiments of the method and structures according to the present invention.
Generally, the present invention is relevant to an environment where there are a number of CPU/cache/Memory nodes (henceforth referred to as �CPU nodes�) interconnected such that each can �snoop� data from the other's cache memory, or read the other's memory directly. The physical architecture of the system is shown in FIG. 1 and described below.
Specifically, FIG. 1 shows the physical architecture (e.g., system 100) of a highly modular scalable computer architecture in which the present invention operates. This architecture is for accommodating flexible and dynamic grouping of processing, input/output (I/O), and storage resources into multiple shared memory processor (SMP) instances which themselves communicate over an interconnection fabric.
The system 100 includes central processing unit (CPU)/cache nodes 101, I/O nodes 102, and memory nodes 103 all connected via an interconnect fabric 104. These subsystems are each described below.
CPU/Cache Nodes
In FIG. 1 (and FIG. 2 which illustrates the processor nodes in greater detail), each �SMP CPU/cache Node� 101 contains multiple processors/level 2 (L2) nodes 101 a, a Level 3 (L3) cache node 101 b, and cache coherency and communication circuitry 101 c (e.g., shown in FIG. 2) that allows the node to participate as a member of a multi-node SMP. Further, a local L3 flush engine 101 d and local L3 directory 101 e are included. These nodes provide the processing power of the architecture.
The architecture is flexible with respect to groupings of CPU/cache nodes into SMP instances. For example, each CPU/cache node can operate as a single instance of a symmetric multiprocessor (e.g., running one copy of an operating system), multiple CPU/cache nodes can be aggregated into a single instance of an SMP, or CPU/cache nodes can be allocated to multiple instances of a symmetric multiprocessor which communicate via the interconnect network 104 shown in FIG. 1.
The present invention is equally applicable to all such configurations mentioned above, but the preferred embodiment will be described below as applying to the case where a CPU/cache node includes a single instance of an SMP, with its recovery being orchestrated by another SMP instance.
A CPU/cache node may or may not contain significant amounts of memory (e.g., a few hundred megabytes) other than the L2/L3 caches. In the preferred embodiment of the invention, CPU/cache nodes do not contain significant amounts of memory other than the L2/L3 caches, which are relatively small high-speed memories containing frequently accessed data and instructions. Keeping this information locally in L2/L3 caches rather than remotely in memory nodes significantly speeds up access thereto, and hence overall performance. The caches are assumed to operate on a line of data, which is on the order of 128 bytes, at a time.
Hence, most memory (e.g., if the memory is 1 TB, then a 1 GB L3 cache contains 0.1% of memory, and an 8 MB L2 cache contains 0.001% memory) resides in Memory nodes, to which all nodes in the system have access. For example, an L2 cache may be 8 MByte, an L3 cache may be 1 GByte, and the memory used by a processor may be 1 TByte.
It is noted that the CPU/cache node 101 contains the Local L3 Directory 101 e (e.g., see FIG. 2) that indicates which lines are held in the local L3 cache, and whether they are held in the �Modified� (e.g., also referred to as �Dirty�), �Exclusive� (also referred to as �Clean�), or �Shared� state.
If the line is held �Modified�, then the local CPU/cache node has updated the line relative to all other copies in the system, which are effectively stale and must not be used. Typically, these will be the only lines which must be flushed, either locally or remotely, since the �Modified� data represents data which has been modified and not yet sent/saved to the memory node.
If the line is held �Exclusive�, then the local CPU/cache node has the sole �legal� copy of the line in the system and may modify it at any time (e.g., at which time it would transition to the �Modified� state). Such modification may occur without having to notify the other nodes.
If the line is held �Shared�, then the local CPU/cache node may be sharing the line with another CPU/cache, Memory, or I/O node, but cannot modify it without participating in a cache coherency protocol with the other holders of that line, which renders it �Exclusive�.
Thus, not all of the data needs to be flushed. Only the �Modified� data needs to be flushed.
In one embodiment of the present invention, as shown in FIG. 2 and as mentioned briefly above, each CPU/cache node contains specialized circuitry called the Local Flush Engine 101 d for flushing (writing) Modified or Exclusive L2 and L3 cache lines from that node over the interconnect network 104 to some other CPU/cache node 101 or Memory node 103 connected to the interconnection network 104. This circuitry 101 d can be commanded to perform a flush by either the CPU/cache node 101 containing the circuitry (i.e., the local node), or by specialized circuitry (e.g., Remote L3 Flush Engine 101 f) on another CPU/cache node 101 or Memory node 103 that wishes to extract the contents of the local node's L2 and L3 caches (the recovery agent).
When modified data is flushed from an L3 cache 101 b, it is sent over the interconnect 104 and written to a Memory node 103, as described below.
An alternative location for the Remote Flush Engine 101 f (e.g., thereby constituting an alternative embodiment of the present invention) in the memory node, as described below in regard to the Memory node.
Memory Nodes
In the architecture 100 described above, a number of CPU/cache 101 nodes share physical memory 103 a (e.g., RAM) that resides in one or more Memory nodes 103 (e.g., as shown in FIG. 1 and in greater detail in FIG. 3). A Memory node 103 may contain the memory of one or more SMP instances, each comprising one or more CPU/cache nodes 101.
The Memory node 103 is connected to the CPU/cache nodes 101, I/O nodes 102, and other Memory nodes 103 via the interconnect fabric 104. Using its coherency control chipset 103 b, it participates in the cache coherency protocols necessary to keep the various copies of the lines loaned out to the various CPU/cache nodes 101 coherent and consistent. All components of the Memory node can be made reliable with respect to hardware faults and power outages by relatively straightforward encoding, replication, and power backup techniques. Thus, the memory node constitutes a stable repository of all data committed thereto.
Along with the coherency control chipset 103 a, the memory node includes a Global Directory 103 c that keeps track of all lines that are currently held by CPU/cache and I/O nodes. Directory 103 c lists every line loaned out from the memory that is held �Exclusively� by and could possibly be modified by any given node. The information in the Global Directory 103 c is useful in confirming that the L2/L3 cache of a particular node has been fully and successfully flushed.
In an alternative embodiment of the invention, as mentioned above, the Memory node 103 also contains a Remote Flush Engine 103 d for generating and transmitting reads of a given set of memory locations.
As part of the normal cache coherency protocol, when any node intercepts a read of a line that is held locally in either the �Exclusive� or �Modified� state, it replies with the data from its cache, which is then written back into the Memory node 103. At this point, the Global Directory 103 c is updated to indicate that the line in question is no longer held by the node in question.
I/O Nodes
Input/Output nodes 102 reside on the Interconnect network 104 and allow the CPU/cache and Memory nodes 101/103 to pass data to the environment external to the computer system via devices such as network, disk, serial communication, and parallel communication devices.
The CPU/cache, memory, and I/O nodes 101/103/102 are connected by the interconnect network 104 which is a high-performance interconnect network. For the purposes of the present invention, preferably the interconnect network 104 has low latency (e.g., preferably in the tens of nanoseconds), high bandwidth (e.g., preferably in the tens of GB/sec), is fault tolerant (e.g., capable of tolerating any fault and continuing operation), and it permits communication between all nodes when one of the nodes is faulty.
Normal Operation
Hereinbelow is described a simplified version of a representative cache coherence protocol used in normal operation of an SMP instance.
Under normal operation, the CPU/cache nodes 101 fetch instructions and data from the Memory node 103, execute the instructions, and write back results from the instruction execution. Any instructions or data fetched from the Memory node 103 are stored in the CPU/cache node's L2 and/or L3 caches (e.g., depending upon whether the LD cache is an inclusive cache or exclusive cache). That is, in an inclusive cache, the processor will try to fetch an instruction and will look in the L2 cache, and assuming that the instruction is not there, then the processor will look in the L3 cache. Assuming that the instruction is not in the L3 cache, then a request will be sent to memory and the line (e.g., having the instruction) is retrieved from the memory. Then, the line is installed in both the L2 and the L3 caches.
In an exclusive cache, the line is installed only in the L2 cache.
When a CPU/cache node 101 that requires a line of data does not find it in its local caches (e.g., L2 or L3), it transmits a read request for that line over the Interconnection network 104. This read request is �snooped� (e.g., received) by all CPU/caches 101 that reside in that SMP instance, as well as by the Memory node 103 containing the mainstore for that SMP instance.
If any other CPU/cache node 101 possesses the line in the �Exclusive� or �Modified� state, it demotes its ownership state to �Shared� and provides the line to the requesting CPU/cache node. The Memory node's Global Directory 103 c is also updated to indicate the new state of the line.
If no CPU/cache node 101 possesses the line in the �Exclusive� or �Modified� state, the Memory node 103 takes on the responsibility of providing the line to the requesting CPU/cache node 101, and updating its Global Directory accordingly. When the CPU/cache node 101 reads a line from the Memory node 103, the node's Local L3 cache directory and the Memory node's Global Directory 103 c are updated to reflect the location of any line of memory, and whether it was requested to be in the �Modified�, �Exclusive�, or �Shared� state.
Thus, at any given time, a CPU/cache node's caches (e.g., both L2 and L3) contain a mixture of lines in the �Exclusive�, �Modified�, and �Shared� state, and the CPU/cache's Local L3 Directory 101 e, as well as the Memory node's Global Directory 103c, contain a record of this information. If a CPU/cache node 101 fails, then the �Exclusive� and �Modified� lines residing in its caches, of which the failed node has the sole copy, must be recovered to a nonfaulty location, such as the Memory node 103.
Recovery from Failures
The present invention exploits the fact that the preponderance of faults that occur in practice do not totally disable a CPU node 101 to the extent that it cannot participate in system-wide cache coherency protocols, or respond to data reads from another node.
For example, when a CPU node 101 suffers a software fault, the hardware is still perfectly usable and can participate in cache coherency protocols.
As another example, when the CPU fails but the remainder of the node 101 is nonfaulty, assuming the CPU failure did not disable the remainder of the node, the L3 cache is still functional and can be accessed from outside the faulty CPU node. It is believed that many transient and permanent hardware faults will in fact leave the L3 and associated circuitry operational. When this occurs, an external recovery agent can access the memory locked in the L3 and L2 caches and cause it to be forced back out to the Memory node 103. Thus, it is noted that there are generally two types of failure examples when the CPU fails. That is, there are permanent failures and transient failures. Permanent failures occur when the hardware simply �dies� (becomes inoperable) in which a physical repair must be undertaken to make the CPU operable again. Transient failures are one-time glitches or system �hiccups� in which the system dies but in which the user may merely reboot or reset the system to revive the system. As evident from the example above, a permanent failure is envisioned in which the CPU failure did not disable the rest of the node, but the L3 cache is still operable. By the same token, a transient failure may occur in which no part of the node is disabled. Transient failures are much more common than permanent failures. With transient failures, the hardware is still available and can participate in its recovery, whereas, with permanent failures, there is a good chance that a sufficient amount of the hardware still may be available to participate in the recovery.
The present invention also exploits the fact that the Memory node can be made to be reliable as outlined above, and therefore need not be reloaded from disk in the event of a CPU/cache node 101 failure. All updates to memory are stable. Moreover, if state updates are either logged into or applied directly to this stable memory, they need not be retrieved from disk and replayed. This reduces the recovery time by the time required to load the full state copy and log from disk and replay the log.
However, even with these benefits of stable memory, �Exclusive� data and �Modified� data remain locked in the cache hierarchy of the failed CPU/cache node. The invention solves this problem, as described below.
Preferred Data Recovery Method/Algorithm
Hereinbelow (and referring to FIG. 4) is described the data recovery method/algorithm according to the present invention when one SMP instance detects a node failure and controls the recovery of another node. The invention generalizes in a straightforward way to other configurations of the server architecture.
In a preferred implementation, the data recovery algorithm utilizes four subsystems: the faulted CPU/cache node 101 (in FIG. 4 designated as �Processor/Cache Node A�, another CPU/cache node 101′ called a �recovery agent� (in FIG. 4, designated as �Processor/Cache Node B�), the interconnect 104, and the Memory node 103.
For illustration purposes, the faulted CPU/cache node 101 comprises one SMP instance, and the recovery agent comprises another. The description of the recovery algorithm proceeds from the point in time at which the faulted node fails and will be described with reference to the exemplary architecture shown in FIG. 4.
First, assume that at the time that CPU/cache node A fails, it contains memory locations X, Y, and Z in an �Exclusive� state. This information is known within CPU/cache node A as well as within the global directory of the Memory node 103.
Error Detection
Recovery Agent
The recovery agent (e.g., in this case CPU/cache node B) detects that the faulted node (e.g., in this case CPU/cache node A) has ceased to participate in the system operation by observing an unusual period of quiescence from the faulted node, or by observing a violation of a protocol by the faulted node.
Subsequently, the recovery agent causes the system (other CPU/cache nodes 101 and I/O nodes 102 (shown in FIG. 1) associated with the SMP instance containing the faulty CPU/cache node A) to halt operation to allow the recovery process to proceed.
In addition, if the faulted SMP instance is a part of a distributed system known as a �cluster�, which possesses SMP instance-level recovery techniques, these techniques may also be invoked at this time. These functions can be accomplished by transmitting special messages over the interconnect 104 to all involved nodes.
Faulted (Failed) Node
When the faulted node has detected its own failure (e.g., in response to a non-maskable interrupt, panic, or other indicator of catastrophic failure), it can optionally flush its L2 cache down into its L3 cache as part of its failure shutdown procedure.
This flushing ensures that the L2 cache does not contain any �Exclusive� or �Modified� lines, and can be used to accelerate the process of flushing the L3 cache since subsequent snoops to these lines will not be required. This option is not necessary for correctness, however, and a faulted node that has failed to flush its L2 cache into its L3 cache can still be recovered. It is merely necessary for the L3 cache to snoop the L2 cache for any �dirty� (�Modified) lines during subsequent phases of the recovery process.
State Recovery
In the next step of the recovery protocol, the faulted node's L2/L3 caches are flushed back into the Memory node 103. Two alternatives exist for flushing the faulted node's caches back to the memory node.
In the first alternative, each CPU/cache node 101 contains specialized circuitry called the Local Flush Engine 101 d (e.g., see FIGS. 2 and 4) that causes the L3 cache to flush �dirty� data.
In the second alternative, the CPU/cache node 101 contains no such extra circuitry, and the flushing is controlled by special circuitry in the Memory node 103 called the Remote Flush Engine 103 d. The advantage of the second alternative is that no additional circuitry needs to be added to the CPU/cache node in order for it to participate in the data recovery protocol, and thus the cost and complexity size of the node is reduced, thereby allowing the use of industry-standard nodes not customized to take advantage of the present invention.
The recovery agent transmits a command to the faulted node to confirm that it is able to participate in the recovery protocol. This command enters the faulted node at its interconnect port and communicates with protocol assist circuitry (not illustrated) within the local flush engine 101 d, if the node has one, or the service processor 101 e.
The protocol assist circuitry performs basic integrity tests on the node (i.e., ensures the L3 cache can be read), and then responds to the recovery agent.
If the response is in the affirmative, the recovery proceeds. Otherwise, a traditional recovery from disk is necessary (e.g., the �traditional� recovery being what is described above in the �Description of the Related Art� section).
Alternative 1: Local Flush Engine
When the CPU/cache node 101 possesses a local L3 flush engine 101 d, the local flush engine 101 d traverses the node's Local L3 Directory 101e and flushes every line from the L3 cache that is marked �Exclusive� or �Modified�. These lines are sent out the interconnect port and are written to the Memory node 103, and the Memory node's Global Directory 103 c is updated to reflect that the line is now no longer owned by the faulted node.
If a more recent copy of a line may reside in the node's L2 cache, as indicated by the Local L3 Directory 101 e, the L2 cache is snooped and the updated line is retrieved prior to being transmitted back to the Memory node 103. When all of the �Exclusive� or �Modified� lines have been written back, the recovery is complete.
Thus, in the above example when the faulted node contains X, Y, and Z in the �Exclusive� state, the Local Flush Engine 101 d writes X, Y, and Z back to the Memory node 103. If at any point the local flush engine encounters an uncorrectable error (e.g., the L2 cache has a �dirty� line that cannot be snooped due to a failure in the CPU itself, or an L3 cache read encounters an uncorrectable error), a modified traditional recovery from stable storage is performed (as discussed below).
Alternative 2: Remote Flush Engine
When the flushing is to be controlled by the Remote Flush Engine 103 d in the Memory node 103, the recovery agent commands the Memory node 103 to begin the flush process.
The Remote Flush Engine 103 d traverses the Global L3 Directory 103c and issues a Read for every line that is held �Exclusive� or �Modified� by the faulted node. These Reads will cause the faulted node's snoop circuitry (e.g., the snoop circuitry is believed to be well known in the art and, for brevity, will not be described further herein) to write back the line to the Memory node 103 and update its Local L3 Directory 101 e, thereby to indicate that it no longer owns the line.
When the line arrives back at the Memory node 103, the Memory node 103 updates its Global L3 Directory 103 c to indicate that the faulted node no longer owns the line, and moves on to the next Global L3 Directory entry that is owned by the faulted node.
In the example where the faulted node contains X, Y, and Z, the Remote L3 Flush Engine 103 d transmits reads of lines X, Y, and Z into the interconnection network 104. The faulted node's cache coherency circuitry snoops these reads and provides the values of X, Y, and Z back to the requesting Memory node 103.
If the Remote Flush Engine 103 d is able to flush all �dirty� lines owned by the faulted node in this manner, then the recovery has been successful. Otherwise, a modified traditional recovery from stable storage is required.
It is noted that a main difference between the local flush and the remote flush is who (e.g., the agent) is directing the data flush. The data to be read in either case is the same. Thus, after the flush is directed, either by the cache node or by the memory node, the actual flush operation is substantially the same in both cases. Hence, for example, the cache control circuitry on the faulted node will receive, from the either the local flush engine of the failed node or the remote flush engine of the memory node, a message which indicates �Disgorge Line 1� and the circuitiy will write back Line 1 to memory. This writing back operation is the same in either a remote flush or a local flush.
Modified Traditional Recovery from Stable Storage
There may be scenarios in which the effects of the fault will be so severe that the �Exclusive� and �Modified� data resident in the faulted node's L3 cache cannot be fully retrieved. Because it has a list of all lines that are held by the faulted node, the Remote L3 Flush Engine 103 d will be able to detect this and, furthermore, will be able to detect which lines could not be recovered from the faulty node.
Thus, the regions of memory that cannot be retrieved can be clearly identified and must be recovered from stable storage, which can be disk or the Memory node 103. This may require that the atomic operation (e.g., a transaction) in progress at the time of the fault may have to be aborted, backed out, and retried.
If the amount of unrecoverable data is small, then this modified recovery strategy will complete in less time than a full recovery from disk because the unrecoverable data can be strictly circumscribed and limited recovery techniques used as opposed to having to assume that all of the memory must be recovered.
FIG. 5 illustrates a flowchart of the method 500 of the present invention of data and error recovery according to the present invention.
First, in step 501, it is determined whether a failed CPU/cache node is detected. If no, the process continues looping until a failed node is in fact detected.
In step 502, upon detection of a failed CPU/cache node, it is determined how to recover (e.g., is it to be a local recovery or a remote recovery). If a local flush is to be performed (e.g., when speed is desired or the faulted node can control its own flush), then step 503 is performed (e.g., the operation described above). If a remote flush is to be performed (e.g., when reliability is desired or the faulted node cannot control its own flush), then step 504 is performed (e.g., the operation described above).
FIG. 6 illustrates a typical hardware configuration of an information handling/computer system for use with the invention and which preferably has at least one processor or central processing unit (CPU) 611.
The CPUs 611 are interconnected via a system bus 612 to a random access memory (RAM) 614, read-only memory (ROM) 616, input/output (I/O) adapter 618 (for connecting peripheral devices such as disk units 621 and tape drives 640 to the bus 612), user interface adapter 622 (for connecting a keyboard 624, mouse 626, speaker 628, microphone 632, and/or other user interface device to the bus 612), a communication adapter 634 for connecting an information handling system to a data processing network, the Internet, an Intranet, a personal area network (PAN), etc., and a display adapter 636 for connecting the bus 612 to a display device 638 and/or a printer (e.g., a digital printer) 639.
In addition to the hardware/software environment described above, a different aspect of the invention includes a computer-implemented method for performing the above method. As an example, this method may be implemented in the particular environment discussed above.
Such a method may be implemented, for example, by operating a computer, as embodied by a digital data processing apparatus, to execute a sequence of machine-readable instructions. These instructions may reside in various types of signal-bearing media.
This signal-bearing media may include, for example, a RAM contained within the CPU 611, as represented by the fast-access storage for example. Alternatively, the instructions may be contained in another signal-bearing media, such as a magnetic data storage diskette 700 (FIG. 7), directly or indirectly accessible by the CPU 611.
Whether contained in the diskette 700, the computer/CPU 611, or elsewhere, the instructions may be stored on a variety of machine-readable data storage media, such as DASD storage (e.g., a conventional �hard drive� or a RAID array), magnetic tape, electronic read-only memory (e.g., ROM, EPROM, or EEPROM), an optical storage device (e.g. CD-ROM, WORM, DVD, digital optical tape, etc.), paper �punch� cards, or other suitable signal-bearing media including transmission media such as digital and analog and communication links and wireless. In an illustrative embodiment of the invention, the machine-readable instructions may comprise software object code, compiled from a language such as �C�, etc.
With the unique and unobvious features and aspects of the invention, a method and system are provided which efficiently retrieves the state that is locked in the caches of a failed CPU/cache node, and especially a large size cache such as an L3 cache (e.g., 1 GByte or larger), such that the relatively large amount of state stored there at the time of the fault, can be efficiently reconstructed.
That is, the invention provides an augmentation to the coherency control mechanism to salvage the data in a failed node CPU/cache node.
While the invention has been described in terms of several preferred embodiments, those skilled in the art will recognize that the invention can be practiced with modification within the spirit and scope of the appended claims.
For example, it is noted that while the exemplary embodiments above have been primarily applied to an L3 cache, other relative cache sizes may be accommodated and indeed the present invention may be applied to other memory products which rely on larger L2 caches.
It is noted that the above-described first and second alternatives could be employed in combination in sequence. Thus, for example, the first alternative could be tried first (e.g., performing a local flush first), and then if the first alternative fails (e.g., does not result in recovery), the second alternative of performing a remote flushing could be performed. The local flush is faster since the data will be literally poured out of the node. Since the local flush is faster, this may be more advantageous if speed is important. However, the local flush is not as reliable/stable as the remote memory flush, since the local node has in fact failed. Hence, a global remote flush may get all of the data but is typically slower since the memory node must launch an address and �walk� its way through the cache line-by-line.
It is noted that the invention can be implemented in a combination of hardware and software.
For example, the software error detection method can be implemented in software and would detect an error in, for example, Node A and would indicate that Node A has failed and would then send a quiesce command to nodes B, C, and D. Software in nodes B, C, and D would respond to the quiesce command and any of the nodes could send a message to Node A for Node A to begin its flush. Alternatively, the other nodes could send a message to the memory node to direct a remote flush. Then, hardware could take over the operation.
It is noted that the specific hardware circuitry (e.g., gates, switches, etc.) are well-known to those of ordinary skill in the art and thus, for brevity, such circuitry has not been described in detail herein.
Patent Citations
Non-Patent Citations
Referenced by
Legal Events
Original Image


text:
After the break, footage from the unveiling. We've also embedded The Powers of Ten, a 1977 short film documentary that reportedly inspired Will Wright to make the game.
Check out all our extensive coverage of Will Wright's Spore!
Posted on Nov 22nd 2014 9:00AM
Posted on Nov 21st 2014 8:30PM
Posted on Nov 21st 2014 3:00PM