text:
2009 NFL Draft
Post-Workout Rankings
The Top Safety Prospects
Pete Fiutak
2009 NFL Combine Numbers
| Running Backs & Fullbacks
| Receivers
Ends | Centers & Guards
Off. Tackles
Def. Ends |
Def. Tackles
Inside LBs
Outside LBs
new Cornerbacks |
new 2009 NFL Draft Post-Workout
Quarterbacks |
Running Backs |
Fullbacks |
Receivers |
Tight Ends
Centers |
Guards |
Off. Tackles |
Def. Ends |
Def. Tackles
Inside LBs |
Outside LBs |
Cornerbacks |
Full Player Breakdowns & Analysis To Come In March-
2008 CFN Draft Central
2008 NFL Draft Breakdown and Analysis
1st Round
2nd Round
3rd Round
4th Round
5th Round
6th Rd
7th Rd
1. Patrick Chung, Oregon (SS)
5-11, 208Pre-Combine Position Rank: 2
Post-Combine Skinny: Great workout. GREAT workout. The 4.47 was a
major boost to put him among the top defensive backs overall and might
have been enough to solidify a spot in the second round. The 25 reps on
the bench didn't hurt, either.Positives: Can do
everything well at either safety position. Could play corner in a pinch
… The type defensive back to build a secondary around … Tremendous
tackler. Doesn't miss.
Negatives: Might struggle a wee bit in deep coverage. He lacks elite
speed … Not huge for a strong safety … Physical play worked in college,
but it might not be enough, because of his size, in the NFL. 2. Louis Delmas, Western Michigan (FS)
5-11, 202Pre-Combine Position Rank: 3
Post-Combine Skinny: It was an impressive workout that showed he
belongs among the top safety prospects. The 4.49 was better than most
corner, and he was ultra-quick through the short drills. Now he's known
for being athletic as well as a big hitter.Positives:
Experienced veteran leader who produced against everyone … Huge hitter.
HUGE hitter … Fast enough to play corner, a big enough popper to play
strong safety.Negatives:
Goes for the kill shot too often. It gets him in trouble … Not
necessarily a form tackler … A bit lanky. Could use a few more pounds of
Sean Smith, Utah (FS/CB) 6-4,
214Pre-Combine Position Rank: 5
Post-Combine Skinny: Easily one of the biggest stars of the
Combine, he might now be viewed as a very big, very athletic corner.
However, he could also be used as a devastating free safety after
checking in a 6-4 and 214 pounds and tearing off a silky smooth 4.47.
Positives: Excellent size. If
he ends up at corner, He'll have dream size … Surprisingly quick for
being so long. Cuts like a much smaller defender … Good instincts even
though he doesn't have that much experience.
Negatives: He'll have to find
a position right away. He'll likely be looked at as a corner, but his
future will be at free safety. Will he be able to master one spot? … Has
an NFL-sized opinion of his skills … Not great against the run. Won't
provide much pop.4. Rashad Johnson, Alabama (FS)
5-11. 203Pre-Combine Position Rank: 1
Post-Combine Skinny: He appeared to be a bit thicker than he was in
college and he ran a solid 4.54. He was athletic enough to start to
convert those who don't believe in his basic skills. However, he's a
free safety and not a pure strong safety. Positives:
Tremendous worker and a great leader … A playmaker. Made things happen
as the star of a strong defense … Good closing ability when the ball is
in the air.Negatives: Not
an elite athlete. A better football player than he'll be in any workout
… Not all that big. Not a huge hitter … He'll have to get bigger and
stronger or there might be a ceiling on what he can do. 5. Otis
Wiley, Michigan State (FS)
6-2, 213Pre-Combine Position Rank: 7
Post-Combine Skinny: He didn't work out, but he measured a
rock-solid 6-1 and 213 pounds. If he can show off decent athleticism and
come in around 4.55, he'll make himself a lot of money.
Positives: Extremely fast for
his size … A playmaker. Always made things happen as a free safety …
Great tackler who always gets around the ball.
Negatives: Only a free safety.
A disaster his junior year at strong safety … Will have to show he can
provide steady run support … Doesn't always play up to his size.
6. David Bruton, Notre Dame (FS)
6-2, 219Pre-Combine Position Rank: 10
Post-Combine Skinny: It was a special workout from a
high-rising prospect. He was big, fast, and smooth with a 41.5"
vertical, an 11-foot broad jump, and a lightning quick 60-yard drill.
Originally considered a late second day prospect, he looked like a
second rounder.Positives:
A standout on the Irish D with a great combination of size and speed …
Excellent tackler. Looked like he was playing at another level at times
… Great character. Will bust his tail to get better.
Negatives: Not a tone-setting
hitter. A good hitter, but not an elite one … Needs to be a more
consistent tackler … Will have to show he can stay with an NFL receiver.
7. Courtney Greene, Rutgers (SS)
6-1, 212Pre-Combine Position Rank: 12
Post-Combine Skinny: He had a really nice weekend. The 4.54 was
strong, and he appeared to be a natural as he motored through the short
drills. It was the workout that proved he belonged in an NFL backfield,
and with his size, it was a workout that showed he could be special with
a little bit of work.Positives:
Exceptional athleticism … An intimidating force. Makes big hits … Good
speed and quickness when the ball is in the air.
Negatives: Tends to go for the
big hit over the routine play … Short. Tough, but short … A strong
safety at the next level. His skills likely won't translate to free.
8. William Moore, Missouri (SS)
6-0, 220Pre-Combine Position Rank: 4
Post-Combine Skinny: This was going to be when Moore stopped the
free-fall of his stock. It didn't happen. The 4.53 was good enough, but
he didn't appear to be too smooth and he didn't stand out.Positives:
The prototype. He has the size and athleticism to make NFL scouts drool
… Fast. Has no problem staying with any receiver … Good at coming up
with the pick. Great break on the ball.
Negatives: A major,
major disappointment in
post-season workouts coming off a subpar final year … Tried to hard to
come up with a big play and missed the easy ones … Would've been a top
15 pick after his 2007 season. The stock has dropped.9. Derek Pegues, Mississippi State (FS)
5-10, 199Pre-Combine Position Rank: 6
Post-Combine Skinny: It was a disappointing workout considering his
potential. He ran a 4.62, but he came up with a nice 23 reps in the
weight room. He appeared to be trying to do too much. He should be
better in private workouts. Positives:
Versatile veteran who dominated against the best of the best in the SEC
… A major-league playmaker. Deadly with the ball in his hands on special
teams or after a pick … Physical for his size.
Negatives: Small. It wasn't a
problem in the SEC, but it might be in the NFL … Won't provide a
major-league pop … He'll have to prove he can be consistent against the
run against the bigger players.10. Michael Hamlin, Clemson (S)
6-2, 214Pre-Combine Position Rank: 8
Post-Combine Skinny: It was a decent workout for his size. Known
for being a big hitter and a physical player, he was relatively athletic
in the drills. The 4.6 in the 40 was expected, but disappointing. A 4.55
would've made a big difference.Positives:
Versatile. Can play either safety spot … Excellent tackler. Extremely
physical … Great against the run. Doesn't miss.
Negatives: Needs to get
bigger. He has to hit the weights hard … Not an elite athlete. Might be
a free safety playing strong safety, or vice versa depending on his
workouts … Will need to prove he's fast enough on deep balls.
11. Chip Vaughn, Wake Forest (FS)
6-1, 221Pre-Combine Position Rank: 13
Post-Combine Skinny: He was flying through the short drills
like a much smaller player. With his size and speed, he showed off all
the raw tools to potentially be a starter. He looked the part.Positives:
Clutch. A good playmaker when things get tight … Strong with an NFL
body. Big-time hitter … Good athleticism and enough speed to get by.
Negatives: Not a fluid athlete
on the field
… Is a free safety, but plays like a strong safety … Might need a bit of
work. Not instinctive.12. Darcel McBath, Texas Tech (FS)
6-0, 198Pre-Combine Position Rank: 15
Post-Combine Skinny: He's the type of player that everyone
wants. He wants to be a top player, he'll work for it, and he ran well
with a 4.57. He was quick enough in the drills and did them
A leader. He took charge of the back seven … A great combination of
speed and size … A playmaker both against the run and against the pass.
Negatives: Not a popper.
Tackles, but doesn't hit … Isn't the best athlete … Not quite physical
enough to scare anyone at the next level.13.
Kevin Ellison, USC (SS) 6-1,
227Pre-Combine Position Rank: 9
Post-Combine Skinny: It wasn't good. The 32 reps on the bench might
make him more of a linebacker, and the stunningly slow 4.82 in the 40
secures it. Now he has to find a spot to play.Positives: An
experienced veteran who was a leader on a defense full of leaders …
Strong and extremely tough. Very physical … Takes great angles. Doesn't
waste many steps.Negatives:
Bad knees … A pure strong safety. Not athletic enough to play free
safety after the knee injuries … Not great at going after the ball when
it's in the air.14. Curtis Taylor, LSU (FS)
6-2, 209Pre-Combine Position Rank: 11
Post-Combine Skinny: It was a good enough Combine to earn him
longer looks. He came up with a huge broad jump of 10'9", but the 4.64
in the 40 was a problem. He looked faster and quicker than he actually
An NFL body, even if he's a bit lanky … Big hitter. Will explode into
the ball-carrier … Moves well for his size.
Negatives: Marginal instincts.
Seems to have to make up for mistakes … Doesn't do enough to get to the
ball when it's in the air … Tries too hard to make the big pop.
15. Jamarca Sanford, Ole Miss (FS)
5-10, 214
Pre-Combine Position Rank: 14
Post-Combine Skinny: The 4.48 made him a player worth
looking into. The 29 reps made him one of the stars of the weekend.
While he's a bit short, that didn't seem to matter after showing off his
surprising athleticism.Positives:
A veteran. Plenty of experience … Great hitter. Pops like a much
bigger player … Great motor. 100% worker.
Negatives: Not big. Not quite
fast enough to be a top free safety, and not bulky enough to play strong
safety … A player. Not a difference maker, but a player … Won't impress
in workouts.16. Sherrod Martin, Troy (S)
6-1, 198Pre-Combine Position Rank: 22
Post-Combine Skinny: He made a big name for himself as one of
the quickest defensive backs in Indianapolis. The 4.52 was more than
good enough at safety, but the 3.98 shuttle and 6.6 cone drills were
Positives: A top-shelf safety
in college, he projects to be a physical corner … A very productive
playmaker against everyone … Excellent upside with good raw skills.
Speedy and strong.Negatives:
He doesn't have a position. Versatility not necessarily a plus …
Needs major technique work on either position, corner or free safety …
Not a huge hitter for being a safety.17. C.J. Spillman,
Marshall (FS) 6-0, 197Pre-Combine Position Rank: 16
Post-Combine Skinny: Known for being ultra-athletic, he didn't
disappoint. He ran a good 4.5 to punctuate the 41.5" vertical leap and
11.27 60-yard drill. He was one of the best athletes at the Combine.
Positives: Good speed and
quickness … Moves extremely well. Effortless … Tough. Willing to stick
his nose in against the run.
Negatives: Mediocre in pass coverage … A good hitter, but not a
consistent tackler … Not huge. A free safety. Not a strong safety.
18. Stephen Hodge, TCU (SS)
6-0, 234Pre-Combine Position Rank: 19
Post-Combine Skinny: One of the stunners among the defensive
backs, he didn't look like an athletic free safety in the drills, but he
ripped off a 4.49 in the 40 and came up with 22 reps in the weight room.
At 234 pounds, he has the size to go along with the surprising speed.Positives: Fast
for the position. Great on the move … A star special teamer. He'll make
a roster on that alone … Big time hitter. Brings the thunder with every
tackle.Negatives: More of
a linebacker than a true safety … Will have to do far different things
in the NFL than he did at TCU … Fast, but not necessarily quick.
19. Bradley Fletcher, Iowa (FS/CB) 6-0, 196Pre-Combine Position Rank: 21
Post-Combine Skinny: No one will know exactly where to put him,
but he might be the type of prospect who someone will fall in love with.
He could be a lightning fast safety after running a 4.47, or he could be
a big corner.
Positives: Great size for a
corner. Was fantastic in post-season all-star workouts and practices …
Will likely make his money as a nickelback or free safety. Will be a
great centerfielder … Can hit and move.
Negatives: He doesn't have a
position … Not experienced. Only got it done for one year … A good
tackler, but not an intimidating hitter in any way.20.
Marcus McClinton, Kentucky (FS)
6-0, 210Pre-Combine Position Rank: 23
Post-Combine Skinny: A physical, tough defender, he needed to
show off a little speed. He did it. The 4.56 was more than fine for the
type of player he is, and now he might be a fast strong safety and not
just automatically considered a free safety.Positives:
Was productive and has the tools … Great tackler who's extremely
physical … Great closing speed.
Negatives: Major character
concerns … Always banged up. Hits too big and gets hurt … Doesn't play
up to his talent level.21. Emmanuel Cook, South Carolina (FS)
5-9, 197
Pre-Combine Position Rank: 17
Post-Combine Skinny: A linebacker trying to play safety, he ran
like a linebacker with a 4.61. It'll be hard to find a spot for him
after this workout. He didn't show anything to make him look like a
starter. Positives:
Major league hitter. Pops like a linebacker … Veteran starter who
saw it all in the SEC … Extremely quick. Blows through a ball-carrier.
Negatives: Small. Too small …
Not quite quick enough to be an elite free safety. A strong safety
who'll need to play free … Character issues.22. Troy Nolan,
Arizona State (SS) 6-1, 207Pre-Combine Position Rank: 18
Post-Combine Skinny: He'll make his money as a hitter. The 4.62
didn't help his cause, but speed isn't his game. He wasn't expected to
work out well, and he didn't.
Positives: An excellent hitter
… Great playmaker. Aggressive in all phases … Like an extra linebacker.
Negatives: Not all that
athletic. Won't work out well … Tries to go for the big play too often …
Lacks instincts. Isn't always efficient to the ball.
19. Bradley Fletcher, Iowa (FS/CB) 6-0, 196Pre-Combine Position Rank: 21
Post-Combine Skinny: No one will know exactly where to put him,
but he might be the type of prospect who someone will fall in love with.
He could be a lightning fast safety after running a 4.47, or he could be
a big corner.
Positives: Great size for a
corner. Was fantastic in post-season all-star workouts and practices …
Will likely make his money as a nickelback or free safety. Will be a
great centerfielder … Can hit and move.
Negatives: He doesn't have a
position … Not experienced. Only got it done for one year … A good
tackler, but not an intimidating hitter in any way.20.
Marcus McClinton, Kentucky (FS)
6-0, 210Pre-Combine Position Rank: 23
Post-Combine Skinny: A physical, tough defender, he needed to
show off a little speed. He did it. The 4.56 was more than fine for the
type of player he is, and now he might be a fast strong safety and not
just automatically considered a free safety.Positives:
Was productive and has the tools … Great tackler who's extremely
physical … Great closing speed.
Negatives: Major character
concerns … Always banged up. Hits too big and gets hurt … Doesn't play
up to his talent level.21.
23. Lendy Holmes, Oklahoma (FS)
6-1, 206Pre-Combine Position Rank: 20
Post-Combine Skinny: For a player who projects to be a free
safety, the 4.71 40 was a disaster. Some might have thought of him as a
big corner. They don't any longer.Positives:
Great athlete. Fast to the ball … Can be used in a variety of ways. Can
be a big corner in an emergency and could also be used as a pass rusher … Tall. Rangy.
Negatives: Not quite a big
enough hitter to star at safety … Character. Off-the-field issues with
schoolwork … More of a corner playing safety.


text:
Breeze Park Apartments600 Breeze Park Dr, Weldon Spring, MO 63304 View On Map (636) 939-5223
For rates, availability, and other information fill out the availability request form and a representative from Breeze Park Apartments will contact you shortly
Breeze Park is located in the St Louis Area in the 63304 zip code. View this apartment on a map.
Back to Top
Maps & Directions
click to view interactive map

subset: commoncrawl


text:
ONE PIECE Ch. 673 Review
By FoxxFireArt 0 Comments
Punk Hazard Island Arc
I've written these reviews on Anime Vice before, but through it might be interesting to add them to my blog list on Comic Vine. Sort of showing what's going on in one of the most popular mangas going on in Japan right now.
ONE PIECE Ch. 673 - Vergo and Joker
As Caesar Clown goes to retrieve the children ha had been experimenting on. An incredibly truth is revealed about the mysterious Vergo. Luffy and his gang may have been locked in a cage with Smoker and Tashigi, but the lighthearted hero sure doesn't seem to let that get him down.
The Good
If it wasn't a big enough shock to find out that mysterious Vergo introduced two chapters ago was actually a Vice Admiral for the Marines. He's actually a pirate who infiltrated their ranks as a spy 15 years ago. All as part of the will of his captain, Donquixote Doflamingo. Color me intrigued.
I should probably slap myself for not realizing who this "Joker" actually is. The name has been tossed around quite a bit recently. It was a nice feint in story telling. Code names aren't something common in ONE PIECE. Joker is in actuality Donquixote Doflamingo, a member of the Shichibukai. Now that I look at how Doflamingo always has that creepy grin and controls his own men and government officials through fear of his powers. I wouldn't be surprised if he was partially based around the Clown Prince of Crime. Oda's based characters on Michael Jackson and ROCK HORROR cast. I wouldn't put it past him.
I was sensing some deja vu with the whole scene of Luffy locked in a cage with Smoker. It was a funny that it's Robin who points it out. Seeing how last time something similar to this happened, Robin was on the outside looking in. It's a nice touch of Oda to add seeing how anyone who's followed this series would make the same comparison. It shows the kind of protagonist Luffy is, as well. All this deep drama going on around him. A big new villain revealed, trapped with enemies, wrapped in chains that takes away his powers, and no apparent path of escape. And still he's got a big grin and he's laughing at the irony of his situation. Add that to the list why ONE PIECE is always a fun read.
The Bad
The early pages do serve a purpose, but feels a bit off place. It just seems to hammer home that to most of the world Vergo is a trustful and loyal man to the Marines, but is actually the head of a major conspiracy. It's just the introduction of such a random character as Yarisugi in the middle of it was overkill - pun intended**
It's also a little frustrating that we still aren't even clear on why Luffy suddenly passed out while fighting Caesar a few chapters back. The conventional wisdom is that the gas got to him, but I'm not so sure.
Verdict 4/5
The plot is thickening to an almost head spinning degree. No one can claim that Oda is slowing down or going in circles. Here I was ready for just another fun adventure on a strange island of fire and ice, but it's turning into a deep and meaningful plot with some incredible twists. One that's using some of my favorite characters in amazing ways. I just home we get a bit more action soon.
Seeing how Trafalgar Law was once part of Doflamingo crew. More and more I'm feeling we are going to get a flashback out of Law in the new future. He's been a compelling character since his introduction. I look forward to learning more about him and why he left Doflamingo to make his own crew.
**See character Yarisugi character page for explanation of that joke.
-Kristoffer Remmell ( FoxxFireArt) is a freelance graphic artist, writer, and over all mystery geek.- Follow for news updates: @ FoxxFireArt


text:
Eskendereya retired, sold
LOUISVILLE, Ky. Injured Wood Memorial winner Eskendereya has been retired due to a leg injury.
Owner Ahmed Zayat said Friday tests showed the 3-year-old colt's had a soft tissue injury in his left front leg. The likely Kentucky Derby favorite was pulled a week before the race with swelling in the same leg.
Zayat also announced he's selling a share of the horse to winemaker Jess Jackson's Stonestreet Stables. The sale price was not disclosed.
Eskendereya (Es-ken-DRAY-ah) stamped himself as the likely favorite in the Kentucky Derby after winning two major prep races by a combined 18 1/4 lengths. He was pulled from the Derby six days before the Run for the Roses due to swelling in the injured leg.
The horse underwent diagnostic testing in Lexington this week. Zayat said the injury is career-ending.
"We had numerous doctors look at him and putting all the things together, it was the right call," he said.
Zayat agreed to sell a portion of Eskendereya as part of a reorganization plan after his racing operation declared bankruptcy. The horse's value was based on his performance in the Wood and the Fountain of Youth, and Zayat the horse sold for more than the valuation price.
"Eskendereya make our (reorganization) plan much more appealing to be confirmed," he said.
Fifth Third Bank alleged in a lawsuit filed in December in U.S. District Court in Lexington that Zayat Stables defaulted on $34 million in loans. Zayat filed for bankruptcy protection on behalf of the stable Feb. 3 and filed a reorganization plan last month.
Zayat called the sale "bittersweet" but said it was important for him to retain a share of the horse. He did not give specifics of the sale, only saying that his ownership was "significant."
"He's a once-in-a-lifetime horse," Zayat said.
Eskendereya is the latest high-profile acquisition by Jackson. He purchased superstar filly Rachel Alexandra shortly after she won the Kentucky Oaks last spring and immediately entered her in the Preakness, where she became the first filly in 83 years to beat the boys.
Rachel Alexandra went on to become the Horse of the Year. She has struggled at age 4, finishing second in each of her two races following a long layoff.
Jackson also owns two-time Horse of the Year Curlin. Zayat said he's not sure where Eskendereya will stand at stud.
"Eskendereya is best of class and his progeny will only add to his legacy and that of American racing," Jackson said in a statement.
Eskendereya certainly appeared to be the best of the 3-year-old crop this spring.
Trainer Todd Pletcher, who won the Derby with Super Saver, called Eskendereya the best 3-year-old he's ever trained and lamented the abrupt end of the colt's career.
"Mr. Zayat and I investigated every possible avenue to get him back into racing form at the level that his fans expect, but after leaving no stone unturned, it became clear that this was the best decision to make for Eskendereya," Pletcher said.
Eskendereya was unbeaten in four starts on dirt and just appeared to be blossoming this spring after ending his 2-year-old campaign with a disappointing ninth-place finish in the Breeders' Cup Juvenile at Santa Anita last November.


text:
School Winners
Cameron Lee
Veterans of Foreign War Post 1040 recently awarded the first-, second- and third-place winners of the VFW Patriots Pen essay contest. From left to right are Brian Seguin, VFW Post 1040 Quartermaster; Martin Spani, VFW post 1040 Commander; Reilly Chapell, second-place winner; John Zambrano, VFW Post 1040 Chaplain; Hollie Hutchison, first-place winner; Frank Martinez, VFW Post 1040 Sr. Vice Commander; and Fran Herr, third-place winner.
From left, Aaron Larson, Emma Bartlett, Cameron Chollar, Rachael Rogers, Audrey Garman and Jack Lund were selected as Students of the Quarter by the Stanwood-Camano Rotary Club.
Veterans of Foreign Wars Post 1040 recently awarded the first-, second- and third-place prizes in the VFW Patriots Pen essay contest.
The first-place award went to Hollie Hutchison, second place to Reilly Chappell and third place to Fran Herr. All three students are in the seventh grade at College Place Middle School. The theme of this year's Patriots Pen essay was "What I would tell my Founding Fathers."
Hollie advances to the next level of competition.
Student nominated for acting awards
Cameron Lee, a Monroe High School freshman, has been nominated for Best Leading Actor in a Musical and Person to Watch in the 2012 Seattle Broadway World awards for the ArtsWest production of "Godspell."
Last year, he was nominated for the 2011 Seattle Broadway World awards as Best Featured Actor in a Play for his first role as a professional, playing Jesse in "Distracted." He also was suggested for a 2011 Gregory Award for the same role.
Cameron fell in love with acting in the fifth grade when he saw the Monroe High School production of "The Sound of Music." He since has been in two Monroe High musicals, as Emilio's son in "South Pacific" and as a chorus member in "Side Show." He starred as Harold Hill in Hidden River Middle School's performance of "The Music Man" last spring.
Cameron broke into professional acting with his role in "Distracted," and added to his resume this year by making commercials for Microsoft and a piano-learning program. He is currently in a professional role playing Peter Cratchit in ACT Theater's production of "A Christmas Carol."
He plans to audition for the 2013 ArtsWest musical in January, and for the Monroe High School spring musical.
Seattle Broadway World awards will be announced at the end of December and are based on votes received through the Seattle World website.
Rotary selects Students of Quarter
Stanwood-Camano Rotary Club has selected its Students of the Quarter.
The students were recognized for their exemplary academic efforts, leadership skills, community service, citizenship, athletics and volunteer service. Award recipients are Aaron Larson, fifth grade, Elger Bay Elementary; Emma Bartlett, fifth grade, Elger Bay Elementary; Cameron Chollar, ninth grade, home-schooled in partnership with Saratoga School; Rachael Rogers, 10th grade, home-schooled in partnership with Saratoga School; Audrey Garman, senior, Stanwood High School; and Jack Lund, senior, Stanwood High School.
Elementary students donate money
The students from Patti Ingalls' first- and second-grade class from Whittier Elementary School raised $135 and donated it to an annual Thanksgiving meal hosted at the Everett Gospel Mission's men's mission.
The students, along with teacher Jocelyn Sievers-Bailey's class, also colored and decorated placemats for the celebration. Students learned about philanthropy through the Imagine Children's Museum guest reader, who read the book "One Hen," and decided to raise money for people who are less fortunate.
Students receive college scholarships
Laurel Graves and Julius Arevalo each recently received the Boeing Explorer Scholarship of $750.
Students received this award based on their participation in the three-day Research Skills Short Course in August at Washington State University.
Graves graduated from Arlington High School in 2010 and plans to graduate from Washington State University in December 2014 with a degree in civil engineering. She is a Regents Scholar and a member of the Honors College. She is an Honors College freshmen seminar facilitator and a member of Tau Beta Pi, an engineering honors society.
Graves is the daughter of Corinn and Paul Graves of Arlington.
Arevalo graduated from Arellano University in Manila, Philippines, in 2005 and plans to graduate from Washington State University in May 2014 with a degree in civil engineering. He is the son of Teresa De Villa of Everett.
To submit items, contact reporter Amy Daybert: 425-339-3491 or firstname.lastname@example.com, or email firstname.lastname@example.com.
Share your comments: Log in using your HeraldNet account or your Facebook, Twitter or Disqus profile. Comments that violate the rules are subject to removal. Please see our terms of use. Please note that you must verify your email address for your comments to appear.
You are logged in using your HeraldNet ID. Click here to update your profile. | Log out.
Our new comment system is not supported in IE 7. Please upgrade your browser here.


text:
Panning Across Omega Centauri
June 10, 2011
This video sequence shows many of the fine details in a new image of the great globular cluster Omega Centauri from the VLT Survey Telescope (VST). The wide field of the telescope and its camera OmegaCAM allow the full scope of the object and its surroundings to be seen, but also imaged in exquisite quality.
Topics: Technology Internet, Globular clusters, Centaurus constellation, Telescopes, VLT Survey Telescope, Omega Centauri, Alpha Centauri, Very Large Telescope, European Southern Observatory


text:
IES TULEVIK 2015 programme at Catthorpe Manor, Catthorpe, LE17 6DF
15-17. May Annual anniversary dinner/ dance
9-15. August Estonian summer camp for children aged 7 to 16.
15. August Rahvapidu – featuring a well known Estonian band.
Info: firstname.lastname@example.com
Bradford Estonian Club Annual General Meeting
Sunday 26th April 2.00 p.m.
Estonian elections have taken place and a new government has been established.
Retrospectively reporting on Estonian Independence Day celebrations across the country.
Articles for the April issue are welcome.
Eestihaal.co.uk - Association of Estonians in Great Britain | Copyright © 2011


text:
Research Output for Norul Hidayah Mamat
Note: The 2012, 2013, 2014, 2015 publications have not been audited.
Jump to: Publications | Grants and Consultancies | HDR completions
Norul Hidayah Mamat may be from another institution or only recently joined Deakin University.
2014 External
Ganesan, Rajalakshmi*, Mamat, Norul Hidayah B. T.*, Mellor, David, Rizzuto, Laura and Kolar, Christina (2014) Procrastination and the 2x2 achievement goal framework in Malaysian undergraduate students, Psychology in the schools, vol. 51, no. 5, pp. 506-516, John Wiley and Sons, Hoboken, N. J. [C1]
ERA journal ID: 6613 – Scopus EID: 2-s2.0-84898544965
DRO entry for this publication
2013 External
Kwan, Lydia Li-Ya*, Mellor, David, Rizzuto, Laura, Kolar, Christina* and Mamat, Norul Hidayah Bt.* (2013) The relationship between perceived interparental conflict and Malaysian young adults' attitudes toward divorce, Journal of divorce & remarriage, vol. 54, no. 7, pp. 596-607, Routledge Taylor & Francis, Abingdon, England [C1]
ERA journal ID: 6025 – Scopus EID: 2-s2.0-84886040882
DRO entry for this publication
2008 External
Mellor, David, Yeow, James*, Mamat, Norul Hidayah* and Hapidzal, Noor Fizlee Mohd* (2008) The relationship between childhood cruelty to animals and psychological adjustment : a Malaysian study, Anthrozoos, vol. 21, no. 4, pp. 363-374, Berg Publishers, Oxford, England [C1]
ERA journal ID: 3056 – Scopus EID: 2-s2.0-57449092961
Citation counts: Scopus - 3, Thomson - 3
DRO entry for this publication
Grants and Consultancies
No grants and/or consultancies found or audited – the 2014 HERDC financial return has not been completed or loaded as yet.
Higher Degree by Research Completions supervised by Norul Hidayah Mamat
No completions found or audited.
- Legend:
- * Not a member of Deakin University at time of output.
DRO to publications collection last synchronised: Tuesday 26th May 2015 10:04pm


text:
A storage system with an array of redundant storage devices places bounds on access request servicing. If a storage device is unable to service the access request within the bounds, the storage system temporarily, individually offlines the specific storage device. Access requests pending for the storage device may be serviced by other storage devices of the redundant array while the storage device is offline. The offline storage device can perform error recovery and be brought back online by the system when error recovery has completed.
1. A method comprising:
dispatching a data access request to a storage device of an array of storage devices;
determining that a time from the dispatching of the data access request plus an expected error recovery time has exceeded a threshold, including identifying an expected error recovery procedure, determining an expected error recovery time for the error recovery procedure, and comparing the time from the dispatching plus the expected error recovery time against the threshold;
in response to determining that the time has exceeded the threshold, temporarily isolating the storage device from other data access requests while allowing data access requests to other storage devices of the array, including logically offlining the storage device to a logical out-of-band state; and
resuming data access to the storage device after a period of time to restore the storage device from isolation.
2. A method according to
3. A method according to
4. A method according to
flushing a data access request from a data access request queue associated with the storage device; and
servicing the flushed data access request with other storage devices of the array.
5. A method according to
servicing a subsequently received data access request for data on the isolated storage device with other storage devices of the array.
6. A method according to
7. A method according to
8. A method according to
performing an out-of-band error recovery on the isolated storage device by performing error recovery during the temporary isolation.
9. A method according to
10. A method according to
11. An article of manufacture comprising a machine accessible storage medium having instructions stored thereon using which an electronic system performs operations including:
issuing an input/output (I/O) transaction request to a drive of an array of redundant drives;
identifying an error recovery procedure to implement if an error occurs;
determining an expected error recovery time for the error recovery procedure;
comparing an amount of time elapsed since the issuing of the I/O transaction request adjusted by the expected error recovery time to a threshold;
if the time elapsed adjusted by the expected error recovery time crosses the threshold, offlining the drive by diverting I/O transactions directed to the drive to other drives of the array for servicing, including logically offlining the drive to a logical out-of-band state; and
determining whether to online the drive based, at least in part, on an error recovery state of the drive.
12. An article of manufacture according to
13. An article of manufacture according to
removing an I/O transaction from an I/O transaction queue associated with the drive; and
servicing the I/O transaction with the other drives of the array.
14. An article of manufacture according to
performing error detection and recovery on the drive.
15. An article of manufacture according to
16. An article of manufacture according to
17. An article of manufacture according to
18. An apparatus comprising:
a memory component having code to define operations including determining that an access operation response time plus an expected error recovery time of a drive of a group of drives in a storage subsystem has exceeded a threshold amount of time, logically isolating the drive from access operations to temporarily place the drive in a logical out-of-band state to prevent access to the drive while maintaining access to the other drives of the group, servicing a request for access to the drive using other drives of the group, and determining whether to restore the drive from logical isolation after a period of time, wherein determining that the access operation response time plus the expected error recovery time of the drive has exceeded the threshold amount of time includes identifying an expected error recovery procedure, determining an expected error recovery time for the error recovery procedure, and comparing operation response time plus the expected error recovery time against the threshold amount of time; and
a processor coupled to the memory component to execute the operations defined in the code.
19. An apparatus according to
20. An apparatus according to
21. An apparatus according to
22. An apparatus according to
23. A storage server comprising:
a processor;
a communication interface coupled to a network to interface with a client over the network;
a storage interface coupled to a mass storage subsystem to access one or more mass storage devices of the subsystem on behalf of the client; and
a memory coupled to the processor having software, which when executed by the processor, causes the storage server to determine that an access operation response time plus an expected error recovery time of a drive of a group of drives in a storage subsystem has exceeded a threshold amount of time, logically isolate the drive from access operations to temporarily place the drive in a logical out-of-band state to prevent access to the drive while maintaining access to the other drives of the group, service a request for access to the drive using other drives of the group, and determine whether to restore the drive from logical isolation after a period of time, wherein determining that the access operation response time plus the expected error recovery time of the drive has exceeded the threshold amount of time includes identifying an expected error recovery procedure, determining an expected error recovery time for the error recovery procedure, and comparing operation response time plus the expected error recovery time against the threshold amount of time.
24. A storage server according to
25. A storage server according to
26. A storage server according to
27. A storage server according to
This application is related to U.S. patent application Ser. No. 11/862,236 of Goel et al., entitled "Selective Disk Offline," and filed concurrently herewith.
Embodiments of the invention relate to storage systems, and more particularly, to providing bounds on I/O service time in a storage system.
A storage server is a special-purpose processing device used to store and retrieve data on behalf of one or more client devices ("clients"), which may access and/or process the data. A storage server can be used, for example, to provide multiple users with access to shared data and/or to backup important data.
A storage server may provide different levels of access to data. For example, a file server is an example of a storage server that provides file-level access to data. A file server operates on behalf of one or more clients to store and manage shared files in a set of mass storage devices, such as magnetic or optical storage based disks or tapes. The mass storage devices may be organized into one or more groupings (physical and/or logical or virtual) of Redundant Array of Inexpensive/Independent Disks (RAID). The data may be organized, managed, and/or accessed as data files. Another example of a storage server may be a device that provides clients with block-level access to stored data, rather than file-level access. The data in such a system may be organized and managed and/or accessed as data blocks, which may include more or less information than a file. Also, a storage server may be able to provide clients with both file-level access and block-level access.
A storage server may have access to multiple mass storage devices, or persistent/non-volatile storage devices, which may be managed based on logical or virtual organization. Data storage across these multiple mass storage devices can be organized into multiple layers of abstraction to provide fault tolerance, as individual disks can (and do) fail. The abstraction layers also allow a logical disk organization, for example, a volume or aggregate, to store larger quantities of data than can fit on a single disk.
For example, a storage server may represent a group of storage devices (e.g., hard disks) as a logical aggregate/grouping of storage devices. The aggregates may be managed to store data in volumes contained within the aggregates. As used herein, volume refers to a logical abstraction of physical storage, combining one or more disks or parts of disks into a single logical storage object. The volumes may in turn be further logically broken down into plexes containing RAID groups. The RAID groups may have multiple disks. While particular terminology is used herein as a reference point to describe particular organizations and/or functions herein, the terminology shall not be construed as limiting, but rather by way of example. Where particular terminology is referred to (e.g., an aggregate, a plex, etc.), these are to be understood as merely examples of data structure abstractions that may be substituted with equivalent or similar data structures that may be referred to by other terms.
Tracking of disks in a logical organization and management of the logical organization may be performed with logical association data. The term "disk" is generally used herein as shorthand to refer to a disk drive, including its actual storage medium or media. A disk or other storage device may have a dedicated area to provide a RAID label and/or other metadata to provide the ability assign and determine which disks are part of which RAID groups, plexes, and aggregates, even as disks are added and failed out of the aggregates. The process of determining the logical data structure to which a disk belongs may be referred to as "RAID assimilation."
A storage device can develop errors (e.g., mechanical instabilities, medium corruption, etc.) that will hinder reading and/or writing to one or more areas/blocks of the storage device. Such errors often result in slow access times, especially if an access request focuses on bad media patches. Many storage devices have error recovery routines that can enable a distressed device to eventually complete the requested I/O (input/output, e.g., read, write) operation successfully. Performing an error recovery routine may include performing a recalibration routine, an automatic retry attempt, etc., and/or a combination of these. A storage device experiencing errors as described above may be referred to herein as a "distressed" drive, or a "spasming" drive. "Spasm" may be used herein to refer to an error resulting in a delay and/or the associated delay that results in slow data access.
Error recovery is traditionally handled "in-band," referring to the process of handling the error in the context of servicing the access request (or I/O) that spawned the error. The delay mentioned refers to the delay in servicing (e.g., processing, responding to, performing actions or operations as a result of) the request. Because all error handling is traditionally performed in- band, delays caused by errors are eventually pushed back to the requesting client. Thus, the client traditionally has to wait for the operation to be completed before the request can be acknowledged as complete or successful, if it can be so acknowledged at all. At worst an error message is passed to the client instead of an acknowledgement of success, or the client request results in an error in the filer or client interface.
Hardware commoditization and the continuously decreasing cost of storage (price per GB) is driving a trend towards using less reliable, higher capacity drives for building storage servers and disk arrays. As deposition densities on drives increases, the effects of bad patches on the media become more apparent. Bad media patches can substantially degrade read/write performance, for example, with increased completion latency, especially if I/O happens to focus on the regions of bad media. Drives use an in-built error recovery scheme (native to the drive) when reading/writing blocks within a bad media patch that may take a relatively long time to complete.
As one example, ATA drives use an in-built recovery that can range in duration from multiple seconds to even minutes. In desktop usage where the inability to read a block can prove fatal to system operation, such latency may be acceptable. However, in server environments, long I/O operation complete times may be unacceptable. In another example, FC drives typically use a deterministic error recovery, which can prevent the long I/O delays discussed above. However, successive media errors can still result in cumulative queuing delays that can result in application timeouts and/or system hangs if I/Os cannot be completed within an acceptable time-period. Traditionally, storage systems have either resorted to failing such drives, which results in a costly drive reconstruction, or simply operated with exposure to the risk of application downtimes.
The duration of recovery routines to prevent failure of I/O operations on a spasming drive can range from multiple seconds to minutes, which may be acceptable for certain environments. However, in certain server environments, many data access protocols (e.g., CIFS (common Internet file services/systems), FC (Fibre Channel), NFS (network file services/system)) rely on lease timeouts for maintaining session state, meaning long delays in I/O access time can lead to unwanted connection terminations. Additionally, in file server/filer system configurations, many I/O accesses (e.g., a stripe write) are associated with parallel accesses to other storage devices in a redundant array (protection unit). Traditional systems delay returning the parallel results until all parallel I/O accesses are completed by the individual storage devices of the protection unit. Thus, a single spasming drive can cause a delay for data access from multiple storage devices.
File server/filer configurations typically use redundant drive arrays with mirroring and/or redundancy (e.g., RAID), enabling the systems to regenerate the data on one drive by accessing a subset of drives within a protection unit to which an unresponsive drive belongs. The latency for reconstructing data in this manner should be of the same order as a single drive latency, which is on the order of milliseconds. Assuming that the probability of multiple drive spasming at the same time is very low, reconstructing blocks to service an I/O request should be much faster than accessing a spasmic drive. Thus, successful access that causes a long delay may cause a more deleterious effect on system performance than would be caused by outright failure of the device. Additionally, a subsequent I/O request to the distressed device may trigger the same fault or a related fault, resulting in further client requests being delayed. However, the cost of failing out a device that does not have fatal errors may not be justified.
Besides individual disk errors and resulting access delays experienced by clients attempting to access data stored on a spasming disk, individual disk errors can compound and result in performance problems with the filer. For example, some error recovery attempts in high workload scenarios can result in a cumulative backoff causing a filer operating system to freeze up.
A storage server having an array of storage devices dispatches a client data access request to a storage device of the array. The time of dispatch of the data access request can be monitored and bounded to prevent the data access request taking longer than a threshold amount of time before the storage server intervenes. The storage server can offline the storage device from receiving/processing data access requests while the storage device is offline. After a period of time, the storage server can determine whether to resume data access to the storage device.
The following description includes discussion of various figures having illustrations given by way of example of implementations of embodiments of the invention. The drawings should be understood by way of example, and not by way of limitation.
As used herein, references to an "embodiment" may be understood as describing a particular feature, structure, or characteristic included in at least one embodiment of the invention. Thus, the appearance of phrases such as "in one embodiment," or "in alternate an embodiment" may describe various embodiments and implementations of the invention, and may not necessarily all refer to the same embodiment. Descriptions of certain details and implementations follow, including a description of the figures, which may depict some or all of the embodiments described below, as well as discussing other potential embodiments or implementations of the inventive concepts presented herein.
A redundant array of mass storage devices (e.g., RAID (redundant array of inexpensive/independent disks)) storage subsystem, or any other configuration where data on a given mass storage device can be recovered by using a subset of other mass storage devices within a protection unit, may be enabled to temporarily cease access (e.g., input/output (I/O) functions/operations/transactions) to a particular mass storage device of an array of mass storage devices. I/O functions/operations/transactions refers herein to access to a mass storage device, for example, a disk read request (or simply "read") or disk write request (or simply "write"). As used herein, mass storage devices refer to any type or form of non-volatile storage device/means suitable for storing large quantities of data. Mass storage devices include, but are not limited to, conventional magnetic disks (e.g., hard disks), optical disks, for example CD-ROM (compact disk—read only memory) or DVD (digital video/versatile disc) based storage, flash storage, etc. Mass storage devices may be referred to as disks, drives, storage units, hard disk, etc., but will be collectively, generically referred to herein as "disks" merely for purposes of convenience in description. Reference herein to disks should be construed only as illustrative, and not as limiting. The organization of disks into protection units refers to RAID groups, or any similar/comparable logical grouping of disks. Many types of RAID arrays or other redundant disk arrays are known, many based on standards, and others proprietary. As used herein, "RAID group" refers generically to any type of RAID array, redundant disk array. Reference to a "logical grouping" of disks is an abstraction that refers to any of multiple layers of virtual organization of physical resource, for example, a RAID group, and/or groupings of RAID groups. Ceasing I/O to a disk may be referred to as "offlining" the disk for a period of time. The following description includes reference to selective offlining, and certain overviews and/or details may be briefly covered. Additional description of selective disk offlining can be found in U.S. patent application No. TBD, entitled "Selective Disk Offlining," of Goel et al., filed TBD.
Because an offline drive is inaccessible for I/O operations, during the offline period reads to the disk may be serviced by reconstructing data from other disks within the redundant array group. Servicing a read (or a write or other I/O request/operation) refers to processing, responding to, performing actions or operations as a result of, etc. the read. Writes can be logged to a "dirty region"log, which refers to a storage region/area to which updates to data on the disk may be placed while the disk is offline. In one embodiment a filesystem employed with this technique allows for writing out of place, or writing anywhere. Writing out of place is in contrast with a write-in-place approach where a data block, when modified, is written to the same physical location on disk. With write out of place, any time a data block is modified it is written to a new physical location on disk. A filesystem that has a write anywhere property can write to any unallocated block on any available disk in a group, and does not have to write to a particular location for a particular "file" or block of data. A write out of place filesystem stores data writes in unallocated blocks instead of writing over already allocated data blocks. Thus, data is not overwritten, and a particular disk/block is not used for any subsequent write that changes data stored at the particular disk/block.
To provide selective offlining, a trigger mechanism indicates when a disk is inaccessible and/or should be offlined. A software module on a storage access path of a filer/file server may receive certain commands and/or interpret/respond to certain conditions that indicate an offlining circumstance. A command may be issued by an administrator, a hardware component, another software module, etc. A condition may be indicated by a hardware component or software that monitors hardware conditions. An example of a condition that may trigger selective offlining is a spasmic disk condition, which may result in deleterious delays in traditional systems that have unbounded I/O access. Spasmic, or a spasmic disk condition, as used herein, refers to a state or condition of error resulting in a delay, and may also refer directly to the delay associated with the error or other condition that results in slow data access. The error may be caused by any number of events/conditions causing a disk to be or become unresponsive, meaning that the disk does not respond to and/or is not timely in completing an I/O transaction or does not complete an I/O in a timely within a period of time. For purposes of example, and not by way of limitation, disk spasm conditions may include disk or shelf firmware freezes/hangs, disk media errors, service access processing errors/exceptions, etc.
A storage system with a selective offline mechanism may further include a mechanism for bounding service times for I/O directed to a disk of a RAID array subsystem. In general bounding service times for I/O includes a triggering mechanism, an I/O handling procedure, and potentially a subsequent recovery procedure for errors (e.g., an out-of-band recovery mechanism). The trigger mechanism may determine whether a disk is experiencing an error, for example, a spasmic error. The trigger could be related to a period of time, for example, comparing a period of time an I/O has been pending against a threshold (e.g., a time limit, a core value, a period of time, an expected duration, an allocated time for response, etc.). The period of time an I/O has been pending may be considered to be a response time of the disk. In one embodiment the trigger accounts for an expected recovery time for the disk. The I/O handling procedure includes processing of I/O for servicing when the spasming disk is unable to complete the operation within the bounds established. The recovery procedure includes error processing of the disk to attempt to correct the error. By bounding I/O service times, a system can ensure service guarantees for access times from disks in the system. Additionally, an out-of-band error recovery can be implemented to allow error recovery without failing the disk out of the system.
In one embodiment one or more software-based mechanisms for bounding disk I/O service times are provided to implement and/or enforce one or more timeout thresholds for disk I/O service time. A timing mechanism determines a timeout threshold, which may be related to a threshold used by an application protocol, etc. The timing mechanism, or another timing mechanism, monitors I/O delay to determine if the I/O has exceeded the threshold. After a timeout threshold has been reached, a disk may be selectively offlined out of the storage system for error recovery (to be reinducted back into the system when error correction completes). Offlining the disk includes isolating the disk from I/O routines for a period of time, and may be referred to as placing the disk out-of-band. Isolating the disk refers to temporarily ceasing I/O to the disk, or not allowing access to the disk for storage service (e.g., reads, writes) during the period of time. The period of time refers to a length of time that may be predefined, or may be as long as necessary for a system to resolve an error condition and/or determine to fail a disk. Recovery of a disk error while the disk is offline may be similarly be referred to as out-of-band recovery.
In general, a software mechanism for error recovery in a driver for a disk can be considered to have two error recovery algorithms: a default process, and a spasmic error process. The two error recovery algorithms coexist in a system. The individual disks of the RAID array subsystem typically include error processing algorithms, which may be referred to as a standard, or built-in error recovery/error processing mechanism. In one embodiment the default error recovery algorithm is identical to the standard error processing mechanism for a disk. The default/standard mechanism relies on a sequence of retries, resets, power cycle phases for servicing an I/O error, and/or waiting for the disk's standard error recovery to attempt to correct an error. The default algorithm may operate on a "best-efforts" basis, meaning all attempts to recover will be made before failing a disk.
Spasmic error processing, instead, will force the driver (i.e., software managing/controlling a disk) to abort an I/O that cannot be completed within a defined threshold time period. The disk may be offlined to perform error recovery to attempt to prevent similar errors from recurring. The storage system employs a multi-stage error processing algorithm to attempt recovery by a disk's standard error recovery routines, as well as providing an additional error recovery layer to place bounds on the implementation of the disk's standard error recovery. The disk can be allowed to attempt recovery in-band, meaning in the context of the client I/O request made to the disk, which implies that the I/O does not necessarily complete until the recovery is in progress. After a period of time, if the error has not been corrected, the in-band error recovery is converted to out-of-band error recovery, by selectively and temporarily removing the spasming disk from the storage system. The entire error recovery process (in-band and out-of-band) proceeds transparently with respect to end clients, and may increase the performance and availability of storage servers by ensuring that the latency expectations of a disk are bounded.
Within a filer/file server, a storage layer communicates spasmic conditions to a RAID controller. A disk offline module within RAID, or accessible to RAID, ceases I/O to the disk, and offlines the disk until it recovers from its spasmic state. In general, the disk offline module removes the disk from a RAID group and/or other logical data structure. A logical data structure that is missing a disk may be considered "degraded," which may temporarily change how the logical data structure is handled. During the offline/degraded period, reads to the disk are serviced by accessing other disks within the redundant group, or reads to a data mirror or backup, and writes are logged using a dirty bitmap, and/or written to a dirty region. After the disk recovers from its spasmic state, RAID onlines the disk and synchronizes the dirty region either via copying from the dirty region log by reconstructing ranges marked dirty with a dirty bitmap.
The logic for indicating I/O for which a spasmic error recovery algorithm is used is orthogonal to the mechanism used for tracking logical block address (LBA) ranges that might potentially be a cause of a spasming condition. RAID uses a flag to indicate to a disk driver those I/Os for which the driver should use the spasmic error recovery algorithm. In one embodiment this flag is set only for client I/Os that are latency sensitive and not for internal RAID generated I/Os (e.g., periodic scrubs). In another embodiment this flag is set for all I/Os. The flag can be appended to individual I/Os to provide a differentiation between the standard and spasmic error recovery algorithms that should be executed for different I/Os. In contrast, a spasmic LBA range is an LBA range that was being processed by the disk when a spasmic condition was detected (e.g., the I/O took longer than the threshold time allowed). The spasmic condition detected could be any command pending on the disk, and is not necessarily a request with a spasmic flag set. Irrespective of command type, as soon as a spasmic condition is detected, I/Os with spasmic flags set should be aborted, forcing the disk to be offlined for spasmic error recovery. For other I/Os already in progress, default error recovery processing can be used. Since the disk is offlined, subsequent I/Os will be temporarily prevented from reaching the disk. Due to command queuing, which permits multiple commands to be dispatched to a disk, it is possible for the filer to not know exactly which command was being processed by the disk, and therefore conclude that all pending commands were the cause of the spasm.
The flag/message can provide a driver the ability to aggressively bound I/O service times for a disk. For example, the driver may track bad media patches, and force an associated disk offline for all I/Os directed to the bad patches. Bounding the I/O service times may also include determining whether a default error recovery is expected to involve more or less cost (e.g., time, processing) than would be involved in forcing the disk offline. For I/Os expected to take shorter than offlining, the default procedure may be used, while a disk is offlined when the default recovery procedure is expected to take longer than offlining. A time threshold may be set that would be based on system latency tolerance, and/or disk type. In one embodiment a formula is used to determine whether to trigger a spasmic condition processing, including offlining the disk. For example, the formula (time since I/O was dispatched)+(expected recovery time)>(I/O threshold) provides one implementation of determining whether to trigger spasmic processing. The value (time since I/O was dispatched) refers to a period of time since the filer transmitted the I/O operation to the storage subsystem hardware for processing. The value (expected recovery time) refers to an optional value that could be used in the formula. In one embodiment a state machine can track the state of an individual disk, and knows a general error recovery procedure that will be used by the disk (e.g., when the disk will be triggered to hard reset, or power cycle, etc.). With knowledge of the error recovery procedure, and the current state, the filer can determine whether to wait for the standard procedure to continue, or whether to aggressively trigger spasmic processing (i.e., prior to the point at which the time since the I/O was dispatched is greater than the threshold). The duration of time the error recovery procedure will take or is expected to take may be referred to as an error recovery time. The error recovery time is the length of time from the state the disk is currently in until an error recovery event (e.g., a reset, a powercycle) should occur. The value (I/O threshold) refers to a value determined for the particular disk, and may be different for each disk and/or subsystem protocol used. In one embodiment the value can be computed based on the timeout thresholds used by an application protocol (e.g., NFS, CIFS, FCP). The value can also be computed based on heuristics for determining the maximum amount of time an I/O can take in order to prevent a freeze-up of a checkpointing file system.
One common storage system configuration includes the use of shelves of disks, or some similar physical grouping hardware/software combination. The physical grouping is controlled by a physical grouping controller that has a processor/control logic, and possibly software/firmware to provide the control functions. Triggering a spasmic condition could be limited to commands already dispatched to the shelf controller or equivalent. In one embodiment commands still queued at the storage layer of the filer and not yet dispatched to the physical grouping controller are not valid candidates for flagging. Limiting the spasmic condition triggering in this way can ensure correct identification of spasms due to misbehaving disks, as opposed to long I/O times due to other causes, for example, deep queues. In one embodiment spasming conditions are triggered for any command, either pending or dispatched to the disk/shelf controller, as long as the I/O threshold is exceeded. Additionally, because offlining a disk results in a mode of temporarily degraded operation, triggering offlining might not be necessary for I/O that are not sensitive to high latency, for example, internal I/Os, scrubs, etc. RAID can use the timeout flag mentioned above to specify I/Os that might be aborted if a spasmic condition is detected.
One cause of spasmic conditions may be bad media patches (e.g., mechanical errors, weak signal stored on the medium, etc.). In some other instances, long I/O times are the result of poorly performing disk firmware. In either case, the command being processed by the disk at the time the spasmic condition is detected is identified as the cause. In certain conditions, it may become difficult to identify the spasmic command, for example, in ATA disks that support command level queuing. In such cases, block ranges corresponding to all commands dispatched to the disk may be treated as bad media patches, resulting in the disk being offlined to an out-of-band recovery condition. Block range(s) suspected to be the source of the spasmic condition can be fixed during the recovery phase.
Spasmic error processing may operate to place a disk in an out-of-band state. A default error processing may begin prior to the disk being offlined (placed out-of-band), and proceed to completion out-of-band. Alternatively, the error processing can be started and completed out-of-band. Placing the disk out-of-band for error processing can simultaneously allow error recovery to proceed to lower the risk of failure (and avoid removing the disk from the system for a recoverable error), while still providing bounded access times. Placing the disk out-of-band to accomplish error recovery may allow the use of more extended diagnostics than would be available to an in-band disk (e.g., for purposes of delay and/or disk accessibility). Use of extended diagnostics may avoid resorting to full reconstructions for spasmic errors, which can increase system reliability by reducing exposure to data loss via non-recoverable media errors and reducing disk failure rates.
In one embodiment, a disk that is unable to respond to an I/O may be expected to recover if a hard reset or a power cycle were performed. If a hard reset or a power cycle is part of the default error recovery, RAID can force such a condition by issuing a dummy I/O. The dummy I/O is an I/O (e.g., a write, or potentially a read) request to the block range corresponding to a bad media patch. The dummy I/O will not have the spasmic flag set to indicate that a full error recovery cycle must be executed. The dummy I/O mechanism can provide a form of out-of-band error recovery by using the default error recovery processing to bring the disk back to service. Issuing the dummy I/O will invoke the error recovery processing, which can result in a hard reset and/or powercycle phase as part of the default recovery process. The hard reset could help the disk recover from a spasmic state. In one embodiment a block range suspected to be the source of the spasmic condition may be fixed during the optimized reconstruction stage by marking it dirty when offlining the disk. The block range can also be fixed during the recovery phase.
Alternatively, a successful reply to a dummy I/O request may indicate recovery of the disk, which may trigger a recovery notification be sent to RAID. RAID uses the notification to online the disk. A synchronization phase of bringing the disk online might either be done by copying data from the dirty range log or by reconstructing block ranges identified in the dirty bitmap. As soon as the disk is onlined (in one embodiment, even before a synchronization phase can be completed), spasmic error processing can be enabled for I/Os to that disk.
In one embodiment RAID keeps track of the spasmic history for each disk. If a disk enters a spasmic state a threshold number of times, RAID marks the disk as failed or potentially failed. The failed disk would typically be replaced with a good spare.
Filer 120 may be coupled locally to storage subsystem 140, which includes one or more mass storage devices 142-144. Mass storage devices 142-144 in storage subsystem 140 may be, for example, conventional magnetic disks (e.g., hard disks), optical disks such as CD-ROM or DVD based storage, magneto-optical (MO) storage, and/or any other type of non-volatile storage devices suitable for storing large quantities of data. Filer 120 includes one or more connectors and/or connection ports with which to couple to storage subsystem 140 and/or one or more of its sub-elements. The connectors and/or ports may be connection points for fiber channel, coaxial cable, twisted-pair communication lines, and/or secure wireless connections. Filer 120 manages storage subsystem 140, including receiving and processing read and/or write requests to data stored in storage subsystem 140, maintaining data structures, organizing the physical resources of storage subsystem 140 into logical storage resources, etc. The data structure and logical storage resources are constructs for storage management to allow more storage in a system, provide reliability in storage access, etc.
Filer 120 includes access time bounding agent 130, which represents one or more hardware and/or software components/modules to provide I/O service time bounding functions to filer 120. Access, or I/O service, time bounding refers to providing bounds or limits on how long an access can take, for example, by limiting how much time a certain access operation will be allowed to proceed without intervention. Accessing bounding agent 130 enables filer 120 to establish and enforce time constraints on access to individual storage devices of storage subsystem 140. For example, if mass storage device 142 experienced an error that resulted in a long access time (e.g., due to error recovery processing), access time bounding agent 130 could service the access request with alternative mechanism(s) and temporarily prevent access to mass storage device 142.
In one embodiment mass storage device 142 is shown having group information 146. Group information 146 may include one or more values, data structures, pointers, etc., to indicate a logical association of mass storage device 142 with other mass storage devices. Thus, group information 146 may represent metadata stored on mass storage device 142 to indicate an association/role/place of mass storage device 142 in the logical organization of the physical resources. Group information 146 may indicate a data structure to which mass storage device 142 belongs.
Filer 120 is coupled to one or more clients 102-104 through communication network 110. Communication network 110 may be, for example, a local area network (LAN). Each client 102 and 104 may be a conventional desktop computer, a laptop, workstation, a network server, etc. Clients 102 and 104 generate requests for access to data on storage subsystem 140. Filer 120 services the requested access functions to clients 102 and 104, depending on what request is made, what permissions a client has, etc.
Filer 120 may have a distributed architecture. Thus, in one embodiment filer 120 may include a separate N- ("network") blade and a D- (disk) blade (neither shown in
Memory 240 represents the main memory of filer 200 to provide temporary storage for code to be executed by processor 220. Memory 240 may include read-only memory (ROM), flash memory, one or more varieties of random access memory (RAM), or the like, or a combination of such devices or technologies. Memory 240 stores, among other things, operating system (OS) 250 that provides one or more functions of filer 200. OS 250 may include one or more functions for selective disk offlining, out-of-band recovery, and disk access time bounding, as described herein.
In one embodiment filer 200 includes access time bounding agent 230, which provides one example of an access time bounding agent according to access time bounding agent 130 of
Processor 220 and memory 240 are coupled to bus system 210. Bus system 210 is an abstraction that represents any one or more separate physical buses, communication lines/interfaces, and/or point-to-point connections, connected by appropriate bridges, adapters, and/or controllers. Therefore, bus system 210 may include, for example, one or more of a system bus, a Peripheral Component Interconnect (PCI) bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (commonly referred to as "Firewire").
Also coupled to processor 220 through bus system 210 are one or more network interface(s) 260, one or more storage interface(s)/adapter(s) 270, one or more input/output (I/O) interface(s) 280, and one or more internal mass storage device(s) 290. Network interface 260 provides filer 200 with the ability to communicate with remote devices (e.g., clients) over a network and may be, for example, an Ethernet adapter. Storage interface 270 enables filer 200 to access a storage subsystem and may be, for example, a Fibre Channel adapter, a SCSI adapter, etc. Filer 200 may typically be accessible to electronic or computer "users" rather than human users. Thus, I/O interface 280 may include one or more interface components to connect filer 200 with other electronic equipment, for example, custom connections, blade adapters, etc. Additionally, filer 200 could be made to be accessible to a human user, and thus have video, audio, and/or alphanumeric interface through I/O interface 280. Internal mass storage 290 may be or include any conventional medium for storing large volumes of data in a non-volatile manner, such as one or more magnetic or optical based disks. Mass storage 290 may hold data/instructions in a persistent state (i.e., the value is retained despite interruption of power to filer 200). In a generic sense mass storage 290 is a "memory," although memory 240 is the executing or operating memory to provide instructions to processor 220. In one embodiment mass storage 290 includes code and/or data 292 that may be accessed and placed in memory 240 for execution by processor 220. Code/data 292 may include machine instructions to provide disk access time bounding functions.
Client interface path 330 is depicted with protocol layer 332 and network access layer 334 associated with protocol layer 332, to allow file system 320 access over a network with clients. Client interface path 330 may be understood to include more components than what are depicted in
Storage interface path 350 includes storage access layer 352 and storage driver layer 354 associated with storage access layer 352, to enable file system 320 to communicate with one or more disks and/or other storage resources in a storage subsystem. Storage interface path 350 is also to be understood as merely representative, and may include more components and/or layers than what are depicted. Storage access layer 352 implements a disk storage protocol, such as RAID, and storage driver layer 354 implements a lower-level disk access protocol, such as Fibre Channel Protocol (FCP), SCSI, advanced technology attachment (ATA), serial ATA (SATA), serial attached storage (SAS), etc., which may implement commands/requests made by the higher level protocol. To facilitate the description of the following details herein, it will be assumed that storage access layer 352 implements RAID, such as RAID-4, RAID-DP (created by Network Appliance Inc.), etc., and therefore may alternatively be referred to as RAID layer 352.
In one embodiment storage interface path 350 includes access time bounding agent 340, which is an example implementation of an access time bounding agent as previously discussed. Access time bounding agent 340 is depicted as potentially intersecting both storage access layer 352 and storage driver 354, which represents that functions of access time bounding agent 340 may interact with and/or implicate functions at both layers. Storage access layer 352 detects disks accessible through storage driver 354 and organizes the disks into logical data structures. These data structures may include RAID groups, plexes, aggregates, volumes, and/or any similar or equivalent data container abstraction. Access time bounding agent 340 provides storage access layer 352 the ability to detect disks that will potentially result in access delays, and in response, selectively affect the organization of the logical data structures. For example, a disk may experience a spasmic condition, and access time bounding agent 340 determines the condition will result in access delays, and offline the disk to convert an in-band error recovery to an out-of-band error recovery. Access time bounding agent 340 provides a mechanism between filesystem 320 and the disk to temporarily interrupt I/O to the disk to isolate it from the system until the error recovery brings the disk back online. Access time bounding agent 340 may provide one or more timing mechanisms for use at the driver level (storage driver 354) to establish and monitor access time thresholds after which a disk should be temporarily offlined.
In one embodiment filer 400 includes memory 420, which may have one or more elements of volatile and/or non-volatile storage. For example, memory 420 may include RAM. Memory 420 may refer to a physical resource of filer 400, or it may refer to a persistent or temporary storage resource accessible to filer 400 that is not necessarily part of filer 400. In one embodiment memory 420 is a memory allocation of the main memory of a system to which filer 400 belongs. Memory 420 includes information referencing the storage on disks 452-472, and may store state information that is used to determine to perform spasmic error processing on one or more of disks 452-472. A data structure representing a virtual layout (e.g., the logical group shown) of the physical disks 452-472 could exist in memory 420. The data structure can be dynamically modified as the system is operational, to cease I/O to/from an individual disk. Selectively ceasing I/O to a disk provides an environment in which bounds can be placed on disk access times.
Storage access 430 may include hardware (e.g., ports, interfaces) and/or software (e.g., a RAID layer) to see and organize the storage in a virtual representation. Storage access 430 includes RAID 440, which sees/detects the physical resources (e.g., disks) of the system and provides logical data structures. For purposes of description, and not by way of limitation,
Within RAID 440, or logically below RAID 440 may be one or more storage interface components 442. RAID 440 may include one or more components, not shown, to access and manage the storage through storage interface 442, such as an assimilation module to provide physical to logical mapping of the physical resources to higher layers 410 of filer 400. Thus, filer 400 may "see" the physical resources as represented in the logical group, with the disks allocated by RAID 440 for storage. The particular allocation of the physical resources may not be significant to filer 400, which will simply provide storage services to clients based on what resources are seen as available for the clients.
The storage of the system of
RAID 440 also includes recovery module 444 within storage interface 442. Recovery module 444 represents one or more components and/or routines to provide error recovery when a disk error is encountered. In one embodiment recovery module 444 represents functions that actually reside on the disks themselves, in addition to features of a device driver for the particular disk. Recovery module 444 may further include timing mechanisms, as described above, with respect to spasmic disk processing to provide access bounds. The timing mechanism of recovery module 444 is optional, and if included, could work in conjunction with one or more timing mechanisms of disk access time bounding module 448, discussed below.
RAID 440 also includes disk offline module 446, which enables RAID 440 to selectively, temporarily disable I/O to a particular disk. RAID 440 includes disk access time bounding module 448, which provides particular functions to filer 400 for triggering and handling slow disk access. Disk access time bounding module 448 works in conjunction with disk offline module 446 and recovery module 444 to bound disk access times. In one embodiment disk access time bounding module 448 provides an additional error recovery layer to the error recovery provided by recovery module 444. Disk access time bounding module 448 may provide timing information and/or timing bounds to recovery module 444, and trigger disk offline module 446 to logically remove a disk temporarily from the logical group in the storage.
Consider an access request generated by higher layers 410 in connection with a client request for data stored on disk 464 (and potentially other disks). If disk 464 were to experience an error in processing the request (e.g., a mechanical error, bad media, an interface component error, etc.), the successful completion of the service requested could be delayed. Traditional systems would wait for disk 464 to complete the request, or fail the disk out of service, and reconstruct disk 464 to a spare (not shown) with data available from disks 462 and 456, and with parity disk 454. Instead, RAID 440 includes disk access time bounding module 448 to allow error processing to proceed when an error is experienced, but to set limits on how long error processing can proceed before other procedures are introduced to ensure bounds on the servicing of the access request. Disk access time bounding module 448 instructs/indicates to disk offline module 446 to selectively remove disk 464 from the logical group, and service the request and subsequent requests with the remaining disks in the logical group. Error recovery can proceed in an out-of-band manner with recovery module 444 while disk 464 is offline, and disk 464 can be assimilated back into the system when the error recovery is complete.
In one embodiment the combination of disk offline module 446, recovery module 444, and disk access time bounding module 448 represents an access time bounding agent, as discussed herein, for example, 130, 230, and/or 340 of
Agent 500 also includes bounding engine 550, which represents one or more functions that enable agent 500 to individually, selectively offline disks and manage issues related to disk reliability and timing of disk access. The functions include one or more of timing bounds feature 552, spasm processing feature 554, disk offline feature 556, and I/O service feature 558. Agent 500 also includes recovery engine 560, which represents one or more functions that enable agent 500 to monitor and manage error recovery of a disk. Recovery engine 560 implies an out-of-band recovery component, which enables agent 500 to have a disk recovery from error outside the context of servicing an I/O request. The functions associated with recovery engine 560 may include cause identification feature 562, recovery feature 564, and disk online feature 566. Other features may be included, making other versions of bounding engine 550 and/or recovery engine 560 that are more or less complex than what is shown in
Timing bounds feature 552 enables bounding engine 550 to establish timing bounds on I/O access for a particular disk. The timing bounds may be different for different disks in a system, based on the type of disk and/or a connection protocol to the disk. In addition the timing bound might even be determined based on the application protocol (e.g., NFS, CIFS, FCP) timeouts. Alternatively all timing bounds can be set to be the same for a system. Timing bounds feature 552 also provides the ability to track the timing associated with an I/O operation, for example, monitoring how long the I/O operation has been pending, comparing the pending time with a threshold (time bound), determining how long an I/O operation is expected to take to complete (e.g., obtaining information from a state machine), etc.
Spasm processing feature 554 enables bounding engine 550 to have additional or secondary processing to handle disk errors, in addition to standard error processing available to a disk and/or its driver software. Spasm processing feature 554 does not necessarily supplant the standard error processing, and enables bounding engine 550 to ensure that I/O will be serviced within the bounds set by timing bounds feature 552. Whereas standard error processing will typically focus on implementing a best-efforts recovery schemed, without regard for timing, before assuming that the disk is failed and/or inaccessible, spasm processing feature 554 determines when the error processing takes longer than a target time for the system.
Disk offline feature 556 enables bounding engine 550 to selectively offline a disk. The ability to selectively offline a disk provides a mechanism to implement spasm processing feature 554. Disk offline feature 556 allows bounding engine 550 to temporarily remove a disk from a RAID group and/or other logical data structure, and place the disk in a state inaccessible to I/O. Being inaccessible to I/O, the disk can proceed to perform error recovery without slowing down the storage system, but without being failed out.
I/O service feature 558 enables bounding engine 550 to handle subsequent requests for I/O services/operations from an offlined disk. When disk offline feature 556 places a disk offline and inaccessible to I/O, I/O service feature 558 provides a mechanism to service subsequent I/O requests without full disk reconstruction, which allow the offline disk to remain part of the storage subsystem. In general, reads to the offline disk will be reconstructed from the redundant disks, and writes will be logged to another area of the storage system, and carefully tracked. Alternatively, reads and writes may occur based on mirrored or backed-up data. Synchronization of the writes can be completed as part of, in preparation for, or in conjunction with induction of the disk back into the subsystem.
Cause identification feature 562 enables recovery engine 560 to determine the cause of a disk error. Many routines for performing error cause identification are known, and will not be discussed in detail herein. Generally, cause identification feature 562 may analyze a disk to determine a state of the media, access a driver to check for an error message/report, run a cause identification algorithm, etc.
Recovery feature 564 enables recovery engine 560 to perform operations, for example, delays, resets, power cycles, retries, scrubbing, etc., to localize and recover from an error. The error may be a transient error that can be overcome by simply waiting and/or retrying. Standard error recovery algorithms are common on disks. Recovery feature 564 in one embodiment refers to the standard features associated with a disk. Alternatively, recovery feature 564 may refer to a specialized error recovery routine written for a particular system/application. In one embodiment recovery feature 564 occurs by default in a system, meaning that when an error occurs, the mechanisms may be in place to detect and begin execution of the recovery routines. In another embodiment a more elaborate or exotic recovery algorithm can be attempted that would decrease the disk failure rates and also improve the performance characteristics of the disk.
Disk online feature 566 enables recovery engine 560 to return a disk to the storage system/subsystem after a disk has recovered from an error. In one embodiment this simply refers to an ability to restore a disk to be accessible to I/O. For example, a flag may be set to prevent the disk from processing I/O requests, and the flag be unset to online the disk. A system with the ability to selectively offline a disk may also have an automatic assimilation feature to induct the disk back into the system to actively serve data.
Access time bounding agent 500 may include hardware, software, and/or a combination of these. In a case where agent 500 includes software, the software data, instructions, and/or configuration may be provided via an article of manufacture by a machine/electronic device/hardware. An article of manufacture may include a machine accessible/readable medium having content to provide instructions, data, etc. The content may result in an electronic device, for example, a filer, a disk, or a disk controller as described herein, performing various operations or executions described. A machine accessible medium includes any mechanism that provides (i.e., stores and/or transmits) information/content in a form accessible by a machine (e.g., computing device, electronic device, electronic system/subsystem, etc.). For example, a machine accessible medium includes recordable/non-recordable media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.), as well as electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), etc. The machine accessible medium may further include an electronic device having code loaded on a storage that may be executed when the electronic device is in operation. Thus, delivering an electronic device with such code may be understood as providing the article of manufacture with such content described above. Furthermore, storing code on a database or other memory location and offering the code for download over a communication medium via a propagated signal may be understood as providing the article of manufacture with such content described above.
Filer 610 is coupled to shelf controller 620, which represents a component at a shelf level. The shelf level represents a component level where multiple disks share a common interface component/circuit in storage system. Shelf controller 620 includes I/O access queue 622, which receives and buffers the requests dispatched from filer 610. The precise details of the functioning of I/O access queue 622 are not necessarily significant to the operation of the access time bounding described herein, and will not be discussed. In one embodiment the dispatching of an I/O access request to the disk (and its subsequent placement in I/O access queue 622) provides the start of a timeout clock with respect to the I/O access request. Thus, the time of dispatch may be the starting time that will be compared against a threshold or bound time when determining if an access request has not been serviced within an acceptable period of time. In another embodiment it is possible to start measuring the time as soon as the I/O request is received as the filer or at the storage layer. It is not necessary to start the timer clock after the request has been dispatched to the disk. Using the time of dispatch can prevent conditions unrelated to a disk's performance from being the cause of invoking spasmic processing, for example, deep queues on filer 610.
At the disk level, a disk control 630 includes an I/O access interface 632, which represents one or more components for interfacing the disk with the shelf and the filer. Disk control 630 may include request buffer 634 to receive requests that are queued from shelf controller 620 to the disk. Alternatively, disk control 630 may service a single request at a time, and shelf controller 620 only issues a request out of I/O access queue 622 when a previously issued request has been completed. Request buffer 634 may alternatively represent a system in which I/O access queue 622 is an abstraction that shows a management and control function that shelf controller 620 may have over separate buffers in individual disks.
Disk control 630 includes functional control 636 (e.g., lookup tables, track/sector maps, etc.) and mechanical control 638 (e.g., servos, electronics), which represent one or more components to interface with a storage medium and translate requests into mechanical actions and other storage functions. The disk includes platter 640, which includes the physical media on which data is stored. Spindle 642 rotates platter 640 and aligns an area of platter 640 with head 648, which can detect and/or cause differences in the physical media, and thereby read/write to the disk. As platter 640 is rotated by spindle 642, a track or cylinder 644 is passed under head 648. Track 644 represents the various concentric divisions of platter 640 used for storing data. Each track, such as track 644, may be further divided into subsections often referred to as sectors, such as sectors 646. While description is made to a disk drive, other types of non-volatile storage device may be used, and the interface hardware and read/write mechanism may be different than what is depicted in
If an access request dispatched from filer 610 and issued from I/O access queue 622 to the disk were directed to a sector 646 when an error or delay occurs, the sector 646 may be considered to be the cause of the error. Diagnostic and/or other error recovery may be performed to attempt to fulfill the request successfully. Such error recovery is discussed herein, and will not be set forth in detail here.
The timing module and/or a bounding module determine if a spasm condition, or a spasm-inducing error, has occurred, 710. The module determines a spasm condition if there is an error or some other condition resulting in the operation of recovery/diagnostic routines. Because a spasm is typically defined as a condition where an I/O takes a long time, it is possible to transition to such a state even if there is no error. If a spasm condition is not detected, the timer continues to run, and the timing/bounding module waits for either the triggering of a spasm condition, or the completion of the I/O operation requested. In one embodiment waiting may include active polling, and in another embodiment waiting consists of performing other operations and/or remaining dormant until an interrupt, flag, message, etc., indicates a condition or a completion. If the I/O operation is complete, 720, the process ends with respect to the dispatched I/O request.
If a spasm condition is detected, 710, error recovery is initiated for the disk, 712. Error recovery may begin as an automatic process in a disk and/or between the disk and its driver. The error recovery that begins may be referred to as a "default" error processing, which attempts to recover the disk. In one embodiment a disk may have a standard algorithm that it will follow, which may include different phases or states of error recovery. Depending on the phase or state that the disk is in with error recovery, the driver and/or a timing module may be able to determine how long an expected error recovery should last. As discussed previously, if error recovery is expected to take longer than a threshold time, the disk may be forced into offline error recovery, even though the threshold has not yet been reached.
The timing/bounding module determines whether the time since the I/O timer was begun exceeds a threshold, 730. The threshold is set to bound how long an I/O access is allowed before servicing the I/O access with another mechanism, which can provide performance constraints on a storage system. In one embodiment the time includes not only the time since the I/O timer is started, but also the expected time of recovery from the current recovery phase/state, as discussed above. Thus, even if the time from initiating the I/O timer is less than the threshold, spasmic processing may be invoked if that time plus the time to recovery is greater than the threshold. If the time has not exceeded the threshold, the spasmic triggering mechanism waits for recovery to complete. If the recovery is complete, 740, spasmic processing may be avoided, and the process finished.
If the time is greater than the threshold, 730, an offline module offlines the disk to perform out-of-band error recovery, 732, where I/O to the disk is redirected from the disk to be serviced by reconstruction/dirty region logging, as previously discussed. Thus, pending I/O directed to the offline disk is flushed from an I/O access request queue, 734. Flushing a pending I/O refers to canceling or removing the I/O from a queue or buffer, and may include queuing or buffering the removed I/O with another disk or system/subsystem. The flushed I/O requests are handled by reconstruction/dirty region logging. If there is a subsequent I/O directed to the offline disk, 750, it will also be serviced with redundant disks, 752. If no subsequent I/O is received, the system may wait for the disk to either successfully complete error recovery, or determine that the disk is failed.
After flushing the I/O requests and directing them to be serviced to place bounds on their completion, the disk can be allowed to complete error recovery, 736. Because subsequent I/O will be handled without interaction with the offline disk, the disk can execute its error recovery processing to completion out-of-band. When the error recovery processing is completed, the disk can be onlined, 738. Onlining the disk may include synchronizing data writes that were logged to the dirty region. Either after the disk is onlined, or after the system determines that the disk should be failed (e.g., an unrecoverable error), 760, the process is finished.
Besides what is described herein, various modifications may be made to the disclosed embodiments and implementations of the invention without departing from their scope. Therefore, the illustrations and examples herein should be construed in an illustrative, and not a restrictive sense. The scope of the invention should be measured solely by reference to the claims that follow.
Patent Citations
Non-Patent Citations
Referenced by
Legal Events
Original Image