text:
Buy Perccoet Online
Buy Percocet Online
Percocet is a result of numerous pharmaceuticals and you can purchase Percocet online and from any medicinal store. What is Percocet? It is a pharmaceutical or we can say an agony reliever which is given to the general population who are experiencing the intense or incessant torment. It is generally given to the general population who are having a mischance for occurrence an auto collision harm or a break. This is a drug which is given to the general population for their advantage yet nowadays it is unfortunately expressed that this pharmaceutical is likewise taking existences of numerous individuals by www.buysomaonlinepharmacy.com/buy-percocet-online/ its extreme habit.
Percocet Online
Percocet, on one hand, is for agony diminishing yet then again it is otherwise called recommended drug. Why? Since it is individuals are utilizing it not as a part of route as recommended by specialist. Specialists endorse it on a specific level which the body of the person who is having it requires however individuals take it past that specific level and take overdose of it. This makes them dependent http://www.dospalosradio.org/buy-percocet-online/ on it and they are not ready to leave that medication effectively. This makes them dependant upon the medication and they are not ready to defeat their torment without the admission of Percocet which in a way makes them dependent on it extremely.
The cells and tissues get to be utilized to of that medication or the endorsed medication and individuals are not ready to abandon it. This thing makes the Percocet infamous when it is celebrated too to treat www.adderallxronline.com/percocet/ extreme agonies as it is an incredible agony executioner.
It is typically given to the general population who are having mishaps and the solution Percocet truly demonstrates great to them if given or taken to a specific point of confinement and if not taken past that farthest point.
Buy Percocet
When we discuss Percocet than we should likewise realize that, you can http://www.adderallxronline.com/percocet/ also. It is a result of pharmaceuticals thus it can be purchased from any website of the pharmaceuticals on the web. It ought to be ensure the site which you are picking is protected and legitimate. On the off chance that you purchase Percocet online then you should know that it has certain preferences also. One of the favorable circumstances is that we can spare our time, vitality and fuel and we get the Percocet at our home by putting in a request as it were. Also we can get it less expensive than the super stores. When you are having a mischance or you are experiencing interminable or intense agony than to purchase Percocet online takes half of the torment off your shoulders by giving it securely at your home with no additional charges. Is it safe to say that it isn't extraordinary? When you are getting alleviation from the torment and you are additionally not heading to the long path to purchase the agony reliever and it is given to you at your place.


text:
Scholarships Closed for Camp 2017
KindTree's scholarship committee, composed of community advisors, KT Board members and members of the KT Camp team, has awarded all full and partial scholarships for 2017. KindTree was able to support 15 campers and caregivers attend Autism Rocks! Friends and Family Camp because of generous donations by the community to the Riley Campbell Memorial Scholarship Fund. KindTree thanks our donors!
Donations received after July 15th will begin to accumulate towards scholarship funds for next year's awards. Campers who were awarded scholarships for 2017 will not be eligible for full scholarships in 2018, but may still apply for partial assistance.
With Talk Of Medicaid Changes, Waiver Services May Be At Risk

subset: commoncrawl


text:
ChloeBrown Chloe Perforated Leather "Charlie" Drawstring Sling Bag
This chic Chloe sling bag is perfect for a casual weekend of city exploring. Constructed of brown leather with trendy perforation throughout the top. Drawstring adjustable top opening with leather snap panel across the top. The structured base has one small zipper pocket on the front. Features a removable leather zipper pochette. Comes in dust bag with authenticity card. Retails for $1,695.
Made in: Italy Color: Brown Fabric Content: Leather, Canvas, Metal Condition: Pre-owned. This item shows signs of wear. Faint markings and discoloration at interior. Shallow scratches throughout the leather. Faint tarnishing and scratching on hardware.
Measurements & Sizing
- Total Height: 16.5"
- TotalWidth: 9.75"
- Depth: 9.75"
- Strap Length: 28.5"
- StrapDrop: 16.5"
Returns & Shipping
Luxury Garage Sale guarantees that all items sold on our site are 100% authentic. All products we accept are inspected for brand-specific guidelines and are put through a rigorous authenticity test process.
LGS is not affiliated with Chloe. We guarantee this Chloe item to be authentic. Chloe® is a registered trademark of Chloe.
Item #: 115445


text:
Dancefloor teams will never be the same
Dancefloor teams will never be the same
There are a few different statuses that DJ Kanban understands for playing, fading, filtering, and looping your sounds. You can set up an agile board with filters, swimlanes, columns and the statuses that you need.
Every issue you have defines a sample with a precise cue point and tempo defined. It could be anything from a short loop to a full song.
The add-on runs an audio engine using pure HTML5 and Javascript. It keeps track of the tempo, cues up sounds based on status changes and makes sure everything is in sync. It also has some visualisations, just because.
More details
Instead of boring old user stories, DJ Kanban responds to issues that represent a sample (such as a drum loop). When an issue is moved from the backlog, sweet sweet music can be heard.
This add-on is intended as a novel demonstration of Atlassian Connect. It requires specific additional setup and is not recommended for use in production instances.
User reviews(8)Sign in to write a review
2015: https://www.youtube.com/watch?v=HBihqRRLx1s
2016 (3 sets): https://www.mixcloud.com/mindsocket/
I'll try to throw something together for setup. In the meantime, I've added some samples to the README to help get things started.
If you just want to see it in action, here you go: https://youtu.be/KtMKy4KEUWU?t=19m20s
Basic support resources are available for this app.
Vendor support resources
Find out how this app works.
Atlassian-hosted discussions connect you to other customers who use this app.
JIRA Cloud • Released 2015-11-02
First release
- Log into your JIRA instance as an admin.
- Click the admin dropdown and choose Add-ons. The Find new add-ons screen loads.
- Locate DJ Kanban.
- Click Install to download and install your app.
- You're all set! Click Close in the Installed and ready to go dialog.


text:
Learn more about GAYFORD PROPERTIES LIMITED. Check the company's details for free and view the Companies House information, company documents and list of directors.
Know about every new company document that is added!
Take advantage of our watch this company feature free of charge and have notifications sent directly to your inbox. Whenever a new document appears on this website, or if their particulars change, you will be immediately informed about such changes. To monitor these changes, all you need to do is click on watch this company.
|Fixed Assets
|£ 312,226
|Tangible Fixed Assets
|£ 300,000
|Current Assets
|£ 79,647
|Tangible Fixed Assets Depreciation
|£ 5,076
|£ 1,178
|Shareholder Funds
|£ 378,294
|Profit Loss Account Reserve
|£ 85,842
|Revaluation Reserve
|£ 272,452
|Called Up Share Capital
|£ 20,000
|Total Assets Less Current Liabilities
|£ 378,294
|Net Current Assets Liabilities
|£ 66,068
|Creditors Due Within One Year
|£ 13,579
|Cash Bank In Hand
|£ 78,469
|Share Capital Allotted Called Up Paid
|£ 6,000
|Number Shares Allotted
|£ 6,000
|Tangible Fixed Assets Cost Or Valuation
|£ 300,000
|Investments Fixed Assets
|£ 12,226
Information about the Private Limited Company GAYFORD PROPERTIES LIMITED has been prepared for information purposes only. It is not intended to be nor does it constitute legal advice. This is public information provided by the official company register.


text:
It's a journey right?
Thursday, December 31, 2015
It's Been A While...
Hello Again Old FriendSuch changes, such changes. I am out of the sports wreath making business as I am currently finishing up a two year Ultrasound program. It has been backbreaking and harder than being pregnant while still being a Mommy. Thank God and the Moon and Stars for my husband cause, the Gospel truth, I wouldn't have been able to do it without him.
It's a journey right?
It's a journey right?
Posted by Christy at 7:27 AM No comments: Links to this post
Labels: dean's list, MCC, Phi Theta Kappa, Radiology, student, ultrasound
Subscribe to: Posts (Atom)


text:
Though she loves acting, dancing in front of the camera is something she's not very fond of. The former Miss World told CS, "I tried my hand at Hindi films but it didn't take me a lot of time to realise that this is not something I want to make a career of.
Having said that, I love acting and am active in it." Diana is looking forward to the release of an offbeat supernatural thriller in which she is playing a lead role.


text:
Ingredient DetailsView All Ingredients
A green pigment derived from minerals with some bio accumulation concerns. Low hazard for health. Suitable for greywater and septic tanks.
Also known as Chromium(III) Oxide, Chrome green is a pigment derived from minerals. In nature, it occurs as the rare mineral Eskolaite.This ingredient contains trivalent chromium, a form of chromium that functions as an essential trace element in human metabolism, Although there are some concerns of its persistence in the environment, which accounts for its moderate EWG score
|INCI Name
|Ingredient Origins
|Common Name
|Chrome Green
|EWG Rating
Learn more on the EWG website
Products using this Ingredient
Enter your location and find a store in your area.
Select your favorite products and have them shipped to your door.Visit our online Shop
We help protect your health by minimising exposure to harmful chemicals.Learn more about us


text:
McCall's 8564; ©1983; Misses' Bathing Suits - For Two-Way Stretch Knits Only: Bathing suit A, cut bias, has elastic in neck, armhole and leg casings. Bathing suit B, cut bias, has side seam gathers, elastic in leg and upper casings and optional tie; inside bra is attached and has feather boning at sides. Bathing suit C has gathered bra with loop, elastic in leg and upper casings and optional tie. All views have top-stitching.
Links to reviews/blog posts Edit
Sources/Vendors Edit
- New Vintage Studio Sizes 12-14-16, FF
- Holding Patterns ATX Sizes 12 14 16
- sew betty and dot
Gallery Edit
[insert your photos of this pattern made up]
Wishlist Edit
[insert your username, and make sure your preferences allow for people to contact you via e-mail. Do not list your email address here!]


text:
LUXAC uniquely combines the elegance of a handbag with the function of a rucksack. A bag that is minimalist by design but will happily carry a laptop. a bag that is both gorgeous and practical.
Designed for 'women who do'
LUXAC is for women on the move, women who perhaps combine the rigours of a school gate drop-off with going to work and a social life. women who need to carry all the usual essentials as well as today's hi-tech accessories. designed for comfort, LUXAC aims to minimize the strain of carrying heavy items but at the same time look cool and desirable. a sharply designed range of bags, ideal for the professional, mother and, perhaps, occasional rock chick.
Designed to work, built to last
The LUXAC collection is made using high quality, hand-finished, vegetable-dyed italian leather and riri zips, probably the best in the world. each bag is fully double-lined for durability. the small interior and exterior zip pockets, sometimes in contrasting colours, and the larger interior accessory-designed pouches are fully lined in the motif cotton, as are the two additional leather-trimmed open interior pockets.
The zips and metal parts combine quality, ease of use and style and come in three fully coordinated ranges: vintage bronze, vintage silver and silver nickel, specific to each bag design. the base of luxac max and midi is studded for extra resistance to damage.
Opening a luxac reveals the hidden gem: neatly concealed within its smart exterior, max, midi and mini feature a colour-matched quality cotton lining ranging from the simple but striking trees (twiggy) to more challenging motifs such as topless cowboys (giant) and skulls (skulls 'n' roses).


text:
IOI's Jeon So Mi discussed J.Y. Park's feedback on the group's new music release.
During a press conference for IOI's 'miss me?' release, Jeon So Mi was asked about the producer's feedback. She answered, "J.Y. Park is my agency's producer. It was my first time singing in front of him and getting feedback, so I was very excited and happy."
SEE ALSO: J.Y. Park expresses his desire to run 'Party People' for the rest of his life
Asked about the specific comments she received, Jeon So Mi said, "He complained that I was bad at aegyo, that I need to add more emotions into the song...I am very nervous about singing this new song." Regarding plans after IOI, she explained, "I don't have any definite plans yet. I already talked to my agency to focus on IOI now, and then I will concentrate on practice then I will comeback."


text:
Melbourne: American world number one Tiger Woods is so good he makes even six-times major winner Nick Faldo sit back in awe, the Briton said Thursday.
What Tiger has been doing did inspire me, Faldo told a news conference after carding a two-over-par 74 in the first round of the Australian Masters in wet and windy conditions in Melbourne.
You learn from the guy, added 45-year-old Faldo, who is enjoying the twin roles of rival and on-course mentor for younger players such as American teenager Ty Tryon.
Asked if the 18-year-old, who had a 74 playing alongside Faldo at the Huntingdale course, was welcome to pick the Englishmans brain during the round, Faldo said: Of course he is. I enjoy it. I get a big kick out of it.
Faldo said Woods had shown the golfing world how to take the game to a new level. Ability is not enough any more. Youve got to be physically strong, youve got to have a good diet and understand that, youve got to be mentally strong and understand the mental side of the game, Faldo said.
So you need a complete package. Tiger has proved that. He is streets ahead of everybody in all three of those areas.
Faldo said he had employed a sports psychologist in 1982, at a time when such things had to be kept quiet. They thought you were loopy if you talked to a psychologist, Faldo said.
I tried eating all the nuts and berries and a bag of grass... I think it comes down to each individuals commitment.
Faldo said his experience had helped him in the tough conditions on Thursday.
It was a seasoned campaigners day today, wasnt it' Where the experience, all my eight million rounds of golf, came in there, he said.
Tryon said he had not played in worse conditions. On number five when it was raining and windy, I did not have my rain gear. I just had a vest and pants, the 18-year-old told a news conference. I did not think if it rained like that we would play in it. I did not know you had hurricanes in Australia. It has been a long round (five-and-a-half hours).
The American said he was thrilled to play with his boyhood idol Faldo. It was a real treat, considering I have watched him win a lot of majors on TV, Tryon said.
To be playing with him is really neat. It is a dream come true. He never hits a shot if he is not comfortable. I think you can learn a lot from that.


text:
Colorado Senate candidate Jane Norton (R) is hoping to derail her Republican rival's primary campaign by using a "caught on tape moment" in her new TV ad.
Norton's 30-second spot features video of Weld County District Attorney Ken Buck pitching for votes because he doesn't "wear high heels."
"I have cowboy boots; they have real bulls--t on them," he said. "And that's Weld County bulls--t."
The video, which has been circulating online, has already generated significant media attention, although versions of it posted on YouTube drew a modest viewership.
That's about to change. The TV ad is airing statewide on broadcast and cable, according to a spokeswoman for Norton. OnMessage Media produced the spot.


text:
Kjaer Weis - Lip Tint
I like the lips to look "modern": not overly lipstick-y; moist, not shiny; a beautiful finish without looking made-up. This Lip Tint contains moisture with a hint of color. I usually apply it with my fingers. It gives a contemporary, effortless look. - Kirsten
"I can't think of anyone this red wouldn't look good on." - Kirsten Kjaer Weis
A warm, deep color inspired by the rich, velvety red petals of a rose. Use a little for a sheer wash of color; layer on more for an opaque red lip. We recommend applying Lover's Choice with the Kjaer Weis Lip Brush.
"I prefer applying 'Bliss Full' with the warmth of the fingers, it is a beautiful bubblegum pink." -Kirsten Kjaer Weis
Bliss Full is a see-through rosy pink, great for Pale to Medium-Dark skin tones. It adds moisture with a hint of color, for the perfect modern lip. The all-natural Beeswax in this formula gives lips a creamy, moisturizing feel, with the light lemon scent of Lemon Peel Oil. Bliss Full achieves a contemporary, effortless look, with the playfulness of a beautiful bubblegum pink.
"'Dream State' is what I wear on my lips. It is a beautiful nude that completes a natural, yet polished look." - Kirsten Kjaer Weis
Dream State is the perfect nude for women who want to complete their look without color on their lips. It's great for neutralizing natural lip colors, and adding velvety moisture, without a glossy finish. Your lips will adore the 95% organic ingredients found in the KW Lip Tint formulas. Shea butter, jojoba oil, beeswax- sourced for a feeling of lushness on the lips, consistent application, and a polished, modern look.
"The 'Goddess' Lip Tint is a rich burgundy tone- beautiful for day, and PERFECT for night." - Kirsten Kjaer Weis
Goddess is a long-lasting, deep burgundy color that looks beautiful on all skin tones. It's a stunning edition to a dark eye for nighttime, without looking overly made-up. Goddess also looks great on its own, for a modern look that gives an effortless type of feel. This rich pigment Lip Tint feels velvety, delivering a lot of moisture and color to lips, without looking shiny.
"'Passionate' is a beautiful raspberry. Almost like a natural lip tone, with a little extra." -Kirsten Kjaer Weis
Passionate adds a raspberry hue to your natural lip color, giving you a healthy, not overly made-up, look. This is a pretty, translucent, berry shade that leaves lips with a hint of a color. Passionate works well with all skin tones, and the beeswax used in this formula adds moisture to lips without looking glossy, leaving lips with a lovely velvet feel.
"'Sensuous Plum' gives the hint of a little make-up, with a chic, polished finish to it." - Kirsten Kjaer Weis
Sensuous Plum is a beautiful, classic plum color. This shade falls between the KW Goddess and Passionate Lip Tints, and can be a little dressed up. This is a great classic color that fabulously compliments a chic look. Sensuous Plum is a colder tone. It's very pretty on Pale, Medium, and Dark skin tones. Gives the subtle hint of a little make-up, with a sophisticated, polished finish.
"'Amazed' is a lip color that's innocent but flirty, inspired by the starlets of the 60s."- Kirsten Kjaer Weis
A pale, fresh lilac pink
Our lightest shade of Lip Tint
Smooth, velvety texture and finish
Try pairing this color with golden-toned makeup, like our Magnetic eye shadow
"'Rapture' is a color a 70's "it" girl would wear, like Bianca Jagger. It looks just as good now as it did then." – Kirsten Kjaer Weis
A bright, blue-based pink suitable for all skin tones, and especially flattering on darker complexions
Smooth, velvety texture and finish
We love pairing this color with a defined eye for a look of understated polish.
"I created our first light brown Lip Tint with a subtle gold shimmer. It's a neutral that plays well with the entire Kjaer Weis line." – Kirsten Kjaer Weis
A beautiful nude/brown that is very versatile.
It is our first lip tint that has a slight golden iridescent finish, very understated but gives the lips a slightly more "glossy" finish
Works beautifully for the Fall look as an understood nude finish to the more poppy cheeks. Works best on pale to medium dark skin tone


text:
While fashion can be hard to predict, CNNMoney.com argues that Deckers Outdoor Corp. is far from experiencing the same kind of consumer backlash that once plagued Crocs Inc. .
For instance, the company's fundamentals point toward an average 30 per cent year-over-year increase in earnings for the past five years, the author states. What's more, analysts are predicting another rise of 24 per cent annually "over the next few years."
Outside the fundamental realm, analysts are backing the shares, while short sellers are backing away. For instance, Sam Poser, an analyst with Sterne, Agee & Leach in New York, told CNNMoney that investors should "no longer dismiss Deckers and the UGG brand as mere footwear fashion fads." What's more, the company's short-to-float ratio has fallen from between 15 per cent to 20 per cent a few years ago to about 8 per cent currently.
The article concludes by stating that while many still believe that "UGG is a fashion faux pas" and that the shoes are "unattractive footwear," there is nothing unattractive about DECK shares.
Contrarian Takeaway:
While it's hard to argue with CNNMoney.com's fundamental analysis of DECK, the article's sentiment indicators could use a bit more perspective. For instance, short interest currently accounts for about 8 per cent of the stock's total float, but only following a rise of nearly 12 per cent in the number of shorted shares during the most recent reporting period. What's more, while there are bullish analysts in DECK's corner, Zacks reports that seven of the 14 brokerage firms following the stock still rate it a "Hold."
As contrarian investors know, rising pessimism on an outperforming stock is a recipe for additional gains. For instance, the stock's ratings backdrop leaves ample room for additional upgrades or initiations, such as the recent "Buy" rating and $75 price target from Jefferies. Furthermore, as the stock continues to rally, short sellers could be forced to reconsider their bearish bets, and an 8 per cent short-to-float ratio could provide more than enough fuel for a potential short-covering rally.
Finally, it should be noted that DECK recently rallied to a fresh all-time high. The stock has soared more than 100 per cent since the start of 2010, and has outperformed the S&P 500 Index by more than 30 per cent, on a relative-strength basis, during the past 60 trading days. Short-term traders may want to be wary of a consolidation period following recent gains, as DECK waits for support in the form of its 10-day and 20-day moving averages to catch up.
Joseph Hargett writes for Schaeffer's Investment Research
Report Typo/Error


text:
Five Reasons Not to Shop on Black Friday
Every year around this time in the United States, we celebrate a day so important, so American that it brings people together like no other day can. Around 3 a.m. this Friday, bleary-eyed shopaholics will power into retail outlets all over the country in an adrenaline-filled stupor to grab the latest in toys, electronics and whatever else they can snatch from the unsuspecting hands a total stranger. We call that day Black Friday.
As American as it may be to literally climb over people in the name of rabid consumerism, we at Hair Balls would like you to consider an alternative and just say no to Black Friday this year.
We don't want you to stop shopping in some sort of anti-capitalist protest like Buy Nothing Day. We're just looking out for your safety, sanity and wallet this holiday season.
If you still aren't convinced, take a look at our list of five reasons not to shop on Black Friday and you will see we are right.
Upcoming Events
Houston Texans vs. Kansas City Chiefs
TicketsSun., Oct. 8, 7:30pm
Houston Texans vs. Cleveland Browns
TicketsSun., Oct. 15, 12:00pm
Monster Jam
TicketsSat., Oct. 21, 7:00pm
Rice Owls Football vs. LA Tech
TicketsSat., Oct. 28, 2:30pm
5. Lowers Your Chances of Being Trampled to Death
We may not live on Long Island and people here may be slightly more polite than these folks, but the heightened state of awareness brought on by the 30 triple shot espressos you had while trying to stay awake in line for 12 hours and praying to God you get your hands on that combination action figure-can opener-bass boat your little boy has been begging for since summer can create some, let's just say, problems. Avoid the crowds. Avoid the cleat marks. That's our motto.
4. Amazon Already Has What You Want...For Less Money
Best Buy may fool a bunch of lesser individuals into rushing the front door of a store at 5 a.m., but we know you are smarter than that. We encourage you to support local retailers, but doing so on Black Friday often means taking your life and your sanity in your own hands. For your own safety, you might consider a little online shopping outfit known as Amazon.com. Sure, they sell books for pedophiles, but they also have laptops, cameras, video games and uranium ore. When you factor in the cost of your dignity, online shopping is the smart choice.
3. Oprah's Favorite Things Won't Piss You Off
Not everyone gets the chance to scream, cry and piss yourself when Oprah starts yelling "You get a car! You get a car!" Not spending any money on Black Friday means you don't have to try and reproduce that thrill to your family members when you scream, "You get a Blu-Ray player that only cost me $30 because I was up at 3 a.m. standing in line at Fry's!" It just doesn't have the same zip to it.
2. No One Wants to Be at Wal-Mart at 3 a.m. for Anything
It's comforting to know that while most of us are safe in our beds, crime will have drifted far from our neighborhoods to the parking lots of Wal-Marts across the country. Perhaps no other store pulls out all the stops (and by "stops" we mean "security") for Black Friday than the retail giant, but it doesn't mean you should camp out in the suburbs to get a spot in line for a television because you might get into a brawl, which is pretty much what Jesus intended for celebrating the day of his birth.
1. You Won't End Up Broke
The most obvious reason not to hit the stores on Black Friday is money. Save it for hookers and blow, which are the gifts that keep on giving.
Get the ICYMI: Today's Top Stories Newsletter Our daily newsletter delivers quick clicks to keep you in the know
Catch up on the day's news and stay informed with our daily digest of the most popular news, music, food and arts stories in Houston, delivered to your inbox Monday through Friday.
Recommended for You
Powered by SailThru
Upcoming Events
Rice Owls Football vs. FIU
TicketsSat., Sep. 23, 6:30pm
Houston Texans vs. Tennessee Titans
TicketsSun., Oct. 1, 12:00pm
Battle of the Piney Woods: SFA vs. SHSU
TicketsSat., Oct. 7, 1:00pm
Rice Owls Football vs. Army West Point
TicketsSat., Oct. 7, 5:30pm
Sponsor Content


text:
|Publication number
|Publication type
|Application number
|Publication date
|Jun 21, 2011
|Filing date
|Mar 9, 2004
|Priority date
|Mar 9, 2004
|Fee status
|Publication number
|Shane S. Owara, Sahn Lam
|Original Assignee
|Netapp, Inc.
|Export Citation
|BiBTeX, EndNote, RefMan
|Patent Citations (49), Non-Patent Citations (52), Referenced by (51), Classifications (13), Legal Events (4)
|External Links: USPTO, USPTO Assignment, Espacenet
The present invention relates to data storage systems and, in particular, to data backup operations and browsing of backups.
A storage appliance is a computer that provides file service relating to the organization of information on storage devices, such as disks. The file server (or filer) includes a storage operating system that implements a file system to logically organize the information as a hierarchical structure of directories and files on the disks. Each "on-disk" file may be implemented as a set of data structures, e.g., disk blocks, configured to store information. A directory, on the other hand, may be implemented as a specially formatted file in which information about other files and directories are stored.
A filer may be further configured to operate according to a client/server model of information delivery to thereby allow many clients to access files stored on a server, e.g., the filer. In this model, the client may comprise an application, such as a database application, executing on a computer that "connects" to the filer over a direct connection or computer network, such as a point-to-point link, shared local area network (LAN), wide area network (WAN), or virtual private network (VPN) implemented over a public network such as the Internet. Each client may request the services of the file system on the filer by issuing file system protocol messages (in the form of packets) to the filer over the network.
A common type of file system is a "write in-place" file system, an example of which is the conventional Berkeley fast file system. By "file system" it is meant generally a structuring of data and metadata on a storage device, such as disks, which permits reading/writing of data on those disks. In a write in-place file system, the locations of the data structures, such as inodes and data blocks, on disk are typically fixed. An inode is a data structure used to store information, such as metadata, about a file, whereas the data blocks are structures used to store the actual data for the file. The information contained in an inode may include, e.g., ownership of the file, access permission for the file, size of the file, file type and references to locations on disk of the data blocks for the file. The to references to the locations of the file data are provided by pointers in the inode, which may further reference indirect blocks that, in turn, reference the data blocks, depending upon the quantity of data in the file. Changes to the inodes and data blocks are made "in-place" in accordance with the write in-place file system. If an update to a file extends the quantity of data for the file, an additional data block is allocated and the appropriate inode is updated to reference that data block.
Another type of file system is a write-anywhere file system that does not overwrite data on disks. If a data block on disk is retrieved (read) from disk into memory and "dirtied" with new data, the data block is stored (written) to a new location on disk to thereby optimize write performance. A write-anywhere file system may initially assume an optimal layout such that the data is substantially contiguously arranged on disks. The optimal disk layout results in efficient access operations, particularly for sequential read operations, directed to the disks. A particular example of a write-anywhere file system that is configured to operate on a filer is the Write Anywhere File Layout (WAFL™) file system available from Network Appliance, Inc. of Sunnyvale, Calif. The WAFL file system is implemented within a microkernel as part of the overall protocol stack of the filer and associated disk storage. This microkernel is supplied as part of Network Appliance's Data ONTAP™ software, residing on the filer, that processes file-service requests from network-attached clients.
As used herein, the term "storage operating system" generally refers to the computer-executable code operable on a computer that manages data access and may, in the case of a storage appliance, implement file system semantics, such as the Data ONTAP™ storage operating system that is implemented as a microkernel. The storage operating system can also be implemented as an application program operating over a general-purpose operating system, such as UNIX� or Windows NT�, or as a general-purpose operating system with configurable functionality, which is configured for storage applications as described herein.
Disk storage is typically implemented as one or more storage "volumes" that comprise physical storage disks, defining an overall logical arrangement of storage space. Currently available filer implementations can serve a large number of discrete volumes (150 or more, for example). Each volume is associated with its own file system and, for purposes hereof, volume and file system shall generally be used synonymously. The disks within a volume are typically organized as one or more groups of Redundant Array of Independent (or Inexpensive) Disks (RAID). RAID implementations enhance the reliability/integrity of data storage through the redundant writing of data "stripes" across a given number of physical disks in the RAID group, and the appropriate caching of parity information with respect to the striped data. In the example of a WAFL file system, a RAID 4 implementation is advantageously employed. This implementation specifically entails the striping of data across a group of disks, and separate parity caching within a selected disk of the RAID group. As described herein, a volume typically comprises at least one data disk and one associated parity disk (or possibly data/parity partitions in a single disk) arranged according to a RAID 4, or equivalent high-reliability, implementation.
In order to improve reliability and facilitate disaster recovery in the event of a failure of a filer, its associated disks or some portion of the storage infrastructure, it is common to "mirror" or replicate some or all of the underlying data and/or the file system that organizes the data. In one example, a mirror is established and stored at a remote site, making it more likely that recovery is possible in the event of a true disaster that may physically damage the main storage location or it's infrastructure (e.g. a flood, power outage, act of war, etc.). The mirror is updated at regular intervals, typically set by an administrator, in an effort to catch the most recent changes to the file system. One common form of update involves the use of a Snapshot™ process.
Snapshotting is further described in TR3002 File System Design for an NFS File Server Appliance by David Hitz et al., published by Network Appliance, Inc., and in U.S. Pat. No. 5,819,292 METHOD FOR MAINTAINING CONSISTENT STATES OF A FILE SYSTEM AND FOR CREATING USER-ACCESSIBLE READ-ONLY COPIES OF A FILE SYSTEM by David Hitz et al., which are hereby incorporated by reference. The term "Snapshot" is a trademark of Network Appliance, Inc. It is used for purposes of this patent to designate a persistent consistency point (CP) image. A persistent consistency point image (PCPI) is a point-in-time representation of the storage system, and more particularly, of the active file system, stored on a storage device (e.g., on disk) or in other persistent memory and having a′name or other unique identifier that distinguishes it from other PCPIs taken at other points in time. A PCPI can also include other information (metadata) about the active file system at the particular point in time for which the image is taken. Note that the terms "PCPI" and "Snapshot™" may be used interchangeably throughout this patent without derogation of Network Appliance's trademark rights.
By way of background, a snapshot or PCPI is a restorable version of a file system created at a predetermined point in time. PCPIs are generally created on some regular schedule. The PCPI is stored on-disk along with the active file system, and is called into the buffer cache of the storage appliance memory as requested by the storage operating system. An exemplary file system data identifier buffer tree structure (using inodes in this example—but other forms of block and data identifiers can be employed) 100 is shown in
When the file system generates a PCPI of a given file system, a PCPI inode 205 is generated as shown in
After a PCPI has been created and file data blocks modified, the file system layer can reconstruct or traverse (or otherwise gain access to) the file system inode structure as it existed at the time of the snapshot by accessing the PCPI inode. By following the pointers contained in the PCPI inode 205 through the inode file indirect block 110 and inode file data block 115 to the unmodified file data blocks 120A-C, the file system layer can reconstruct the file system as it existed at the time of creation of the snapshot.
Generally, a PCPI is an image (typically read-only) of a file system at a point in time, which is stored on the same primary storage device as is the active file system and is accessible by users of the active file system. Note, that by "active file system" it is meant the file system to which current input/output (I/O) operations are being directed. Each time a PCPI occurs, the old active file system becomes the new PCPI, and the new active file system carries on, recording any new changes. A set number of PCPIs may be retained depending upon various time-based and other criteria. The PCPI process is described in further detail in U.S. patent application Ser. No. 09/932,578, entitled INSTANT SNAPSHOT by Blake Lewis et al., now issued as U.S. Pat. No. 7,454,445 on Nov. 18, 2008, which is hereby incorporated by reference as though fully set forth herein
In "replication" of a file system to a destination (or secondary) backing store, above-described PCPIs are copied over a network (such as the well-known Internet) from a source (or primary) system to the remote (or otherwise discrete) storage site. Note that, as used herein, the term "primary" refers to a storage system on which data is being actively modified (e.g. through applications), while the term "secondary" refers to each storage system used to store backup copies of the primary's data.
The complete recopying of the entire file system to a remote (destination/secondary) site over a network may be quite inconvenient where the size of the file system is measured in tens or hundreds of gigabytes (even terabytes). This full-backup approach to remote data mirroring or replication may severely tax the bandwidth of the network and also the processing capabilities of both the destination and source filer. One solution has been to limit the replica to only portions of a file system volume that have experienced changes. Hence,
In brief summary, the source creates a pair of discrete time-separated PCPIs of the to volume. These can be created as part of the commit process in which data is committed to non-volatile memory in the filer or by another mechanism. The "new" PCPI 410 is a recent PCPI of the volume's active file system. The "old" PCPI 412 is an older or "base" PCPI of the volume, which should match the image of the file system mirrored/replicated on the destination mirror. Note that the file server is free to continue work on new file service requests once the new PCPI 412 is made. The new or "incremental" PCPI acts as a checkpoint of activity up to that time rather than an absolute representation of the then-current volume state. A differencer 420 scans the blocks 422 in the old and new PCPIs. In particular, the differencer works in a block-by-block fashion, examining the list of blocks in each PCPI to compare which blocks have been allocated. In the case of a write-anywhere system, the block is not reused as long as a PCPI references it, thus a change in data is written to a new block. Where a change is identified (denoted by a presence or absence of an 'X' designating data), a decision process 400, shown in
Approaches to volume-based remote mirroring of PCPIs are described in detail in commonly owned U.S. patent application Ser. No. 09/127,497, entitled FILE SYSTEM IMAGE TRANSFER by Steven Kleiman, et al., now issued as U.S. Pat. No. 6,604,118 on Aug. 5, 2003, and U.S. patent application Ser. No. 09/426,409, entitled FILE SYSTEM IMAGE TRANSFER BETWEEN DISSIMILAR FILE SYSTEMS by Steven Kleiman, et al., now issued as U.S. Pat. No. 6,574,591 on Jun. 3, 2003, both of which are expressly incorporated herein by reference.
In some circumstances it may be desirable to replicate less than an entire volume structure. The volume-based approach typically forces an entire volume to be scanned for changes and those changes to be transmitted on a block-by-block basis. In other words, the scan focuses on blocks without regard to any underlying information about the files, inodes and data structures, which the blocks comprise. The destination is organized as a set of volumes so a direct volume-by-volume mapping is established between source and destination. Where a volume may contain a terabyte or more of information, the block-by-block approach to scanning and comparing changes may still involve significant processor overhead and associated processing time. Often, there may have been only minor changes in a sub-block beneath the root inode block being scanned. Since a list of all blocks in the volume is being examined, however, the fact that many groupings of blocks (files, inode structures, etc.) are unchanged is not considered. In addition, the increasingly large size and scope of a full volume make it highly desirable to sub-divide the data being mirrored into sub-groups (for example, qtrees, which are described further below) because some groups are more likely to undergo frequent changes, it may be desirable to update their associated images on the backing store more often than other, less-frequently changed groups. In addition, it may be desirable to mingle original and imaged (Snap-shotted) sub-groups in a single volume and migrate certain key data to remote locations without migrating an entire volume.
One such sub-organization of a volume is the well-known qtree. Qtrees, as implemented on an exemplary storage system such as described herein, are subtrees in a volume's file system. One key feature of qtrees is that, given a particular qtree, any file or directory in the system can be quickly tested for membership in that qtree, so they serve as a good way to organize the file system into discrete data sets. The use of qtrees as a source and destination for replicated data may be desirable. An approach to remote asynchronous minoring of a qtree is described in U.S. patent application Ser. No. 10/100,967 entitled SYSTEM AND METHOD FOR DETERMINING CHANGES IN TWO SNAPSHOTS AND FOR TRANSMITTING CHANGES TO A DESTINATION SNAPSHOT, by Michael L. Federwisch, et al., now issued as U.S. Pat. No. 6,993,539 on Jan. 31, 2006, the teachings of which are incorporated herein by reference.
The use of magnetic tape to store a sequential backup of a data (files, directories, etc.) from a storage system has been popular for many decades. A typical tape backup data is stored on tape media in association with a "backup index." This index points to the location on the tape or another image structure where the particular directory, file, etc. resides. The index allows the user to ascertain the nature of the data stored and to retrieve it when needed for restoration to the storage system (for, example to an active in-core or on-disk file structure). Certain backup applications present aggregated lists across several incremental backups (each increment typically backing up a time-based and/or changed-file chunk of data.
The index traditionally takes the form a table of contents of the various files and directories stored as backups serially on the tape or other media. It is often written directly to the tape as a header (before the backup data) in an appropriate location so that it can be read back to the system at any time. This header may also contain physical tape locations cross referenced to certain backup files for easy location of the data on the tape. This index information (including physical tape locations) may also be located in an on-disk file that is managed by the backup application.
When a tape backup receives an incremental backup (e.g. only certain data, typically containing current changes to files and directories), the data is not a complete picture of the file system. To effectively browse and restore such data, the backup application must first coalesce all relevant incremental backups into a single image and backup indices into an overall table of contents to create the full image of the file system. This requires time and resources.
Where a destination backing store (storage system or filer) receives incremental PCPIs of data, it stores the PCPIs as discrete, identifiable versions of the data with key information (file name and creation time) stored directly in the file system directory tree where it can be easily traversed by the file system, and readily provides the user with a full image of the file system or other data structure (e.g. a qtree) at a certain point in time. Hence, it is desirable to provide a mechanism for indexing data directly from such PCPI data structures on the destination.
This invention overcomes the disadvantages of the prior art by providing a system and method for managing backup and restore operations on a storage system, typically between source (or "primary") storage system and destination (or "secondary") storage system using a backup management client that employs a version of the Network Data Management Protocol (NDMP) to interface with the destination file server and generate an "index" of data structures (such as directories and files) from directly from scanning the trees of PCPIs ("Snapshots™") stored on the destination. The management client includes a command line interface for entry of instructions by a user or application and a web-based user interface and allows the index to be displayed as a series of web pages (HTML-based) on a graphical user interface. The index can be browsed and selected data structures can be restored to a source filer/file server as desired. All backups/PCPIs can be contemporaneously browsed, allowing greater flexibility in selecting a best backup image to restore. Within a given backup, the relative contents can be browsed, providing further assurance to an administrator that the correct/desired versions of particular data sets (files) within the browsed backup are present. Likewise, the creation of PCPIs for use as backups can be scheduled through the management client. In accordance with an advantageous of this invention coalescing of individual backups and indices into a resulting image and table of contents is unnecessary, as incremental backups may be accessed and browsed directly via their associated PCPIs on the backing store.
In accordance with this invention, the user may advantageously access any of a number of incremental backups via their associated PCPIs and browse these destination is backups at will and relatively concurrent with their creation. Restoration of such backups to the source from the destination is straightforward using then associated graphical user interface. Likewise, the user may access the source via NDMP and perform scheduled and unscheduled backups to the destination. The source may be browsed by scanning the associated directory trees. A plurality of source and/or destination storage systems (filers, etc.) can be arranged into a single grouping so that multiple source systems transmit backups to one or more destinations and/or destination volumes.
The above and further advantages of the invention may be better understood by referring to the following description in conjunction with the accompanying drawings in which like reference numerals indicate identical or functionally similar elements:
By way of further background,
The destination system, in this example is a file server or filer 612 that manages one or more destination volumes 616, comprising arrays of disks 660. The source 610 and destination 612 are linked via a network 618 that can comprise a local or wide area network, such as the well-known Internet. An appropriate network adapter 636 and 630, residing the source and destination 610, and 612 respectively, facilitates communication over the network 618. Note, as used herein, the term "source" can be broadly defined as a location from which the subject data of this invention travels and the term "destination" can be defined as the location to which the data travels. While a source system and a destination filer, connected by a network, is a particular example of a source and destination used herein, a source and destination could be computers/filers linked via a direct link, or via loopback (a "networking" arrangement internal to a single computer for transmitting a data stream between local source and local destination), in which case the source and the destination may comprise the same filer. Likewise, the term's source and to destination as used herein can be alternatively referred to as the "primary" and "secondary" storage systems, file servers or filers as appropriate.
The destination filer 612 comprises a processor 620, a memory 625, a network adapter 630 and a storage adapter 640 interconnected by a system bus 645. The destination filer 612 also includes a storage operating system 700 (
It will be understood to those skilled in the art that the inventive technique described herein may apply to any type of special-purpose computer (e.g., file serving appliance) or general-purpose computer, including a standalone computer, embodied as a storage system. To that end, the destination filer 612 can be broadly, and alternatively, referred to as a "storage system." Moreover, the teachings of this invention can be adapted to a variety of storage system architectures including, but not limited to, a network-attached storage environment, a storage area network and disk assembly directly-attached to a client/host computer. The term "storage system" should, therefore, be taken broadly to include such arrangements.
In the illustrative embodiment, the destination memory 625 comprises storage locations that are addressable by the processor and adapters for storing software program code. The memory comprises a form of random access memory (RAM) that is generally cleared by a power cycle or other reboot operation (i.e., it is "volatile" memory). The processor and adapters may, in turn, comprise processing elements and/or logic circuitry configured to execute the software code and manipulate the data structures. The storage operating system 700, portions of which are typically resident in memory and executed by the processing elements, functionally organizes the filer by, inter alia, invoking storage operations in support of a file service implemented by the filer. It will be apparent to those skilled in the art that other processing and memory means, including various computer readable media, may be used for storing and executing program instructions pertaining to the inventive technique described herein.
The network adapter 630 comprises the mechanical, electrical and signaling circuitry needed to connect the filer 612 to the network 618, which may comprise a point-to-point connection or a shared medium, such as a local area network. Moreover the source 610 may interact with the destination filer 612 in accordance with a client/server model of information delivery. That is, the client may request the services of the filer, and the filer may return the results of the services requested by the client, by exchanging packets encapsulating, e.g., the TCP/IP protocol or another network protocol format over the network 618.
The storage adapter 640 cooperates with the operating system 700 (
Each system 610, 612 may also be interconnected with one or more clients (not shown) via the network adapters 638, 630, respectively. The clients transmit requests for file service to the destination filer 612 respectively, and receive responses to the requests over a LAN or other network (618). Data is transferred between the client and the filer 612 using data packets defined as an encapsulation of the Common Internet File System (CIFS) protocol or another appropriate protocol, such as NFS.
In one exemplary filer implementation, the destination filer 612 can include a nonvolatile random access memory (NVRAM) 635 that provides fault-tolerant backup of data, enabling the integrity of filer transactions to survive a service interruption based upon a power failure, or other fault. The size of the NVRAM depends in part upon its implementation and function in the file server. It is typically sized sufficiently to log a certain time-based chunk of transactions (for example, several seconds worth). The NVRAM is filled, in parallel with the buffer cache, after each client request is completed, but before the result of the request is returned to the requesting client.
In an illustrative embodiment, the disks 660 are arranged into a plurality of volumes (for example, source volumes 614 and destination volumes 616), in which each volume has a file system associated therewith. The volumes each include one or more disks 660. In one embodiment, the physical disks 660 are configured into RAID groups so that some disks store striped data and some disks store separate parity for the data, in accordance with a preferred RAID 4 configuration. However, other configurations (e.g. RAID 5 having distributed parity across stripes) are also contemplated. In this embodiment, a minimum of one parity disk and one data disk is employed. However, a typical implementation may include three data and one parity disk per RAID group, and a multiplicity of RAID groups per volume. Information in the volumes may be organized as files, directories and virtual disks (vdisks). That is, to facilitate access to the disks, the destination storage operating system 700 implements a write-anywhere file system that logically organizes the information as a hierarchical structure of directory, file and vdisk objects (hereinafter "directories", "files" and "vdisks") on the disks. A vdisk is a special file type that is translated into an emulated disk or logical unit number (lun) as viewed by a storage are network (SAN) client, which may implement a block-based protocol for storage organization. Each "on-disk" file may be implemented as set of disk blocks configured to store information, such as data, whereas the directory may be implemented as a specially formatted file in which names and links to other files and directories are stored. Vdisks are further described in U.S. patent application Ser. No. 10/216,453, entitled STORAGE VIRTUALIZATION BY LAYERING VIRTUAL DISK OBJECTS ON A FILE SYSTEM, by Vijayan Rajan, et al., now issued as U.S. Pat. No. 7,107,385 on Sep. 12, 2006, the teachings of which are hereby incorporated by reference. As will be made more clear with reference to the storage operating system 700 below, the exemplary destination filer 700 may be characterized more particularly as a multi-protocol storage appliance, capable of receiving storage requests according to a plurality of protocols and performing storage service operations according to either a file-based or block-based organization.
In the illustrative embodiment, the source volume 614 is mirrored to the destination volume using a qtree-based minoring technique, such as that described in the above-incorporated patent application entitled SYSTEM AND METHOD FOR DETERMINING CHANGES IN TWO SNAPSHOTS AND FOR TRANSMITTING CHANGES TO A DESTINATION SNAPSHOT, by Michael L. Federwisch, et al. However, it should be noted that the technique of the present invention may be utilized with any acceptable mirroring technique that may be suitably modified to utilize the teachings of the present invention. To facilitate the transfer of replica data from the source 610 to the destination 612 via the network 618, a formatted data stream 690 is employed. While it is contemplated that a variety of data transmission schemes and formats can be employed according to various alternate embodiments of this invention, and exemplary data format and related concepts are described in commonly owned U.S. patent application Ser. No. 10/777,978, entitled SYSTEM-INDEPENDENT DATA FORMAT IN A MIRRORED STORAGE SYSTEM ENVIRONMENT AND METHOD FOR USING THE SAME, by Stephen L. Manley, now issued as U.S. Pat. No. 7,478,101 on Jan. 13, 2009, the teachings of which are expressly incorporated herein by reference.
The destination storage system 612, as the repository of backup data, organized a set of PCPIs, interfaces with an administrator's management client 692. This can be any acceptable computer or terminal console that runs or manipulates a software backup management application 693 and that (in this example) provides the user/administrator with a graphical user interface (GUI) 694 for displaying and manipulating data. The graphical user interface in this embodiment employs a web-capable browser application 695 that enables data to be displayed and controlled via conventional web pages 696. The web pages are generated using conventional Hypertext Markup Language (HTML) script 697, an engine for which is provided in the backup management application 693. The administrator's management client, the user interface, and its communication with the file server are described, by way of further background, in U.S. patent application Ser. No. 09/862,949, entitled SYSTEM AND METHOD FOR CONSOLIDATED REPORTING OF CHARACTERISTICS FOR A GROUP OF FILE SYSTEMS, by Brian M. Hackworth, published as U.S. Patent Publication No. 2002/0175938 on Nov. 28, 2002, the teachings of which are expressly incorporated herein by reference. In an exemplary commercial implementation, the system software (for use with compatible third-party hardware) is available from Network Appliance, Inc. of Sunnyvale, Calif. as the Data Fabric Manager™.
Note, while a web-based user interface 694 with associated browser 695 and web-pages 696 is shown by way of example, it is expressly contemplated that the user interface can support an alternate graphical user interface or command-line interface standard. For example, the interface can be based upon common gateway interface (CGI) scripts, Sun Microsystems' Java™, or any other acceptable graphical user interface schema. Accordingly the terms "user interface" and "graphical user interface" should be taken broadly to include a variety of approaches to deliver information to a user, and to allow the user to act upon that information and issue commands thereupon.
The management client 692 interfaces with the destination system 612 through a network or other link 698. The interconnection between the management client 692 and the destination system 612 can be direct, or via a distributed network, such as a LAN or the Internet (618). As will be described further below, one protocol that is employed for communication between the management client 692 and destination system 612 is an appropriate version (version 3 or 4, for example) of the open source Network Data Management Protocol (NDMP). NDMP enables a standard set of functions and instructions to be transmitted between a source file system and backup device under control of a management application that is adapted to generate NDMP messages. In this embodiment, NDMP messages are communicated between the backup manager 693 on the management client and an NDMP extension 796 on the storage operating system 700 of the destination system. Where appropriate, an NDMP extension (685) can also reside on the operating system 611 of the source system 610. A full specification for various versions of NDMP is available via the World Wide Web at http:/www.ndmp.org. By way of example, a description of NDMP Version 3 can be found in NDMP-Network Data Management Protocol, written by Network Working Group, R. Stager and D. Hitz, September 1997, the teachings Of which are expressly incorporated herein by reference. Note, while an open source NDMP is used in an exemplary embodiment herein, it is expressly contemplated that another communication protocol can be employed. In general, as depicted in
In the illustrative embodiment, the storage operating system resident on the destination filler is the NetApp� Data ONTAP™ operating system available from Network Appliance, Inc., Sunnyvale, Calif. that implements a Write Anywhere File Layout is (WAFL™) file system. However, it is expressly contemplated that any appropriate storage operating system, including a write in-place file system, may be enhanced for use in accordance with the inventive principles described herein. As such, where the term "WAFL" is employed, it should be taken broadly to refer to any file system and/or storage operating system that is otherwise adaptable to the teachings of this invention.
As used herein, the term "storage operating system" generally refers to the computer-executable code operable on a computer that manages data access and may, in the case of a storage appliance, implement data access semantics, such as the Data ONTAP storage operating system, which is implemented as a microkernel. The storage operating system can also be implemented as an application program operating over a general-purpose operating system, such as UNIX� or Windows NT�, or as a general-purpose operating system with configurable functionality, which is configured for storage applications as described herein.
In addition, it will be understood to those skilled in the art that the inventive technique described herein may apply to any type of special-purpose (e.g., storage serving appliance) or general-purpose computer, including a standalone computer or portion thereof, embodied as or including a storage system. Moreover, the teachings of this invention can be adapted to a variety of storage system architectures including, but not limited to, a network-attached storage environment, a storage area network and disk assembly directly-attached to a client or host computer. The term "storage system" should therefore be taken broadly to include such arrangements in addition to any subsystems configured to perform a storage function and associated with other equipment or systems.
An iSCSI driver layer 745 provides block protocol access over the TCP/IP network protocol layers, while a FC driver layer 740 operates with a Fibre Channel adapter in the filer 612 to receive and transmit block access requests and responses to and from the integrated storage appliance. The FC and iSCSI drivers provide FC-specific and iSCSI-specific access control to the luns (vdisks) and, thus, manage exports of vdisks to either iSCSI or FCP or, alternatively, to both iSCSI and FCP when accessing a single vdisk on the storage appliance. In addition, the storage operating system includes a disk storage layer 760 that implements a disk storage protocol, such as a RAID protocol, and a disk driver layer 765 that implements a disk access protocol such as, e.g., a SCSI protocol.
Bridging the disk software layers with the integrated network protocol stack layers is a virtualization system 770 that is implemented by a file system 785 interacting with virtualization modules illustratively embodied as, e.g., vdisk module 780 and SCSI target module 775. It should be noted that the vdisk module 780, the file system and SCSI target module 775 can be implemented in software, hardware, firmware, or a combination thereof. The vdisk module 780 interacts with the file system 785 to enable access by administrative interfaces in response to a system administrator issuing commands to the filer 612 in its arrangement as a multi-protocol storage appliance. In essence, the vdisk module 780 manages SAN deployments by, among other things, implementing a comprehensive set of vdisk (lun) commands issued through a user interface by a system administrator. These vdisk commands are converted to primitive file system operations ("primitives") that interact with the file system 785 and the SCSI target module 775 to implement the vdisks.
The SCSI target module 775, in turn, initiates emulation of a disk or lun by providing a mapping procedure that translates luns into the special vdisk file types. The SCSI target module is illustratively disposed between the FC and iSCSI drivers 740, 745 and the file system 785 to thereby provide a translation layer of the virtualization system 770 between the SAN block (lun) space and the file system space, where luns are represented as vdisks.
The file system 785 illustratively implements the WAFL file system having an on-disk format representation that is block-based using, e.g., 4-kilobyte (KB) blocks and using inodes to describe the files. The WAFL file system uses files to store metadata describing the layout of its file system; these metadata files include, among others, an inode file. A file handle, i.e., an identifier that includes an inode number, is used to retrieve an inode from disk. A description of the structure of the file system, including on-disk inodes and the inode file, is provided in U.S. Pat. No. 5,819,292, entitled METHOD FOR MAINTAINING CONSISTENT STATES OF A FILE SYSTEM AND FOR CREATING USER-ACCESSIBLE READ-ONLY COPIES OF A FILE SYSTEM, by David Hitz, et al., the teachings of which are hereby incorporated by reference.
The storage operating system 700 further includes, in the illustrative embodiment, a mirroring application 795 that performs volume-based, qtree-based, or another subvolume level mirroring/backup function. The mirroring application 795 is illustratively integrated with the TCP 715 and IP 710 layers and a PCPI processes 790 within the file system 785. The mirroring application 795 utilizes TCP/IP for communication with the upstream and/or downstream within a cascaded chain of storage systems. The mirroring application 795 utilizes the PCPI processes 790 within the file system 785 to determine changed blocks within volumes and/or qtrees.
As described briefly above, overlying the storage operating system 700 is the NDMP extension, executing a version of the NDMP standard in accordance with the above-referenced open source specification. The NDMP extension interfaces with the mirroring application 795 and with the networking layers 710 and 715. It allows the administrator to monitor and browse the directory structure residing on the destination (via the mirroring application), and to instruct the mirroring application to restore selected data to the source as desired.
By way of further background, the general principles of remote asynchronous minoring of a volume or sub-volume/qtree data set from a source to a destination are described in detail in the above-incorporated U.S. patent application Ser. No. 10/100,950, entitled SYSTEM AND METHOD FOR ASYNCHRONOUS MIRRORING OF SNAPSHOTS AT A DESTINATION USING A PURGATORY DIRECTORY AND INODE MAPPING by Stephen L. Manley, et al., now issued as U.S. Pat. No. 7,225,204 on May 29, 2007, and other related applications incorporated hereinabove. The illustrative embodiment of the invention employs the techniques described in these above-incorporated patent applications to receive and store, as a replica data set of the source, on the destination side, a data stream generated by the source. The source data may be organized in a variety of manners and appropriate mechanisms on the source side (in its replication agent 613) are used to generate the formatted data stream 690 in accordance with the illustrative embodiment of this invention. In one embodiment, both the source system and the destination system include respective pipelines (described further below) for generating and decoding the formatted data stream.
With further reference to
As changed information is forwarded over the network, it is received at the destination pipeline piece 830. This pipeline also includes a networker 832 to read out TCP/IP packets from the network into the snapshot replication data stream format 690 encapsulated in TCP/IP. A data reader and header stripper 834 recognizes and responds to the incoming format 690 by acting upon information contained in various format headers (described below). A writer 836 is responsible for placing data derived from the format into appropriate locations on the destination file system. The storage and handling of backup data is implemented by the destination mirroring process 840. Note: the interface of the destination mirroring process with the backup manager, via NDMP, is described above with reference to
Various details and embodiments of the mirroring/replication process 840 are described in detail in the above-incorporated SYSTEM AND METHOD FOR ASYNCHRONOUS MIRRORING OF SNAPSHOTS AT A DESTINATION USING A PURGATORY DIRECTORY AND INODE MAPPING and related applications thereto. Briefly, according to one example, the destination pipeline 830 forwards data and directory information to the main destination mirror process 840. The exemplary destination mirror process 840 consists of a directory stage 842, which builds the new replicated file system directory hierarchy on the destination side based upon the received changes in a PCPI on the source. To briefly summarize, the directory stage creates, removes and moves files based upon the received formatted information. A map of inodes from the destination to the source is generated and updated. In this manner, inode numbers on the source file system are associated with corresponding (but typically different) inode numbers on the destination file system. Notably, a temporary or "purgatory" directory is established to retain any modified or deleted directory entries until these entries are reused by or removed from the replicated PCPI at the appropriate directory rebuilding stage within the directory stage. In addition, a file stage of the destination mirror process populates the established files in the directory stage with data based upon information stripped from associated format headers.
The job management framework 910 provides an infrastructure for abstracting complex operations (e.g. a restore operation having many steps to complete) into a high-level, user-friendly concept. Each high-level concept is characterized as a discrete "job." Some examples of "jobs are backing up a destination volume (922), restoring files to a source directory (922) and adding a source storage system (or filer) to the backup management structure of this embodiment (924). The framework 910 provides primitives for starting parallel operations, logging job status and messages, aborting in-progress operations and other job-handling functions.
The NDMP API 908 provides an interface to all requests defined in the NDMP extension 796 (
The scheduler 914 is a long-running process that interfaces with the database 912 and the CLI 906. In this embodiment, it is adapted to "wake up" or otherwise activate hourly (or another desired time interval), wherein it checks the backup schedules stored in the database 912. Once the schedule is checked, the scheduler initiates any backup jobs that are due. Scheduled jobs are created using the job management framework 910. The jobs are saved to the database and run in a background process.
The backup management application performs a number of functions that are manipulated through the various pages of the GUI 694 (described in further detail below). For the purposes of this description two significant functions are the ability to browse the destination system's file system to locate and retrieve various backups based upon time-created, identity of the source system (for example the particular filer) volume, directory and the like. A basic flow diagram for a browse request procedure 1000 is shown in
Once the requested information has been read at the storage system, the NDMP extension in step 1030 returns the information (e.g. returns existing PCPIs and directories therein) in the NDMP format to the API, and whence to the GUI. This information is displayed as a web page, or any other acceptable user interface/graphical user interface format (see below) with the desired information for the user to act upon (step 1040).
Unlike the browse command, the restore command involves both the destination and source, under control of the backup management application.
Where the user/administrator desires a backup process to be performed from the source to the destination, the source is messaged via NDMP over the network from the GUI to perform the backup in a manner similar to the restore process 1100 of
The following is a description of various user interface "web pages" displayed by the GUI according to an embodiment of this invention and various generalized functions that they perform.
By clicking the "edit" button 1690 for a selected volume, the screen 1700 of
When clicking the "browse backups" button 1680 for a given volume in
If the user clicks the restore button 1980 on a given directory, the backup manager GUI directs the user through a number of screens that specify which files or other data containers from a selected directory will be restored, and from which backup. First, in screen 2100 (
Once the files and/or directories for restoration are selected, the destination 2410 is selected in screen 2400 (
While backups are typically carried out according to a preset schedule, any destination/secondary volume that is already configured can be backed-up at an unscheduled point in time. Screen 2700 (
In maintaining the health and security of a backup system, the backup management application should monitor the time between backups (termed herein the "lag"). As a top menu function, the GUI also allows access by the user to a threshold option screen 2800 (
Note that the functions of the various GUI screens described herein may be carried out by the backup manager CLI 906 by automatic or non-user-initiated functions in various applications. Appropriate CLI command scripts are employed by the applications to carry out these functions via the CLI.
The foregoing has been a detailed description of embodiments of the invention. Various modifications and additions can be made without departing from the spirit and scope thereof. For example, while files and directories are denoted herein, the data can be organized around a variety of data structures and the terms "file," "file system," "directory" and the like should be taken broadly to include a variety of "data structures," "data sets," or "data organizations." Likewise, while the disk storage layers utilizes a RAID organization, it is contemplated that a variety of storage arrangements can be employed. Similarly, while the storage devices described herein are disks, the principles of this invention can be applied to a variety of storage devices or media including, but not limited to, electro-optical, solid-state, magnetic, and the like. Finally, it should be understood that any and all of the principles described herein can be implemented as hardware, software that consists of a computer readable medium executing program instructions on a computer, or a combination of hardware and software. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this invention.
|Cited Patent
|Filing date
|Publication date
|Mar 28, 1983
|Feb 11, 1986
|Allen Bruce S
|Man machine interface
|Sep 8, 1989
|Nov 10, 1992
|Auspex Systems, Inc.
|Parallel i/o network file server architecture
|Jun 26, 1992
|Apr 13, 1993
|Thinking Machines Corporation
|Storage system using multiple independently mechanically-driven storage units
|Dec 20, 1990
|Jan 11, 1994
|International Business Machines Corp.
|Version management system using pointers shared by a plurality of versions for indicating active lines of a version
|Oct 13, 1992
|Oct 11, 1994
|Auspex Systems, Inc.
|Parallel I/O network file server architecture
|Apr 8, 1994
|Jan 16, 1996
|Auspex Systems, Inc.
|Multiple facility operating system architecture
|May 27, 1994
|Apr 23, 1996
|Hitachi, Ltd.
|File data multiplexing method and data processing system
|Oct 11, 1994
|Sep 1, 1998
|Auspex Systems, Inc.
|Parallel I/O network file server architecture
|May 31, 1995
|Oct 6, 1998
|Network Appliance, Inc.
|Method for maintaining consistent states of a file system and for creating user-accessible read-only copies of a file system
|Feb 25, 1997
|Apr 27, 1999
|International Business Machines Corporation
|Logical volume manager and method having enhanced update capability with dynamic allocation of storage and minimal storage of metadata information
|Jul 30, 1997
|Aug 3, 1999
|Auspex Systems, Inc.
|Parallel I/O network file server architecture
|Dec 31, 1997
|Aug 24, 1999
|Crossroads Systems, Inc.
|Storage router and method for providing virtual local storage
|Jun 5, 1995
|Sep 7, 1999
|Network Appliance, Inc.
|Method for providing parity in a raid sub-system using non-volatile memory
|Jun 30, 1998
|Oct 5, 1999
|Network Appliance, Inc.
|Write anywhere file-system layout
|May 31, 1995
|Mar 14, 2000
|Network Appliance, Inc.
|Method for allocating files in a file system integrated with a RAID disk sub-system
|Jun 7, 1995
|May 16, 2000
|Auspex Systems, Inc.
|Multiple software-facility component operating system for co-operative processor control within a multiprocessor computer system
|Aug 25, 1998
|Sep 12, 2000
|Network Appliance, Inc.
|Coordinating persistent status information with multiple file servers
|Jul 21, 1999
|Oct 24, 2000
|Network Appliance, Inc.
|Method for allocating files in a file system integrated with a raid disk sub-system
|Jan 22, 1998
|Feb 6, 2001
|Bull, S.A.
|Computer system with distributed data storing
|Sep 27, 2001
|Jul 23, 2002
|Crossroads Systems, Inc.
|Storage router and method for providing virtual local storage
|Dec 2, 1999
|Aug 13, 2002
|Emc Corporation
|Snapshot copy facility for a data storage system permitting continued host read/write access
|Oct 25, 1999
|Jun 3, 2003
|Network Appliance, Inc.
|File systems image transfer between dissimilar file systems
|Feb 20, 2001
|Aug 12, 2003
|Hewlett-Packard Development Company, L.P.
|System and method for accessing a storage area network as network attached storage
|Feb 4, 2004
|Jun 3, 2008
|Emc Corporation
|Internet protocol based disaster recovery of a server
|May 20, 1999
|May 24, 2001
|Jennifer French
|System and method for authentication of network users with preprocessing
|Jul 31, 1998
|Nov 22, 2001
|Steven Kleiman
|File system image transfer
|Aug 17, 2001
|Jun 27, 2002
|Network Appliance, Inc.
|Instant snapshot
|Dec 18, 2000
|Aug 15, 2002
|Spinnaker Networks, Inc.
|Mechanism for handling file level and block level remote file accesses using the same server
|Dec 7, 2000
|Aug 22, 2002
|Spinnaker Networks, Inc.
|Method and system for responding to file system requests
|May 22, 2001
|Nov 28, 2002
|Hackworth Brian M.
|System and method for consolidated reporting of characteristics for a group of file systems
|Sep 28, 2001
|Apr 3, 2003
|Brannock Kirk D.
|Method for atomically updating a plurality of files
|Dec 5, 2001
|Jun 5, 2003
|Microsoft Corporation
|System and method for persisting and resolving application assembly binds
|Apr 12, 2002
|Jul 10, 2003
|Hiroshi Arakawa
|Virtualized volume snapshot formation method
|Jan 22, 2003
|Sep 4, 2003
|Green Robbie A.
|Persistent Snapshot Management System
|Mar 19, 2002
|Sep 25, 2003
|Chen Raymond C.
|System and method for redirecting access to a remote mirrored snapshop
|Mar 19, 2002
|Sep 25, 2003
|Federwisch Michael L.
|System and method for determining changes in two snapshots and for transmitting changes to destination snapshot
|Mar 19, 2002
|Sep 25, 2003
|Manley Stephen L.
|System and method for storage of snapshot metadata in a remote file
|Mar 19, 2002
|Sep 25, 2003
|Manley Stephen L.
|System and method for asynchronous mirroring of snapshots at a destination using a purgatory directory and inode mapping
|Mar 19, 2002
|Sep 25, 2003
|Manley Stephen L.
|Format for transmission file system information between a source and a destination
|Sep 30, 2002
|Jan 15, 2004
|Anand Prahlad
|System and method for generating and managing quick recovery volumes
|Feb 28, 2003
|Feb 5, 2004
|Thomas Charles Gomer
|Localization and targeting of data in broadcast streams
|Aug 9, 2002
|Feb 12, 2004
|Vijayan Rajan
|Storage virtualization by layering virtual disk objects on a file system
|Oct 7, 2003
|Dec 9, 2004
|Anand Prahlad
|System and method for managing stored data
|Feb 12, 2004
|Sep 1, 2005
|International Business Machines Corporation
|Method of converting a filesystem while the filesystem remains in an active state
|Nov 15, 2004
|Sep 1, 2005
|Anand Prahlad
|System and method for performing an image level snapshot and for restoring partial volume data
|May 4, 2005
|Sep 29, 2005
|Filesx Ltd.
|Fast backup storage and fast recovery of data (FBSRD)
|Feb 8, 2007
|Aug 9, 2007
|Anand Prahlad
|System and method for performing an image level snapshot and for restoring partial volume data
|Jun 25, 2007
|Oct 25, 2007
|Columbia Data Products, Inc.
|Persistent Snapshot Methods
|Jul 28, 2003
|Feb 19, 2004
|Network Appliance, Inc.
|Multi-protocol storage appliance that provides integrated support for file and block access protocols
|"Network Appliance Ends NAS/SAN War" by Computerwire, http://www.theregister.co.uk/content/63/27368.html; posted on Feb. 10, 2002.
|Akyurek, Sedat, Placing Replicated Data to Reduce Seek Delays, Department of Computer Science, University of Maryland, UMIACS-TR-91-121, CS-TR-2746, Aug. 1991.
|Asante Desktop EN/SC Adapters User's Manual Apr. 1996.
|Asante EN/SC Adapter Family Installation Guide May 1994.
|Bitton, Dina, Disk Shadowing, Proceedings of the 14th VLDB Conference, LA, CA 1988.
|Callaghan B. "NFS Version 3 Protocol Specification", Request for Comments (RFC) 1813 Jun. 1995.
|Celerra File Server Architecture for High Availability, EMC2 where information lives, Aug. 1999, pp. 1-7.
|Chaudhuri, Surajit, et al., Self-Tuning Technology in Microsoft SQL Server, Data Engineering Journal 22, Feb. 1999 pp. 20-27.
|Chutani, Sailesh, et al., The Episode File System, In Proceedings of the USENIX Winter 1992.
|Common Internet File System (CIFS) Version: CIFS-Spec 0.9, Storage Networking Industry Association (SNIA), Draft SNIA CIFS Documentation Work Group Work-in-Progress, Revision Date: Mar. 26, 2001.
|Coyne, Robert A., et al., Storage Systems for National Information Assets, Proc. Supercomputing 92, Minneapolis, Nov. 1992, pp. 626-633.
|David Ritz et al. TR3002 File System Design for a NFS File Server Appliance published by Network Appliance, Inc.
|Fielding et al. (1999) Request for Comments (RFC) 2616, HTTP/1.1.
|Finlayson, Ross S., et al., Log Files: An Extended File Service Exploiting Write-Once Storage Department of Computer Science, Stanford University, Report No. STAN-CS-87-1177, Sep. 1987.
|Gray, Jim, et al., The Recovery Manager of the System R Database Manager, ACM Computing Surveys, (13)2:223-242 1981.
|Hecht, Matthew S., et al. Shadowed Management of Free Disk Pages with a Linked List, ACM Transactions on Database Systems, 8/4, Dec. 1983, pp. 503-514.
|Howard, John H., An Overview of the Andrew File System, Carnegie Mellon University, CMU-ITC-88-062 1988.
|Howard, John, H. et al., Scale and Performance in a Distributed File System, ACM Trans. Computer System, 6(1), Feb. 1988 pp. 51-81.
|Howard, John, H. et al., Scale and Performance in a Distributed File System, Carnegie Mellon University, CMU-ITC-87-068, Aug. 1987.
|International Search Report for International Application No. PCT/US03/23661 filed on Jul. 28, 2003, three pages, International Search Report mailed Mar. 9, 2004.
|Kazar, Michael L., et al., DEcorum File System Architectural Overview, USENIX Summer Conference, Anaheim, California, 1990.
|Kazar, Michael Leon, Synchronization and Caching Issues in the Andrew File System, Carnegie Mellon University, CMU-ITC-88-063.
|Kemper, Alfons, et al., Performance Tuning for SAP R/3, Data Engineering Journal 22, Feb. 1999 pp. 33-40.
|Kent, Jack et al., Optimizing Shadow Recovery Algorithms, IEEE Transactions on Software Engineering, 14( 2): 155-168, Feb. 1988.
|Kistler, et al., Disconnected Operation in the Coda File System, ACM Transactions on Computer Systems, vol. 10, No. 1, Feb. 1992, pp. 3-25.
|Lorie, Raymond, A. Physical Integrity in a Large Segmented Database, ACM Trans. Database Syst., vol. 2, Mar. 1977, pp. 91-104.
|Lu, Y. "Performance Study of iSCSI-Based Storage Subsystems", IEEE Communications Magazine, pp. 76-82, Aug. 2003.
|Maintenance Procedures ND (8C) nd-network disk control Feb. 1985.
|Misc. Reference Manual Pages ND (4P) nd-network disk driver Jul. 26, 1985.
|Ousterhout, John et al., Beating the I/O Bottleneck: A Case for Log-Structured File Systems, Technical Report, Computer Science Division, Electrical Engineering and Computer Sciences, University of California at Berkeley, Oct. 30, 1988.
|Patterson, D., et al., A Case for Redundant Arrays of Inexpensive Disks (RAID),-SIGMOD International Conference on Management of Data, Chicago, IL, USA, Jun. 1-3, 1988, SIGMOD Record (17)3:109-16 Sep. 1988.
|Patterson, D., et al., A Case for Redundant Arrays of Inexpensive Disks (RAID),—SIGMOD International Conference on Management of Data, Chicago, IL, USA, Jun. 1-3, 1988, SIGMOD Record (17)3:109-16 Sep. 1988.
|Patterson, D., et al., A Case for Redundant Arrays of Inexpensive Disks (RAID),-Technical Report, CSD-87-391, Computer Science Division, Electrical Engineering and Computer Sciences, University of California at Berkeley 1987.
|Patterson, D., et al., A Case for Redundant Arrays of Inexpensive Disks (RAID),—Technical Report, CSD-87-391, Computer Science Division, Electrical Engineering and Computer Sciences, University of California at Berkeley 1987.
|Performance Without Compromise: The Virtual Storage Architecture 1997.
|Peterson, Zachary Nathaniel Joseph, Data Placement for Copy-on-Write Using Virtual Contiguity, University of CA, Santa Cruz, Master's Thesis for the Department of Science in Computer Science, Sep. 2002.
|Quinlan, Sean, A Cached WORM File System, Software-Practice and Experience, 21(12):1289-1299 1991.
|Rosenblum, Mendel, et al. The Design and Implementation of a Log-Structured File System Jul. 24, 1991 pp. 1-15.
|Rosenblum, Mendel, et al., The Design and Implementation of a Log-Structured File System, In Proceedings of ACM Transactions on Computer Systems, (10)1:26-52, Feb. 1992.
|Rosenblum, Mendel, et al., The LFS Storage Manager, Computer Science Division, Electrical Engineering And Computer Sciences, Univ. of CA, presented at Summer '90 USENIX Technical Conference, Anaheim, CA Jun. 1990.
|Rosenblum, Mendel, The Design and Implementation of a Log-Structured File System, 1992 pp. 1-93.
|Schiefer, Berni, et al., DB2 Universal Database Performance Tuning, Data Engineering Journal 22, Feb. 1999 pp. 12-19.
|Seltzer, Margo I., et al., Journaling Versus Soft Updates: Asynchronous Meta-Data Protection in File Systems, Proceedings of 200 USENIX Annual Technical Conference, Jun. 18-23, 2000.
|Shasha, Dennis, Tuning Time Series Queries in Finance: Case Studies and Recommendations, Data Engineering Journal 22, Feb. 1999 pp. 41-47.
|Sidebotham, Bob, Volumes: The Andrew File System Data Structuring Primitive, EEUG Conference Proceedings, Manchester, UK, Autumn 1986.
|Stager et al., Network Data Management Protocol, Network Working Group Internet Draft Version 1.7.3, Oct. 24, 1996.
|Subramanian, Muralidhar, et al., Performance Challenges in Object-Relational DBMSs, Data Engineering Journal 22, Feb. 1999 pp. 28-32.
|U.S. Appl. No. 10/126,822, Federwisch et al.
|U.S. Appl. No. 10/777,978, Stephen L. Manley.
|Weikum, Gerhard, et al., Towards Self-Tuning Memory Management for Data Servers, Data Engineering Journal 22, Feb. 1999 pp. 3-11.
|West, Michael, et al. The ITC Distributed File System: Prototype and Experience, Carnegie-Mellon University, Technical Report CMU-ITC-040, Mar. 1985.
|Zayas, Edward R, AFS-3 Programmer's Reference: Architectural Overview, Transarc Corporation, Pittsburgh, PA, 1.0 edition 1991.
|Citing Patent
|Filing date
|Publication date
|Mar 31, 2009
|Jul 10, 2012
|Symantec Corporation
|Method and apparatus for simultaneous comparison of multiple backup sets maintained in a computer system
|Nov 24, 2008
|Nov 20, 2012
|Symantec Corporation
|Method and apparatus for performing granular restoration of data objects from machine images stored on sequential backup media
|Oct 31, 2006
|Apr 16, 2013
|Netapp, Inc.
|System and method for automatic scheduling and policy provisioning for information lifecycle management
|Jul 31, 2009
|Jun 4, 2013
|Microsoft Corporation
|Erasure coded storage aggregation in data centers
|Jul 15, 2008
|Oct 8, 2013
|American Megatrends, Inc.
|Continuous data protection journaling in data storage systems
|Aug 30, 2010
|Nov 19, 2013
|Oracle International Corporation
|Reduced disk space standby
|Oct 20, 2009
|Apr 1, 2014
|International Business Machines Corporation
|Point-in-time copies in a cascade using maps and fdisks
|May 27, 2009
|Apr 22, 2014
|American Megatrends, Inc.
|Continuous data protection of files stored on a remote storage device
|May 17, 2012
|Apr 29, 2014
|International Business Machines Corporation
|Point-in-time copies in a cascade using maps and fdisks
|Aug 30, 2010
|Sep 16, 2014
|Oracle International Corporation
|Controlling data lag in a replicated computer system
|Jun 15, 2011
|Oct 21, 2014
|Oracle International Corporation
|Method for maximizing throughput and minimizing transactions response times on the primary system in the presence of a zero data loss standby replica
|Feb 12, 2008
|Oct 21, 2014
|Oracle International Corporation
|Database system with active standby and nodes
|Jun 3, 2013
|Dec 23, 2014
|Microsoft Corporation
|Erasure coded storage aggregation in data centers
|Sep 30, 2011
|May 5, 2015
|Emc Corporation
|Efficient building of restore list
|Jul 9, 2012
|May 12, 2015
|Symantec Corporation
|Method and apparatus for simultaneous comparison of multiple backup sets maintained in a computer system
|Aug 15, 2011
|May 26, 2015
|Symantec Corporation
|System and method to repair file system metadata
|Dec 27, 2012
|Jun 30, 2015
|Commvault Systems, Inc.
|Restoration of centralized data storage manager, such as data storage manager in a hierarchical data storage system
|Nov 5, 2014
|Aug 11, 2015
|Axcient, Inc.
|Systems and methods for restoring a file
|Sep 25, 2014
|Aug 11, 2015
|International Business Machines Corporation
|Reusing snapshots in a network data management protocol backup operation
|Jun 24, 2013
|Aug 11, 2015
|International Business Machines Corporation
|Recording backup information for backed-up data items in a data item list
|Nov 5, 2013
|Sep 29, 2015
|Oracle International Corporation
|Reduced disk space standby
|Oct 18, 2005
|Oct 6, 2015
|Oia Intellectuals, Inc.
|Wearable capture and communication
|Sep 25, 2013
|Dec 15, 2015
|Axcient, Inc.
|Systems, methods, and media for synthesizing views of file system backups
|Feb 17, 2011
|Jan 12, 2016
|Axcient, Inc.
|Systems and methods for maintaining a virtual failover volume of a target computing system
|Mar 7, 2013
|Mar 22, 2016
|Axcient, Inc.
|Systems and methods for providing efficient and focused visualization of data
|Jun 30, 2011
|Mar 22, 2016
|Amazon Technologies, Inc.
|Shadowing storage gateway
|Dec 28, 2012
|Jun 7, 2016
|Emc Corporation
|Method and system for network user access impersonation for multi-threaded backup applications
|Mar 7, 2013
|Jul 19, 2016
|Axcient, Inc.
|Protection status determinations for computing devices
|Sep 28, 2012
|Oct 4, 2016
|Emc Corporation
|Method and system for enhanced backup database indexing
|Jan 25, 2016
|Jan 31, 2017
|Commvault Systems, Inc.
|Data recovery operations, such as recovery from modified network data management protocol data
|Dec 8, 2014
|Jan 31, 2017
|Axcient, Inc.
|Cloud-based virtual machines and offices
|Apr 16, 2013
|Mar 21, 2017
|Microsoft Technology Licensing, Llc
|Local erasure codes for data storage
|Sep 28, 2012
|May 9, 2017
|EMC IP Holding Company LLP
|Method and system for file recovery using enhanced backup database indexing
|May 7, 2013
|Jul 11, 2017
|Axcient, Inc.
|Cloud storage using Merkle trees
|Oct 18, 2005
|Sep 6, 2007
|Oia Intellectuals, Inc.
|Wearable capture and communication
|Feb 12, 2008
|Sep 11, 2008
|Oracle International Corporation
|Database system with active standby and nodes
|Oct 31, 2008
|Mar 5, 2009
|Jakob Holger
|Method and system for function-specific time-configurable replication of data manipulating functions
|May 27, 2009
|Jan 21, 2010
|Paresh Chatterjee
|Continuous Data Protection of Files Stored on a Remote Storage Device
|Jul 9, 2009
|Mar 18, 2010
|Liang-Chi Chen
|Data backup system and method thereof
|Apr 3, 2009
|Oct 7, 2010
|Microsoft Corporation
|Restoration of a system from a set of full and partial delta system snapshots across a distributed system
|Nov 29, 2009
|Dec 30, 2010
|Pi-Hsi Tsai
|Data backup apparatus and data backup method
|Jul 31, 2009
|Feb 3, 2011
|Microsoft Corporation
|Erasure Coded Storage Aggregation in Data Centers
|Oct 20, 2009
|Aug 25, 2011
|International Business Machines Corporation
|Flashcopy handling
|Aug 30, 2010
|Mar 1, 2012
|Oracle International Corporation
|Controlling Data Lag in a Replicated Computer System
|Apr 8, 2011
|Oct 11, 2012
|International Business Machines Corporation
|Reusing Snapshots in a Network Data Management Protocol Backup Operation
|Jun 30, 2011
|Jan 3, 2013
|Sorenson Iii James Christopher
|Shadowing Storage Gateway
|Jun 3, 2013
|Oct 17, 2013
|Microsoft Corporation
|Erasure coded storage aggregation in data centers
|Dec 27, 2012
|Jul 3, 2014
|Commvault Systems, Inc.
|Restoration of centralized data storage manager, such as data storage manager in a hierarchical data storage system
|May 9, 2014
|Sep 4, 2014
|Salesforce.Com, Inc.
|Methods and systems for backing up a search index
|Oct 16, 2014
|Sep 3, 2015
|Synology Incorporated
|Network device and operating method thereof
|Jan 30, 2015
|Aug 4, 2016
|Splunk Inc.
|Systems and Methods for Distributing Indexer Configurations
|U.S. Classification
|707/654, 707/674, 707/956
|International Classification
|Cooperative Classification
|European Classification
|Jul 22, 2004
Owner name: NETWORK APPLIANCE, INC., CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:OWARA, SHANE S.;LAM, SAHN;SIGNING DATES FROM 20040621 TO20040624;REEL/FRAME:015593/0512
|Mar 9, 2011
Owner name: NETAPP, INC., CALIFORNIA
Free format text: CHANGE OF NAME;ASSIGNOR:NETWORK APPLIANCE, INC.;REEL/FRAME:025924/0990
Effective date: 20080317
|Mar 20, 2012
|Certificate of correction
|Dec 22, 2014
|Fee payment
Year of fee payment: 4


text:
Here are some great ways you can connect with others at Woodland Hills:
Discussion Tables
Twice a year (fall and late winter/early spring) our church plans special "connecting" sermon series designed to incorporate round table discussion groups over a six week period. Each table has a facilitator and discussion questions to review with the group. This is an easy way to meet others as you share, learn and grow in community with each other. For more information, contact Becky at firstname.lastname@example.com or 651-287-2096.
On Tuesday nights we offer learning opportunities designed to help you meet people and promote personal growth and Kingdom values. Most classes start at 6:30 pm and end at 8:30 and depending on the course, run from 6 weeks to 13 weeks in length. Typically we have several classes offered in the fall session (starts in September) and in the winter/spring session (starts in January). Childcare is provided. We post updates about Cultivate classes to our blog; check them out here.
Arts Collective
Designed to foster the arts within Woodland Hills and beyond. Getting involved in this community is a great way to meet others who are passionate about creative expression. Learn more.
Resonate is made up of 18-24 year olds. In addition to weekly mentoring there are weekly house groups and regular gatherings focused on worship, outreach, and hanging out. Learn more.
Men's Bible Study
The mission of the Men's Bible Study is to connect with other men from our community in study, prayer and fellowship. Learn more.
Refuge Groups
The Refuge is a collection of people who are broken in different areas. Together we meet the grace of God through music, dinner and issue-specific support groups. Learn more.
One of the best ways to get to know people is by volunteering together! If you're interested in helping out in one of our ministry areas and meeting some people from church at the same time, take a look at these options.


text:
As some of you may know, I'm a little into sports, and as a fan of the Mets, Nets, and Jets, things are looking surprisingly good. The New York Mets own the best record in baseball, at least for now (probably won't last…or will it?). The Brooklyn Nets beat the Atlanta Hawks last night to tie up the series at 2 games apiece against the best team in the Eastern Conference (though their chances of advancing are still small). And with the offseason moves the Jets have made, they are, in my opinion, just a decent quarterback away from being potential playoff contenders (a tall order, I know).
So, with that said, I'd like to request/plead for sports-related queries. If you have a novel or nonfiction work on sports, I would love to take a look at your material.
Fair warning: books about sports are tough to sell for a very simple reason: the market. Publishers won't buy a book that they cannot definitively say will appear to X number of readers or Y demographic. It's no secret that women tend to read more than men, and it also isn't a secret that men tend to be more interested in sports than women. Except that—and this is a secret—neither are true. Well okay, maybe the former is, but I have male friends who read voraciously and female friends who bleed red and blue (NY Giants, NY Rangers).
Sports are universal. There is a market. And when books about sports work, they really, really work. In fact, they often work themselves onto the big screen. Yeah, I'm looking at you FRIDAY NIGHT LIGHTS by H.G. Bissinger, MONEYBALL and THE BLIND SIDE by Michael Lewis, and SEABISCUIT by Laura Hillenbrand. Oh, and novels can work too—case in point: THE ART OF FIELDING, Chad Harbach's wildly successful debut and one of my favorite novels in recent years.
Notice any commonalities between any of those books mentioned above? They're great sports stories, told beautifully, and aren't really about sports. Query me if you think your book fits the bill.


text:
ChanelGold Toned Chain Link "CC" Rhinestone Embellished Pendant Necklace
Statement necklace has spring clasp closure.
Made in: France Color: Gold Fabric Content: Metal and Rhinestone Condition: Good. Very faint hairline scratches. Slight discoloration.
Measurements & Sizing
- Total Length: 20.5"
- Necklace Drop: 9.75"
- PendantHeight: 1"
Returns & Shipping
Luxury Garage Sale guarantees that all items sold on our site are 100% authentic. All products we accept are inspected for brand-specific guidelines and are put through a rigorous authenticity test process.
LGS is not affiliated with Chanel. We guarantee this Chanel item to be authentic. Chanel® is a registered trademark of Chanel.
Item #: 104601