text:
Cleaning Services
- Home Watch
- Vacation Rentals
- Linen Rental
- Home Cleaning
- Move In / Out
- Window Washing
- Carpet Cleaning
- Power Washing
- Our Guarantee
- Contact Us
Our Work
Quick Pics
North Beach Towers
Grande Dunes
Oceans One
Oceanfront Homes
Luxury Homes
Bed Linen Rental
Bath Linen Rental
Cleaning Service Rates
Please note that pricing is based on numerous factors. The rates shown below are a general guideline to follow. In most cases we are able to provide accurate pricing over the phone at (843) 400-0323. If you would like to schedule a free on-site estimate please fill out this short form.
House Cleaning and Vacation Rental Cleaning
Rates for house cleaning and vacation rental cleaning are based on size and condition of property. Typical questions we would like to know before quoting your cleaning job are:
Approximate square footage, if known?
How many levels?
Number of bedrooms and bathrooms?
How many ceiling fans/ how high?
Do you have/allow pets?
Other areas to be cleaned - garage, basement, attic ?
Frequency of cleanings?
Condo cleaning in Myrtle Beach starts at $60.00 per unit (efficiency) for a top quality cleaning
Thorough and detailed single family home cleaning services. Please call or email for rates.
Window Cleaning
Online Window Cleaning Estimator
Pressure Washing
Houses - Starting at $125.00 (single story) - $195.00 (two story)
Driveways and sidewalks - $0.10 per square foot
Carpet Cleaning
Hot water extraction - Please call or email for current rates
Stairs / steps - $1.50 each
Scotchgard™ treatment - $0.10 per square foot
Most couches - $75.00
Gutter Cleaning
Please request an estimate for gutter cleaning


text:
Status: Closed
Know more about this business than we do? Cool! Please submit any corrections or missing details you may have.Help us make it right
Treatment Services Northwest is located at the address 9370 Sw Greenburg Rd in Portland, Oregon 97223. They can be contacted via phone at (503) 246-5238 for pricing, hours and directions. Treatment Services Northwest specializes in Addiction Counseling, Sleep Disorders, Depression.
Treatment Services Northwest has an annual sales volume of 2M – 4,999,999. .
Treatment Services Northwest...
Northwest Treatment Services is located at the address 9370 Sw Grnbrg rd Ste 101 in Portland, Oregon 97223. They can be contacted via phone at (503) 246-7030 for pricing, hours and directions.
Northwest Treatment Services has an annual sales volume of 2M – 4,999,999. .
For maps and directions to Northwest Treatment Services view the map to the right. For reviews of Northwest Treatment...
Treatment Services Northwest is a Psychologist office located in Portland, OR. A psychologist focuses on the evaluation, prevention, diagnosis, and treatment of mental, emotional and behavioral health issues. A clinical psychologist uses psychotherapy and other counseling skills to improve emotional and mental health.
Please call Treatment Services Northwest at (503) 246-5238 to schedule an...
Be the first OregonLive.com user to add photos or videos of Treatment Services Northwest
Dr. Cotton is Director of NW Treatment on SW Greenburg Road. He is never available to talk to. He is insulated in his hidden office. The counselors he employs are inept, rigid and always disagree with what you have to say. The costs are huge. If I didn't have to, I would never go back.
Cotton Chris Ph.D. is located at 9370 SW Greenburg Rd Ste 601, Portland, OR. This location is in the Tigard Neighborhood Area 4 neighborhood. This business specializes in Mental Health.
Treatment Services Northwest can be found at Sw Greenburg Rd 9370. The following is offered: Mental Health Clinics. The entry is present with us since Sep 8, 2010 and was last updated on Nov 14, 2013. In Portland there are 28 other Mental Health Clinics. An overview can be found here.

subset: commoncrawl


text:
Police have charged a 25-year-old man with two counts of breaching a domestic violence order, as well as three counts of assault occasioning bodily harm (domestic violence) following a disturbance at a Burpengary East address on Saturday night, March 19.
It will be alleged a man was at a Margaret Street address when an altercation occurred with a woman. Police will allege the woman was assaulted and when a 36-year-old man came to her assistance, he too was assaulted.
The man then left the residence only to return around 7.30pm with two other men. Another disturbance occurred inside the property during which time the woman and the 36-year-old man were again allegedly assaulted. Police were called and the trio fled with one of the men running off into nearby bushland and the other two driving off in a vehicle.
A police dog and his handler tracked the man and he was taken into custody after being followed into bushland. The man, who was hiding in the dark in dense bush, was bitten by the police dog when the man allegedly grabbed the dog around the neck and throat. The man received medical treatment for dog bite injuries at a local hospital before being transported to the watchhouse where he was denied police bail before his court appearance this morning.
Following police investigations two other men, aged 23 and 38, were arrested and charged last night with one count each of assault occasioning bodily harm whilst in company in relation to the alleged assault of the 36-year-old man.
A 25-year-old man- who remains in police custody and will be appearing in the Redcliffe Magistrates Court this morning- has been charged with a total of seven offences including two counts of assault occasioning bodily harm (domestic violence) whilst in company, one count of assault occasioning bodily harm (domestic violence), two counts of contravening a domestic violence order, and one count each of wilful damage and obstruct police.
The 22-year-old woman and the 36-year-old man, both from Burpengary, sustained bruising and swelling injuries consistent with allegedly being punched about the head and face.
All three offenders are known to each other and come from Deception Bay.


text:
- Bronx Banter - http://www.bronxbanterblog.com -
Beat of the Day
Posted By Alex Belth On June 30, 2011 @ 11:18 am In Arts and Culture,Beat of the Day,Funk | Comments Disabled
Article printed from Bronx Banter: http://www.bronxbanterblog.com
URL to article: http://www.bronxbanterblog.com/2011/06/30/beat-of-the-day-360/
URLs in this post:
[1] Image: http://bronxbanter.arneson.name/wordpress/wp-content/uploads/2011/06/jj.jpg
Click here to print.
Copyright © 2011 Bronx Banter. All rights reserved.


text:
|Thread Tools
4 to 4.5 was really hard for my son, and for me. He was an emtional mess, yelled, fits....then we had a great few months and now he is acting up again. Almost every issue revolves around his sister who is 3. They fight a lot. He teases her all. day. long. If she hits him for whatever reason, he will whack her back..hard.
His sense of revenge is incredible. It is the same with friends he is playing with. If they are play fighting and he accidently gets hit, he will hit back hard.
I am not sure what I should do when they are fighting. I am trying the whole "if you hit, you sit" but it isn't working. I have tried talking to them about using their words, but they get so worked up so quickly.
The other thing i am really struggling with is his attitude. Just today the kids were outside playing and I was straightening up inside. I heard crying and when I went outside I found out that my daughter hit my son, so he hit her back really hard. I calmly explained that we had to come back inside and sit on the couch to calm down. My son started laughing, shrugging, saying things like "I don't care" "Sure" "Thats fine" with a smile on his face. It makes me so mad. I am also 7 months pregnant and emotional, but he is bringing me to the point of tears everyday. And the past few days he makes me so mad that I start yelling.
Today I lost it. I yelled. I took his favorite toys away and told him no tv for a week. WHich is funny because they only watch 2 hours of tv a week, and the next tv day is Monday. I have never done timeouts, taking toys away, and rarely yell. But I feel I have no consequences for the hitting, yelling, or attitude.
What do you when your 4.5 year old gives you attitude, is mean, bullies his sister and is hitting?
Its really good the way you tend to act calmly in fights...but having to sit on a couch as a consequence to hitting doesnt seem like a punishment at all. Why dont you like time outs? Or taking toys away(It isnt violent and there is an actual consequence)? In real life if you do something like hit somebody else there are major consequences, and i believe as a parent we are supposed to be teaching our children how to function in the real world, right?
There is a great book called Siblings Without Rivalry that would probably help you, the same authors also wrote How To Talk So Kids Will Listen and Listen so Kids Will Talk, another good read. First off I would not let them play outside alone since things seem to escalate quickly. Second do not take sides(although it doesn't sound like you did) b/c that creates rivalry even if one person does seem to be "at fault". Kids this age are still impulsive when angry and just need some time before the phase passes. You can help them communicate and help them identify their feelings with words b/c consequences are not going to change how they get along or change their personalities. Making sure everyone is well rested and fed when interacting helps greatly also, so does one on one time with mom and dad w/out the sibling:)
This is a way to work with fighting also http://www.psychologytoday.com/blog/peacemeal/201010/3-steps-transform-sibling-conflict-sibling-camaraderie
Cathy mom to 13 y/o DD, 10 y/o DD, 7 y/o DS
|Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page
|Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off
Online Users: 10,446
|11 members and 10,435 guests
|30Second , jamesmorrow , katelove , mantorosaline4 , NaturallyKait , RollerCoasterMama , scaramouche131
|Most users ever online was 449,755, 06-25-2014 at 12:21 PM.


text:
Can Starbucks Break Out?
With shares of Starbucks (NASDAQ:SBUX) trading around $58, is SBUX an OUTPERFORM, WAIT AND SEE or STAY AWAY? Let's analyze the stock with the relevant sections of our CHEAT SHEET investing framework:
T = Trends for a Stock's Movement
Starbucks is a roaster, marketer and retailer of coffee operating in 60 countries. The company purchases and roasts coffees that it sells, along with handcrafted coffee, tea and other beverages and a variety of fresh food items, through company-operated stores. It also sells a variety of coffee and tea products and licenses its trademarks through other channels, such as licensed stores and national food service accounts. Through its stores and other channels, Starbucks is able to provide the convenient items that many consumers around the world are demanding.
T = Technicals on the Stock Chart are Strong
Starbucks has seen a long-term uptrend over the last couple of decades. Like many stocks, it did suffer during bad economic times but has recovered. Starbucks stock is now trading near all-time high prices. Analyzing the price trend and its strength can be done using key simple moving averages. What are the key moving averages? The 50-day (pink), 100-day (blue), and 200-day (yellow) simple moving averages. As seen in the daily price chart below, Starbucks is trading below all of its rising key averages which signal bullish price action in the near-term.
(Source: Thinkorswim)
Taking a look at the implied volatility (red) and implied volatility skew levels of Starbucks options may help determine if investors are bullish, neutral, or bearish.
What does this mean? This means that investors or traders are buying a very significant amount of call and put options contracts, as compared to the last 30 and 90 trading days.
As of today, there is an average demand from call buyers or sellers and low demand by put buyers or high demand by put sellers, all neutral to bullish over the next two months. To summarize, investors are buying a very significant amount of call and put option contracts and are leaning neutral to bullish over the next two months.
E = Earnings Are Increasing Quarter-Over-Quarter
Rising stock prices are often strongly correlated with rising earnings and revenue growth rates. Also, the last four quarterly earnings announcement reactions help gauge investor sentiment on Starbucks's stock. What do the last four quarterly earnings and revenue growth (Y-O-Y) figures for Starbucks look like and more importantly, how did the markets like these numbers?
Starbucks has generally seen increasing earnings and revenue over the last four quarters. From these figures, the markets have been very pleased with Starbucks's last two earnings announcements.
P = Poor Relative Performance Versus Peers and Sector
How has Starbucks stock done relative to its peers, Dunkin Brands (NASDAQ:DNKN), McDonald's (NYSE:MCD), Green Mountain Coffee Roasters (NASDAQ:GMCR), and sector?
Starbucks has been a relative underperformer year-to-date.
Starbucks provides products that have received a heavy rise in demand, around the world, over the last few years. The demand for its products has consistently increased earnings and revenue which has led investors to bid the stock near all-time high prices. However, Starbucks has been underperforming its strong peers and sector. Look for Starbucks stock to OUTPERFORM.
Copyright Wall St. Cheat Street. All rights reserved.


text:
Listed here are guides to finding information in particular formats.
Finding the full text of an article depends on what information you currently have:
Here is a list of audio books on cassette. (Note that this lists includes interviews and other non-book materials.) Equipment for listening to these cassettes is available for use in Newman Library. Check out headphones from the Circulation Desk.
The libraries subscribe to a number of databases that provide audio resources: streaming and downloadable music, spoken word recordings, and text to speech reading of article content.
Bibliographies are lists of sources used to write a publication. You may want to find a bibliography on a topic because
A book review is an article that is published in a newspaper, magazine, or scholarly work that describes and evaluates a book. Keep in mind while searching for reviews of a book that many books are published each year, only a small fraction of them are reviewed. Reviews are written for different purposes and they will vary in terms of content and depth of evaluation. Since reviews are printed in many different kinds of publications, you may need to search several sources.
Business, company, and industry information, including scholarly articles, financials, and reports can be found in a number of online databases and print reference sources.
In response to questions regarding the availability of congressional hearings online through GPO Access, GPO staff have prepared the following report on the current status and future plans at GPO for providing hearings electronically.
All congressional hearings are not centrally available on GPO servers at this time. This is due in large part to the fact that committees from the House and Senate do not provide GPO with all hearings to place on GPO servers. Also, there is frequently a delay between the receipt of hearings in hard copy and their subsequent availability online through GPO Access. Below, we outline five ways for users to find congressional hearings on or through GPO Access, and also present some enhancements currently being developed to further improve the ability of customers to access congressional hearings.
The Virginia Tech Libraries provide access to over 700 databases where you can search for articles (and more) on your research topic. Select the appropriate database(s) using these strategies:
The Virginia Tech Libraries has subscribed to a number of ebook packages, giving us access to hundreds of thousands of books online. (Since these are paid subscriptions, be sure to use Off Campus Sign In before attempting to connect to an ebook outside the campus network.) Most of our ebooks can be found through Summon —that is the best place to start a search for ebooks. All allow reading the ebook through the browser, and many allow downloading for offline reading or transfer to a mobile device or dedicated ebook reader. Most of our ebooks are PDFs, and thus require no additional software to view. See each entry for formats and restrictions.
The University Libraries has a number of foreign language learning materials, including books, CDs, and software. In addition, our collections include international literature titles in the original language and translated into English. We also offer titles on American Sign Language.
The Journal Title Database includes four types of links:
Do not attempt to use this resource as you might use a general Internet search engine. It does NOT provide subject or topic keyword access to articles or journal contents.
For example, a search on TV and violence will not supply articles or citations on the subject of TV and violence.
Manuscripts are a type of primary source material. They provide first-hand testimony or direct evidence of events and experiences—both public and personal, and are created by witnesses or recorders who experienced the events or conditions documented. While primary sources may either be published or unpublished, manuscripts are unpublished primary sources. Manuscripts include diaries, journals, letters, correspondence, family papers, scrapbooks, business or other institutional records, legal documents, architectural drawings, oral histories, or photographs.
The map collection at Virginia Tech, the largest in the State outside of the Washington, D.C. area, is divided into two primary collections. Original historic maps are located in Special Collections on the first floor of Newman Library. A Guide to the Map Collection in the Special Collections Department, inventories the collection. The majority of maps are housed in the maps section on the first floor of Newman Library.
Topographic coverage of the U.S. and foreign areas produced by the U.S. Geological Survey, Defense Mapping Agency, Australian National Mapping Agency and others form the backbone of the collection. Other important map series include maps of the Census Bureau, CIA, National Geographic Society, National Forest Service and the National Park Service. The collection is rounded out by an extensive collection of state and city road maps, raised relief maps, reproductions of historic maps and views, and nautical and aeronautical charts.
Several digital mapping resources are available. Software and data for map creation, such as ArcView (for Arc Info formatted data files), LandView (for TigerLine and other U.S. government data files) and Map Expert (for roadmaps of the U.S.), are located in Port, Torgersen 3300. Additional digital mapping resources can be found on the Maps, GIS, and cartographic data page.
Microforms are non-circulating photographic reductions of print materials produced in the interest of saving space,obtaining or preserving rare or heavily used materials, or providing duplicate copies. Newman Library has a microforms collection numbering near 6 million items, including prints of books, magazines, newspapers, and engineering and technical reports.
A movie review is an article that is published in a newspaper, magazine, or scholarly work that describes and evaluates a movie. Reviews are typically written by journalists giving their opinion of the movie. Some reviews include score (4 out of 5 stars) or recommendations (thumbs up). Since reviews are printed in many different kinds of publications, you may need to search several sources.
A movie criticism is written by a scholar or expert in film studies to discuss the movie within a historical, social, political, or theoretical context. It differs from the opinion or recommendation that a movie review provides in terms of length, content and focus. Criticisms can be found in cinema studies journals as well as discipline-specific sources, depending on the plot or themes of the movie.
The Virginia Tech Libraries own thousands of movies on DVD, VHS, and laser disc. Most are located at the Newman circulation desk; a few are in the branch libraries. Instructors can book a movie in advance to show in class.
While the University Libraries at Virginia Tech are primarily focused on serving the research needs of the university community, we still provide a number of pleasure reading books and popular literature.
Speeches and other oral communications on topics such as politics, history, science and business can be vital sources for research. Often you will be looking for transcripts of these speeches as opposed to recordings of the speech itself. The library also has sources for transcripts of radio and television programs, particularly news programs. When accessing text of speeches or programs, be sure to check if the text is a summary, except or the full transcript of the speech or program.
We have several sources for streaming videos that can be shown in class or assigned for viewing by students anywhere.
Tests and measures can include any type of instruments (quantitative or qualitative) that can be used in research. We have multiple search tools, databases, books, and reference materials that can aid you in finding the best instrument for your research.
Each semester, the library receives a list from the bookstore of all textbooks and required readings for that term. We use this list to identify any copies of those books we have in our collection and place those books on textbook reserve. (We do not own every textbook, however.) Faculty may also place personal copies of textbooks on reserve.
A dissertation (also called thesis or disquisition) is a document that presents the author's research and findings and is submitted in support of candidature for a degree or professional qualification. Theses have been published at Virginia Tech since 1903. Theses and dissertations have been submitted electronically since 1996, with most of these freely available online. The Virginia Tech Libraries owns few theses or dissertations published at other colleges and universities; expect to obtain most of these documents via interlibrary loan. In addition to the search tools referenced below, most subject specific databases (like PsycInfo, Sociological Abstracts, and Engineering Village) will provide citations for theses and dissertations in their covered fields.


text:
Carolina Women's 6 in. Aluminum Comfort Oxford
Carolina Women's 6in. Aluminum Toe Oxford. Designed so they reduce weight, promote flexibility in the right places, and manage moisture. They were designed and constructed with your ultimate comfort in mind.
Price: $113.99
- Medium Brown Leather Upper
- Aluminum Safety Toe Cap
- Leather/Mesh Lining
- Removable AG7™ PU Insole with Pillow Cushion™ Footbed
- VA Midsole
- Non-Metallic Shank
- Electrostatic Dissipative
- Cement Construction
- Slip Resisting Rubber Sole
- Safety Toe Meets or Exceeds ASTM F/ANSI Standards
Sorry, this item is sold out.
View other Carolina Women's Safety Toe Shoes. View other Carolina products.
Or use the search bar above to look for items.


text:
SSC CGL Exam 2017: Complete Guide eBook
Rs. 199
Quantitative Aptitude Formula eBook
Rs. 99
General English for Banking e-Book
Current Affairs July 2017 eBook
Rs. 65
Current Affairs July 2017 eBook Hindi
Mathematics Formula & Shortcut ebook Hindi
Rs. 59
Mathematics Formula & Shortcuts eBook
CBSE Class 12th PCB Solved Practice Paper 2017 Set...
Rs. 108
CBSE Class 12th PCM Solved Practice Paper 2017 Set...
CBSE Class 10th Science & Math Solved Practice Pap...
Rs. 145
IAS Prelims 2017 A Complete Guide eBook
Rs. 249
Current Affairs Supplement (April to June) 2017 Pa...
Rs. 155
General Knowledge eBook July 2017
Rs. 15
General Knowledge eBook July 2017 Hindi
IAS Prelims 2017 Current Affairs ebook
Indian History Question Bank ebook
Rs. 149
NCERT Class 12 PCM Solutions
Rs. 55
SSC CGL General Awareness Previous Year Question B...
Rs. 30
CBSE Class 12th PCM Solved Guess Papers 2017: eBoo...
© 2017 Josh Shop. All Rights Reserved.


text:
Bergson Art Print Poster
Henri Bergson
Henri Bergson (1859 - 1941) was born the same year as the publication of Darwin's "On the Origin of Species", to father of Polish Jewish ancestry, an accomplished musician, and a mother of English and Irish Jewish ancestry. The family immigrated to France when Bergson was a child.
Brought up with a religious education, as a teenager Bergson experienced a moral dilmena tied to his discovery of the theory of evolution positing that man had descended from the ape, totally at variance with his schooling.
As a student Bergson was adept in both science and mathematics, and published his first mathematical work before he was 20 - a solution to a mathematical problem which Pascal claimed to have solved, but left unpublished.
Bergson opted for a career in the humanities, and settled into the quiet life of professor But he was to become a major force in 20th century philosophy as it attempted to come to grips with a lava flow of new scientific discoveries.
Bergson's first major work, Time and Free Will, argued that misconceptions about freedom and will had arisen from the erroneous tendency to describe them in spatial terms. Bergson posited that real life is not experienced as a succession of demarcated conscious states traveling along an imaginary path, but as a continuous flow. There was a difference between the concept of time, and the actual experience of time. While the physicist could observe objects and events in succession, time presents itself to consciousness as duration – an endless flow, which cannot be simply represented as mathematics. Time in reality was experienced as duration and apprehended through intuition, a continuous flow, not something perceived through separate operations of sensory instinct and intellect.
An Introduction to Metaphysics took this a step further, positing that it was not analysis, but intuition – the direct apprehension of process – which revealed the real world.
Matter and Memory (1896) and Creative Evolution attempted to integrate biology with a theory of consciousness. Bergson posited that intuition was deeper and more fundamental than intellect, and his work become the spearhead of challenges to the mechanistic view of the universe. Features of the theory of relativity, and theories of the mind posited by modern science have been seen as foreshadowed in Bergson's work.
"In reality, the past is preserved by itself automatically. In its entirety, probably, it follows us at every instant; all that we have felt, thought and willed from our earliest infancy is there, leaning over the present which is about to join it, pressing against the portals of consciousness that would fain leave it outside."
(from Creative Evolution)
"In reality, the past is preserved by itself automatically. In its entirety, probably, it follows us at every instant; all that we have felt, thought and willed from our earliest infancy is there, leaning over the present which is about to join it, pressing against the portals of consciousness that would fain leave it outside."
(from Creative Evolution)
In Creative Evolution Bergson elaborated his concept of élan vital, "creative impulse" or "living energy", an immaterial force, whose existence cannot be verified scientifically, but which provides the "vital impulse" that continuously shapes all life.
Bergson's work, in particular Creative Evolution, earned him a place on the list of books banned by the Catholic Church.
Bergson's final major work, The Two Sources of Morality and Religion, was not only a finalized setting out of his philosophy, but also reflected concerns about emergent nationalist racist policies, and the changed nature of warfare.
Although Bergson's scientific and philosophical investigations had prompted him to abandon the practices of his religion, after the German occupation of France Bergson, a Jew, chose to stand with the persecuted and register himself as a Jew. He declined the Vichy government's offer to exempt him from their anti-Semitic laws, renounced the many honors and posts accorded to him, and reaffirmed his core values. Bergson died in 1941.


text:
TMG-L ArchivesArchiver > TMG > 2000-01 > 0947554941
From: Allen Mellen <>
Subject: Re: TMG-L: Sort date for Eudora?
Date: Mon, 10 Jan 2000 20:42:21 -0500
AMike Spathaky wrote:
>I need a sort date as well as a true date for Eudora.
>I take the digest version of this List and sometimes find two or three
>volumes in my mailbox which download in random order, so the first thing I
>do is clik the date-sort facility on the folder. Today this reasulted in
>two volumes disappearing.
>Volume 894 was fine - dated 09/01/00 (would be 01/09/00 in American style)
>Volume 895 wasd dated 31/12/70 (but the header showed 31/12/69)
>Volume 896 was dated 01/01/70
>Volume 897 was dated OK again as 09/01/00
>Looking back I see there was a similar hiccough with Volumes 876 and 877 on
>03/01/00, although everything seems to have gone swimmingly on the nig
>night itself.
Try sorting on Subject. It worked for me!
Allen Mellen -- New York, NY
Fax: (212) 202-6007
- ---------------------------------------------------------------------------
TMG-L The Internet Mailing List for The Master Genealogist
To unsubscribe: Send an e-mail to with 'UNSUBSCRIBE TMG-L'
or... if you get the digest version: 'UNSUBSCRIBE TMG-L-DIGEST'
- ---------------------------------------------------------------------------
This thread:
|Re: TMG-L: Sort date for Eudora? by Allen Mellen <>


text:
|Publication number
|Publication type
|Application number
|Publication date
|Apr 6, 2010
|Filing date
|Apr 14, 2005
|Priority date
|Apr 14, 2005
|Fee status
|Also published as
|Publication number
|Dimitris Achlioptas, Sean B House
|Original Assignee
|Microsoft Corporation
|Export Citation
|BiBTeX, EndNote, RefMan
|Patent Citations (41), Non-Patent Citations (8), Referenced by (13), Classifications (13), Legal Events (3)
|External Links: USPTO, USPTO Assignment, Espacenet
This invention is related to data packet distribution systems and methods, and more specifically, to a system where load balancing is maintained among hosts as the data packets are distributed to the hosts.
Global communications networks such as the Internet are now ubiquitous with an increasingly larger number of private and corporate users dependent on such networks for communications and data transfer operations. As communications security improves, more data can be expected to traverse the global communications data backbone between sources and destinations (typically, server hosts) placing increasing demands on those entities that handle and store data. Such increased demands are typically addressed at the destination by adding more switching devices and servers to handle the load. However, this can be an expensive proposition in terms of hardware, software, setup, and administration.
Network load-balancers provide client access to services hosted by a collection of servers (herein known as "hosts"). Clients connect to a load-balancer, which transparently (to the clients) forwards them to a host according to a set of rules. This general load balancing context includes the following: packets form sequences, called sessions; sessions should be allocated among the available hosts in a "balanced" manner; and, every packet of each session should always be directed to the same host, so long as the host is alive (the latter is known as "session affinity").
This problem is most often solved through the use of a single monolithic load-balancer that monitors the status (liveness/load) of the hosts and maintains state in the form of a table of all active sessions. When a new session arrives, the load-balancer selects the least-loaded host that is available and assigns the session to that host. In order to provide session affinity, the load-balancer must "remember" this assignment (routing) decision by adding an entry to its session table. When subsequent packets for this session arrive at the load-balancer, a single table lookup determines the correct host. However, an individual load-balancer can be both a single point of failure and a bottleneck; the size of its session table (and thereby the amount of state maintained) grows with increased throughput and routing decisions for existing session traffic require a state lookup (one per packet). Circumventing these limitations requires multiple load-balancers working in tandem (scale-out), and/or larger, more powerful load-balancers (scale-up). However, scaling-out these load balancing devices is complicated, due most notably to the need of maintaining consistent state among the load-balancers. Likewise, scaling them up is expensive, since cost versus throughput in fixed hardware is non-linear (e.g., a load-balancer capable of twice the throughput costs significantly more than twice the price).
The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not an extensive overview of the invention. It is not intended to identify key/critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later.
The subject invention provides a software architecture for stateless, affinity-preserving load balancing among hosts. The invention mitigates (and/or circumvents) the single largest obstacle to scale-out (state maintenance), and allows significantly more flexible scaling options than those of a hardware load-balancer. Further, unlike a hardware load-balancer whose per-packet processing cost increases with the size of the system (due to the increased cost of state lookup as the session table grows), this invention exhibits a substantially constant per-packet processing cost by way of its statelessness. This facilitates maintaining constant steady-state per-host bandwidth as system size increases. Moreover, the load balancing scheme of the subject invention responds rapidly to changing load/traffic conditions in the system.
The scale-out of load balancing services per the present invention can be achieved utilizing one or more replicated packet forwarders operating in tandem. Generally, the replicated forwarders share part of the task of routing incoming packets, but neither maintains nor shares state with its peers. To maximize scale-out, the forwarders receive incoming packets from an external device whose purpose it is to distribute the incoming stream among the forwarders by sending each packet to one forwarder using methods such as round-robin, random, layer-3/4 hashing, and so forth. It is to be understood that this "first level" packet distribution does not need to have the property that packets from the same session go to the same forwarder. Any degree of preserving session affinity achieved by this first level device is leveraged by the invention and can significantly increase the achievable scale-out. If no such device were available, the forwarders would be required to distribute the traffic among each other through an appropriate mechanism between forwarders.
The forwarders employ one or more routing functions (e.g., hash functions) that assign incoming packets to available hosts that service the packets. Given the same information about the state/availability of the hosts in the system, the forwarders generate identical routing functions, independently, based on a static pre-agreed upon protocol. This sharing of current "system conditions" is effectively the only shared information among the forwarders. The input to a routing function is an identifier that uniquely identifies the session to which the packet belongs (herein known as a "session identifier" or "session ID"), and the output is the host to which the packet belongs. The system manages changes in the system (e.g., in the number of available hosts) that affect packet routing by creating new, additional, routing functions on each forwarder. One feature of the invention is that when such updates are made, the currently used hash functions are generally, not replaced by the new ones. Instead, the new routing functions serve to distribute future load according to the revised state of the hosts, while the old routing functions serve to maintain session affinity for the sessions already present at the time each new hash function is added. To prevent the accumulation of hash functions over time, when the system achieves steady state load distribution employing the new routing functions, efficient protocols are utilized between the forwarders and the hosts to determine which previous routing functions are no longer required.
To the accomplishment of the foregoing and related ends, certain illustrative aspects of the invention are described herein in connection with the following description and the annexed drawings. These aspects are indicative, however, of but a few of the various ways in which the principles of the invention can be employed and the invention is intended to include all such aspects and their equivalents. Other advantages and novel features of the invention will become apparent from the following detailed description of the invention when considered in conjunction with the drawings.
The invention is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the invention. It may be evident, however, that the invention can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing the invention.
As used in this application, the terms "component" and "system" are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a server and the server can be a component. One or more components can reside within a process and/or thread of execution, and a component can be localized on one computer and/or distributed between two or more computers.
Referring now to
It is noted that a session is a logical series of requests and responses between two network entities that can span several protocols, many individual connections, and can last an indeterminate length of time. Some common session types include TCP (Transmission Control Protocol), FTP (File Transfer Protocol), SSL (Secure Socket Layer), IPSec(IP Security)/L2TP(Layer 2 Tunneling Protocol), PPTP (Point-to-Point Tunneling Protocol), RDP (Remote Desktop Protocol), etc. The characterization of a session for most protocols is well defined; there is a clear beginning and end to each session, and an associated identifier by which to distinguish the session. Some session types, however, may have a distinct beginning, but an inferred end such as an idle timeout or maximum session duration.
Each packet of a session 110 has a session ID 112 (denoted associatively with the first session as ID1). Additionally, the first packet 114 of each session is distinguishable from the rest of the session packets by both the routing component 102 as well as the hosts 106. As illustrated, the first packet 114 of each session may include a special tag 116 to indicate that it is the first packet of the session (denoted P1ID1 114 with special tag TAG1 116), but this is only illustrative; the mechanisms for distinguishing the first packet of a given session depend on several factors, including the session protocol, and many such mechanisms exist. As illustrated herein, the session 110 has the session ID1 112, which is understood to represent that each packet in the session 110 has the same session ID associated therewith (e.g., ID1 112 ). Further, the session 110 can consist of any number of packets, indicated by the last packet X of the session, denoted PXID1 118. Although the session 110 is shown as a group of the same session packets entering the routing component 102 sequentially, this is for illustrative purposes only, since such session packet will be transmitted essentially randomly among, for example, the packets of the other sessions to the input of the routing component 102.
The routing component 102 detects and interrogates each session packet for routing information that includes the session ID 112, the special tag 116 on the first session packet 114, and the last session packet 118. Thus, any packet that is not either the first packet 114 or the last packet 118, is an intermediate session packet. Moreover, when a session ID has been generated and assigned, it will not be used again for subsequent sessions, such that there will not be ambiguity regarding the session to which a given packet belongs. Generally, an assumption can be made that a given session ID is unique for a session, whereby uniqueness is provided by standard network principles or components.
In accordance with the invention, the routing component 102 directs each session packet to a designated host (denoted HOST1, . . . ,HOSTM) as predefined by the routing function 108. Since, for each session packet, the session ID 112 is used as the input to the routing fiction 108, session affinity is preserved; that is, each packet of a given session 110 can be routed to the same host by routing function 108. Further, sufficient load-balancing occurs, since the routing component 102 determines to which of the hosts 106 each session will be assigned and routed, taking into consideration the current loading state of all of the hosts 106.
Thus, the invention generates and utilizes one or more routing functions that define the current availability of one or more of the destination hosts. The routing function also takes into consideration destination loading such that load-balancing is performed between the available hosts. Packets of the same session continue to be routed to the same destination host to preserve session affinity.
Referring now to
The sequence of packets 104 is received into the forwarders 202 (also denoted FORWARDER1, FORWARDER2, . . . ,FORWARDERN) from any of a number of similar or different types of data sources (not shown). In support of distributing the session packets of packet stream 104 to the forwarders 202, a very fast switching system 204 can be employed. The switching system 204 takes as input the entire input stream 104 and distributes it to the forwarders 202 by sending each packet to one forwarder. The switching system 204 may be either software or hardware and may distribute the packets to the forwarders using mechanisms such as round-robin, random, or layer-3/4 hashing (to preserve in-order delivery of packets for a given session), and so forth. The switching system 204 is generally not bound by any requirements involving the session information, e.g., the session ID. The switching system 204 operates to process and route a large number of session packets to any number of the forwarding components 202 without regard for session affinity. For example, the session packets can be randomly switched or directed to any of the forwarders 202 from the switching system 204.
Note that this switching system 204 is not a requirement of the invention, but rather an observation that can significantly improve the achievable scale-out if such a system is available. If no such switching system is available, the forwarders 202 can begin by distributing the packet streams 104 among each other, in a similarly random, round-robin, etc., fashion.
The forwarders 202 are responsible for classifying the incoming session packets received from the switching system 204, load-balancing the packet traffic across a network 206 (which can be, for example, a LAN, WAN, or the Internet) among the plurality of hosts 106, and facilitating that session affinity is maintained once a session is established with a host. Each forwarder achieves load-balancing and session affinity by consulting an identical routing function 108 to process each session packet (thereby rendering which forwarder processes a given session packet irrelevant). These identical routing functions are generated by each forwarder independently given the same state information for the available hosts 106.
In one implementation in accordance with the invention, generation of the routing function is performed by the routing component 102 (of
With respect to the routing function, a fundamental problem in conventional systems is that the system needs to "remember" what to do with the session packets once a session is established with a given host. Finding a least loaded server is not expensive. However, the process of making a determination as to where each and every packet should be sent by searching a look-up table to determine which was the least loaded server at the time the corresponding session begun is very costly. Thus, the routing table can eventually require a large physical memory for all of the sessions and the throughput required. Because a large memory is needed and that memory must now be in a single computing system, the concept becomes problematic as the system grows.
Implementation of the routing function alleviates the need for the large memory and expensive processing required for searching the table. The routing function contains the "memory" necessary to preserve affinity for the existing sessions in the system. Moreover, there can be many different routing functions employed to facilitate stateless, affinity-preserving load balancing. If a host drops offline or is simply overwhelmed, for example, where it can no longer handle its input stream effectively, the forwarders 202 (see e.g.,
Each of the forwarders 202 has the capability to assess availability of the hosts 106, and in response thereto, generate the routing function 108 for local storage and use. Each received packet is then processed against the local routing function 108 to determine to which of the hosts 106 the packet will be routed, and then routed to the assigned host. Given that packets of a session can be switched from the switching system 204 to any of the forwarders 202, each forwarder generates the identical routing function to further transmit packets of the same session to the same host. It is to be appreciated, however, that it is not required that each of the forwarders 202 include the routing function 108. Where the forwarders 202 are interconnected, only selected forwarders need to include the routing function 108 in a partially distributed fashion. For example, forwarders can be paired such that only one of the two forwarders stores the routing function, and whose processing capabilities support the added processing of the other forwarder. In yet another implementation, there can be an external routing function system 208 that handles all of the routing function processing of the pair of forwarders, or where a suitable system is provided, routing function processing for all of the forwarders.
If a routing function such as a hashing function is to be used to provide load balancing and session affinity, then the input to the hashing function should be some identifier that uniquely identifies a session, e.g., the session identifier. This is because sessions can span multiple protocols and multiple connections; the session identifier is the only persistent parameter on which to hash. For different session types, the session identifier can be drastically different; for a TCP connection, the session identifier is the TCP 5-tuple; for SSL, the session identifier is a 32-byte binary number; for a web commerce application, the session identifier may be an HTTP cookie.
Note that some protocols, such as TCP, lend themselves very naturally to session ID extraction, while others, such as SSL do not. Therefore, for such protocols, it may be necessary to add a layer of "intelligence" on top of the hashing scheme in order to associate arbitrary packets with known sessions in progress. The details of this "intelligence" are beyond the scope of this description. When the session identifier for a given packet is recognized, the hashing scheme depicted herein operates as described to provide the required session affinity.
Referring now to
Proceeding to 300 of
Referring now to
If the packet does not include the special tag, progress is from 404 to 410 where the system determines if the packet is the EOS (end-of-stream) packet. If NO, at 412, the packet is an intermediate session packet and is delivered to the same host. The system then receives the next packet. On the other hand, if the session packet is marked as the EOS packet, flow is from 410 to 414 where the session is terminated after routing the EOS packet to the same host.
In a world where the state of the available hosts never changes, the load balancing problem is trivial. A single routing function shared among a set of forwarders is sufficient to ensure that every packet of a given session is delivered to the same host regardless of which load balancer handles any given packet of the session. However, in the real world, the availability of the hosts will change over time. For example, a host may move to an offline state as a result of hardware or software failure. Additionally, a host can remain online, yet its availability is impacted. For example, the host may exhibit a load balance skew such as an overloaded condition or an under-loaded condition that requires a load balancing change. Still further, a new host can be added that will impact a change in load balancing. Changes to the availability of the host population are the most cumbersome operations related to session affinity because host changes further require a change in the routing function of the forwarders. In such instances, the architecture of the invention employs additional routing functions.
A new routing function must therefore be generated to reflect each change in the availability of the hosts. It is clear that switching to a new routing function and discarding the currently used one can cause some portion of the existing sessions in the system to malfunction since the new routing function may route the corresponding packets to the wrong host. An important concept herein is not to switch to the new function, but rather to add the new function for processing in parallel with the existing function over time. When packets arrive, the corresponding session identifier is applied to both the existing and the new functions. If the answer for host routing from both functions is the same, then there is no ambiguity, and the packet is forwarded to the correct host. If, however, the results from the two functions differ, then the correct host destination is ambiguous. The packet is then duplicated and sent to both hosts. Because the hosts are the authoritative source of knowledge concerning the sessions that they are servicing, it is a trivial process for them to discard packets that they receive erroneously.
Essentially, to allow the forwarders to be stateless, the session affinity problem is pushed to the network and hosts when ambiguity exists. By doing so, it can be guaranteed that the packet is delivered to at least the correct host, and potentially some incorrect hosts who then discard the packet. Put differently, the old routing functions contain "memory" necessary to preserve affinity for the existing sessions in the system. Thus, it is important to obtain the benefits of this memory by not switching to the new routing function, but by adding the original routing function to the list of functions to be processed.
Put another way, forwarders can be considered as a set of w computers, any of which can fail or malfunction. Each forwarder maintains a sequence of routing functions (that take as an input a session ID and provide an output that is a host), and a list L of hashing exceptions. This sequence of routing functions are denoted as (ƒ1, ƒ2, . . . ,ƒt) and can be hash functions. At any given moment, the last function in the list, ƒt, is called the driving routing function for that forwarder. All functions ƒi map identifiers into a subset of all hosts (different functions will have different ranges). The list of exceptions L contains pairs of the form [identifier, host] and serves as an explicit list of routing decisions that supersede the sequence of routing functions (ƒ1, ƒ2, . . . ,ƒt). The purpose and generation of exceptions is discussed in greater detail herein below.
When a forwarder receives a packet with identifier p, it executes the following. If this is the first packet of a session, it marks the first packet as special and sends it to ƒt(p). Otherwise, the list of exceptions is consulted, and if the identifier is in that list, the record tells the forwarder exactly where to send the packet. Otherwise, let U be the following multiset: the hosts ƒ1(p), . . . ,ƒt(p). Remove any duplicates from U to get a set S, and send the packet to all of the hosts in set S.
Each host h maintains a session list Lh of all the sessions it is currently hosting, which is trivial overhead for each host. For example, a host h consumes a received packet with packet identifier p, if p ε Lh; otherwise, the host drops it. In contrast to conventional architectures, the first packet of each session will be marked as special. A host always consumes a special packet, and a forwarder sends a given special packet to only one host. The role of the hosts in load balancing is therefore minimal, but critical; each host keeps track of all the sessions it is currently hosting, and inspects each packet it receives to determine whether it should consume it or drop it (by consulting its list of hosted sessions).
The evolution of the routing function begins by the forwarders taking a routing function ƒ and placing it at the end of their routing function sequence (i.e., making it the driving routing function). As long as no host changes are noted, the system operates according to the existing set of routing functions. When changes in host availability are noted, intuitively, a new routing function is generated aimed at appropriately accommodating the new steady-state load distribution (assuming no further changes in host availability). The idea is that host availability changes are relatively infrequent and, as a result, most routing functions will spend sufficient time as the driving function of the system to induce steady state. The non-driving routing functions are there to guarantee session affinity.
After a new routing function ƒ has been added, subsequent packets are addressed according to the old routing function(s) and the new routing function. Consider the packets of a session that was established before the addition of the new routing function ƒ. By induction, all of these packets will reach their assigned host, since the set of routing functions simply increased. At the same time, any host receiving such a packet because of routing function ƒ will drop it, since the packet's session identifier will not be in that host's list (also, by induction). Observe that our earlier care taken by forwarders to trim multisets to sets ensures that the relevant host will not receive multiple copies of the same packet due to multiple routing functions.
Consider the first packet of a new session s that begins after the introduction of routing function ƒ. By assumption, the forwarder receiving this packet will recognize it as special. Therefore, it will mark it as special, and process it only using the driving function ƒt. This guarantees that the packet will go to a live host that will accept it, for example, host h. All later packets of s will be processed using all routing functions. As a result, host h will receive all such packets because of the driving routing function ƒt. Moreover, since no host other than host h will ever receive the first packet of s, every such other host will drop all packets of s.
It is noted that each of the forwarders and/or associated processes relating thereto has the capability to distinguish the first packet of a given session from subsequent session packets. Such a first packet is only processed through the newest (or, so called, driving) function, and is not replicated to multiple hosts. This is not only required for correctness in nearly all protocols, but it also ensures that the set of sessions that rely on a routing function does not increase after the routing function stops being the driving routing function in the system. As a result, when all of the sessions that rely on that function eventually terminate, the routing function can be safely removed.
When adding or removing a host as described above with respect to
In order to minimize the impact of introducing a new hash function, the system uses maximally backward compatible (MBC) routing functions to preserve session affinity. The purpose of using the MBC functions is that, as much as possible, it is desired that the set of routing functions "disagree" on the output for a given input as little as possible. Multiple active routing functions are employed not only to preserve session affinity, but also to account for changes in host availability. Thus, all of the routing functions are processed, and the input packet is duplicated if necessitated by differing destination hosts that are indicated by the routing functions.
In one implementation, the load to the overloaded host is reduced by allowing a reduced number of new sessions to be processed by that host. This impacts system bandwidth to a small degree where a packet is sent multiple times to multiple destinations to ensure the packet reaches the desired host, yet maintains session affinity, an acceptable tradeoff given the extensive problems associated with data restructuring at the failed host that traditionally would be considered. Eventually, it is no longer necessary to keep the old function, as it is no longer needed to preserve session affinity, and can be removed. The routing function can be a special hash function or other type of function that preserves existing host assignments as much a possible.
Note that this mechanism is not limited to two routing functions (the existing function and a new function). If another change in the state of the hosts occurs while two functions are active, a third function (a second new function) can be added to reflect the new state of the host pool, and so on. The cost of doing so is the potential N-way replication of traffic to the hosts (where N is the number of hosts) in the worst case. To reduce the replication of traffic necessary to preserve session affinity, two other techniques can be employed: purging routing functions when they are no longer required, and using exceptions that can supersede use of the routing function(s). An exception is a session for which none of the current routing functions provides the proper affinity. These techniques are described in greater detail below.
Referring to
In other terms, the current set F of routing functions is received by all the hosts in order to test which routing functions are still relevant. The testing (to be described shortly) is identical on all hosts. Upon completion of this testing, each host sends to the forwarders a message which indicates for each routing function ƒi in the set F whether ƒi is necessary (relevant) in order to maintain the affinity of sessions currently hosted by the host. If a hash function is deemed irrelevant by all hosts then it is removed from the set F of routing functions, as it is safe to do so without compromising session affinity.
For a host h to test whether a hash function ƒi in set F is relevant to it, h considers the sessions it is currently hosting, one by one, as follows. For each session s, it determines the set of hosts to which s would be routed if ƒi was to be removed from the set of currently used hash (or routing) functions F. To do this, it simply performs the exact same reasoning as the forwarders do for s, by using the set of routing functions F but with ƒi removed. If the resulting set includes h, then clearly ƒi is unnecessary with respect to maintaining the affinity of session s to the host h; otherwise ƒi is necessary. So, h will report that ƒi is irrelevant to it, if and only if the routing function ƒi is unnecessary (irrelevant) for every session hosted by h.
Accordingly, at 700 of
While this mechanism for purging routing functions is described herein as being Boolean (an "all or nothing" operation), it is also possible to force the removal of a routing function before all sessions that require it for session affinity have completed. An exception is a session for which none of the current routing functions provides affinity. These exceptions supersede the routing functions, in that if an exception exists for a given session, the routing function output(s) are irrelevant. So long as the number of exceptions required in order to purge a given hash function is small, it may be advantageous to do so, especially in the case of very long lasting sessions such as with RDP or EPSec/L2TP, for example.
In fact, much more can be said. Let hƒ be the set of sessions hosted by host h that fail to map to h using the routing functions in F. If the routing function is not redundant for all hosts, it is determined if the number of non-redundant hosts is sufficiently small. That is, if the size of
is sufficiently small, routing function ƒ can still be removed by simply adding to the list of exceptions, L, the pairs [identifier, host] corresponding to the sessions and hosts in Lƒ.
Referring now to
As indicated hereinabove, one method by which to provide load balancing for the incoming client load is through the use of a routing function that is an MBC hash function that can be used to map incoming packets to hosts. One example of an MBC hash function that provides such a capability is a consistent hash function. Stated simply, given some input related to a given packet, a hash function is executed whose output is the host to which the packet should be delivered. In order to ensure session affinity, the stated input to the hash function is the session ID associated with the given packet.
As far as correctness is concerned, there does not need to be any relationship between the functions (ƒ1, . . . ,ƒt). It is sufficient that the range of each such function is the set of live hosts at the time of its addition. Nevertheless, it is intuitively clear that the system will perform less bookkeeping (and the purging will be much more successful) if each hash function "respects" the previous hash functions as much as possible. In other words, consider that a hash function ƒ: N→{1, . . . ,k+1} respects a hash function g: N→{1, . . . k} if the fraction of inputs on which ƒ and g differ is proportional to 1/k.
The idea is to create a "meta-hash-function" Φ that takes as input a string x ε {0,1}q and outputs a hash function ƒ whose range is the subset of {1, . . . ,q} corresponding to the "1's" in x. It can be said that the function Φ yields a hash function family, if for any x,x′ with Hamming distance of one, Φ(x) respects Φ(x′). One can consider x as representing the set of live hosts (which set changes dynamically) and of Φ as a machine for producing appropriate hash-functions. Each host can be partitioned into a large number of virtual hosts and apply hashing to those (for fine load balancing). Additionally, the forwarders will be sharing Φ so they only need to reach consensus on the changes in x. It is to be appreciated by one skilled in the art that there are many constructions of hash functions that can be employed in accordance with the invention.
Since it is changes in the mappings of session identifiers to hosts that place the principal strain on maintaining session affinity, it is a significant benefit to minimize these changes whenever possible. In fact, the implementation of generating hash functions can be made "session aware", in that the hashing process can be made to minimize the number of disrupted sessions by ensuring that the hash entries that have to change are those with the minimum number of established sessions. Further, the number of sessions that require special handling during hash function transition may be able to be reduced by ignoring sessions with the least stringent requirements for affinity. Such capabilities can be employed using artificial intelligence (AI), which AI capabilities are within contemplation of the invention.
It is within contemplation of the subject invention that the disclosed architecture can be suitably configured to process multiple hash function transitions at any given time. However, there may be reasons to not allow the number of active routing functions to grow above a certain level (e.g., two) and that in order to employ a new hash function, the oldest hash function should first be purged, or purging of the oldest hash function is forced before adding the new hash function (e.g., using exceptions). Note, then, that in one implementation, only one hash function transition may be in progress at any given time. For example, if another host comes online during a hash function transition, the current transition must be terminated before the new host can be added to the target host pool.
Further, depending on the relative cost of a hash function transition, it is conceivable that when a host or hosts are added, the hash function transition is actually a series of transitions in order to quickly "bring the host up to speed." More specifically, when a host or hosts are initially added, a hash function is generated that loads the new host(s) more heavily than the others, as it has no existing load (sessions). Additionally, as the host becomes loaded, subsequent hash function transitions are needed in order to reduce the load percentage of the new host to the optimal value. In one implementation, the new host is initially overloaded, but then the overload situation is corrected over time until a "uniform" load is reached (which is not necessarily uniform by definition).
Transient changes in the capacity of a server due to a server being under- or over-loaded are very similar in hashing behavior to the cases already described above except that the availability is not a Boolean indication (alive or dead), but rather a measure of the server's current capacity, which can vary based on the number and type of sessions that it is currently servicing. Such changes in capacity likewise require the generation of and transition to a new hash function. Therefore, the responsiveness of the load-balance to current the system state must be weighed against the overhead necessitated at the forwarders to transition to a new hash function.
Referring now to
Generally, program modules include routines, programs, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the inventive methods can be practiced with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, hand-held computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which can be operatively coupled to one or more associated devices.
The illustrated aspects of the invention may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.
A computer typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computer and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media can comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer.
Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer-readable media.
With reference again to
The system bus 908 can be any of several types of bus structure that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory 906 includes read only memory (ROM) 910 and random access memory (RAM) 912. A basic input/output system (BIOS) is stored in a non-volatile memory 910 such as ROM, EPROM, EEPROM, which BIOS contains the basic routines that help to transfer information between elements within the computer 902, such as during start-up. The RAM 912 can also include a high-speed RAM such as static RAM for caching data.
The computer 902 further includes an internal hard disk drive (HDD) 914 (e.g., EIDE, SATA), which internal hard disk drive 914 may also be configured for external use in a suitable chassis (not shown), a magnetic floppy disk drive (FDD) 916, (e.g., to read from or write to a removable diskette 918 ) and an optical disk drive 920, (e.g., reading a CD-ROM disk 922 or, to read from or write to other high capacity optical media such as the DVD). The hard disk drive 914, magnetic disk drive 916 and optical disk drive 920 can be connected to the system bus 908 by a hard disk drive interface 924, a magnetic disk drive interface 926 and an optical drive interface 928, respectively. The interface 924 for external drive implementations includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.
The drives and their associated computer-readable media provide nonvolatile storage of data, data structures, computer-executable instructions, and so forth. For the computer 902, the drives and media accommodate the storage of any data in a suitable digital format. Although the description of computer-readable media above refers to a HDD, a removable magnetic diskette, and a removable optical media such as a CD or DVD, it should be appreciated by those skilled in the art that other types of media which are readable by a computer, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the exemplary operating environment, and further, that any such media may contain computer-executable instructions for performing the methods of the invention.
A number of program modules can be stored in the drives and RAM 912, including an operating system 930, one or more application programs 932, other program modules 934 and program data 936. All or portions of the operating system, applications, modules, and/or data can also be cached in the RAM 912. It is appreciated that the invention can be implemented with various commercially available operating systems or combinations of operating systems.
A user can enter commands and information into the computer 902 through one or more wired/wireless input devices, e.g., a keyboard 938 and a pointing device, such as a mouse 940. Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, touch screen, or the like. These and other input devices are often connected to the processing unit 904 through an input device interface 942 that is coupled to the system bus 908, but can be connected by other interfaces, such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.
A monitor 944 or other type of display device is also connected to the system bus 908 via an interface, such as a video adapter 946. In addition to the monitor 944, a computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.
The computer 902 may operate in a networked environment using logical connections via wired and/or wireless communications to one or more remote computers, such as a remote computer(s) 948. The remote computer(s) 948 can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer 902, although, for purposes of brevity, only a memory storage device 950 is illustrated. The logical connections depicted include wired/wireless connectivity to a local area network (LAN) 952 and/or larger networks, e.g., a wide area network (WAN) 954. Such LAN and WAN networking environments are commonplace in offices, and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which may connect to a global communication network, e.g., the Internet.
When used in a LAN networking environment, the computer 902 is connected to the local network 952 through a wired and/or wireless communication network interface or adapter 956. The adaptor 956 may facilitate wired or wireless communication to the LAN 952, which may also include a wireless access point disposed thereon for communicating with the wireless adaptor 956.
When used in a WAN networking environment, the computer 902 can include a modem 958, or is connected to a communications server on the LAN, or has other means for establishing communications over the WAN 954, such as by way of the Internet. The modem 958, which can be internal or external and a wired or wireless device, is connected to the system bus 908 via the serial port interface 942. In a networked environment, program modules depicted relative to the computer 902, or portions thereof, can be stored in the remote memory/storage device 950. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers can be used.
The computer 902 is operable to communicate with any wireless devices or entities operatively disposed in wireless communication, e.g., a printer, scanner, desktop and/or portable computer, portable data assistant, communications satellite, any piece of equipment or location associated with a wirelessly detectable tag (e.g., a kiosk, news stand, restroom), and telephone. This includes at least Wi-Fi and Bluetooth™ wireless technologies. Thus, the communication can be a predefined structure as with conventional network or simply an ad hoc communication between at least two devices.
Wi-Fi, or Wireless Fidelity, allows connection to the Internet from a couch at home, a bed in a hotel room or a conference room at work, without wires. Wi-Fi is a wireless technology like a cell phone that enables such devices, e.g., computers, to send and receive data indoors and outdoors; that is, anywhere within the range of a base station. Wi-Fi networks use radio technologies called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, fast wireless connectivity. A Wi-Fi network can be used to connect computers to each other, to the Internet, and to wired networks (which use IEEE 802.3 or Ethernet). Wi-Fi networks operate in the unlicensed 2.4 and 5 GHz radio bands, with an 11 Mbps (802.11a) or 54 Mbps (802.11b) data rate or with products that contain both bands (dual band), so the networks can provide real-world performance similar to the basic 10BaseT wired Ethernet networks used in many offices.
Referring now to
Communications can be facilitated via a wired (including optical fiber) and/or wireless technology. The client(s) 1002 are operatively connected to one or more client data store(s) 1008 that can be employed to store information local to the client(s) 1002 (e.g., cookie(s) and/or associated contextual information). Similarly, the server(s) 1004 are operatively connected to one or more server data store(s) 1010 that can be employed to store information local to the servers 1004.
What has been described above includes examples of the invention. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the invention, but one of ordinary skill in the art may recognize that many further combinations and permutations of the invention are
|Cited Patent
|Filing date
|Publication date
|Oct 31, 1989
|Dec 10, 1991
|International Business Machines Corporation
|One-sided crosspoint switch with distributed control
|Dec 7, 1994
|Dec 12, 1995
|International Business Machines Corporation
|Multi-media flow control over FDDI synchronous networks
|Jun 20, 1997
|Apr 11, 2000
|Telefonaktiebolaget Lm Ericsson
|Segment performance monitoring
|Dec 18, 1998
|Mar 13, 2001
|Nortel Networks Limited
|Method and system for storing and retrieving information in a communications node
|May 30, 1997
|Feb 26, 2002
|International Business Machines Corporation
|Loading balancing across servers in a computer network
|May 12, 1999
|Jul 16, 2002
|International Business Machines Corporation
|Process and system for tracing documents in an interlocking computer system
|Oct 8, 1997
|Jul 23, 2002
|International Business Machines Corporation
|Affinity-based router and routing method
|Nov 15, 1999
|Jun 8, 2004
|International Business Machines Corporation
|Method and apparatus for the load balancing of non-identical servers in a network environment
|Feb 29, 2000
|Nov 8, 2005
|Utstarcom, Incorporated
|Applying modified mobile internet protocol (IP) in a wireless mobile data network interface
|Jul 2, 1999
|May 23, 2006
|Cisco Technology, Inc.
|Integrating service managers into a routing infrastructure using forwarding agents
|Jan 7, 2004
|Oct 23, 2007
|Burnside Acquisition, Llc
|Data repository and method for promoting network storage of data
|May 12, 2004
|Jan 8, 2008
|Sourcefire, Inc.
|Systems and methods for determining the network topology of a network
|Nov 7, 2001
|Jun 27, 2002
|Henderson Alex E.
|Switch-based network processor
|Apr 3, 2001
|Oct 3, 2002
|International Business Machines Corporation
|Automatic affinity within networks performing workload balancing
|Apr 30, 2001
|Oct 31, 2002
|Colasurdo David B.
|Method and apparatus for maintaining session affinity across multiple server groups
|Jul 24, 2001
|Jan 30, 2003
|Delima Roberto
|Dynamic HTTP load balancing method and apparatus
|Oct 17, 2001
|Apr 17, 2003
|Kave Eshghi
|Allowing requests of a session to be serviced by different servers in a multi-server data service system
|Nov 16, 2001
|Jun 5, 2003
|Connors Dennis P.
|Method and implementation for a flow specific modified selective-repeat ARQ communication system
|Apr 26, 2002
|Oct 30, 2003
|Samsung Electronics Co., Ltd.
|Router using measurement-based adaptable load traffic balancing system and method of operation
|May 31, 2002
|Dec 4, 2003
|Chavez David L.
|Emergency bandwidth allocation with an RSVP-like protocol
|Oct 17, 2003
|Jul 15, 2004
|Gemini Mobile Technologies, Inc.
|Distributed, fault-tolerant message store
|Mar 31, 2003
|Sep 30, 2004
|Alok Kumar
|Method and apparatus for packet classification using a forest of hash tables data structure
|Feb 19, 2004
|Nov 4, 2004
|Hypertrust Nv
|Additional hash functions in content-based addressing
|Sep 8, 2003
|Dec 30, 2004
|Darling Christopher L.
|Network load balancing with traffic routing
|Aug 28, 2003
|Mar 3, 2005
|Aiken John Andrew
|Methods, systems and computer program products for application instance level workload distribution affinities
|Nov 17, 2003
|May 19, 2005
|Kamal Jain
|Systems and methods for approximating optimal distribution via networked systems
|Feb 22, 2005
|Jul 14, 2005
|Microsoft Corporation
|Method and system for managing data records on a computer network
|Mar 26, 2004
|Sep 29, 2005
|Stacy John K
|Hardware filtering support for denial-of-service attacks
|Jun 20, 2003
|Apr 27, 2006
|Bowen Hoyt E Jr
|Load balancing in data networks
|Feb 9, 2005
|Aug 24, 2006
|International Business Machines Corporation
|Method and apparatus for automatic recovery from a failed node concurrent maintenance operation
|Feb 25, 2005
|Aug 31, 2006
|Ramana Rao
|Network address filter including random access memory
|Apr 5, 2005
|Nov 9, 2006
|Shimon Muller
|Network system
|Jul 8, 2005
|Jan 11, 2007
|Microsoft Corporation
|Load balancer management
|Jun 14, 2006
|Dec 20, 2007
|Microsoft Corporation
|Transparently extensible firewall cluster
|Mar 1, 2005
|Jan 24, 2008
|Piper Andrew A
|Method And System For Affinity Management
|Feb 28, 2007
|Aug 28, 2008
|Microsoft Corporation
|Detect User-Perceived Faults Using Packet Traces in Enterprise Networks
|Jun 25, 2007
|Dec 25, 2008
|Microsoft Corporation
|Scaling network services using dns
|Oct 2, 2007
|Apr 2, 2009
|Microsoft Corporation
|Uncovering the differences in backbone networks
|Oct 2, 2007
|Apr 2, 2009
|Microsoft Corporation
|Uncovering the differences in backbone networks
|Sep 27, 2007
|Apr 2, 2009
|Microsoft Corporation
|Intelligent network address lookup service
|Feb 29, 2008
|Sep 3, 2009
|Microsoft Corporation
|Monitoring network performance to identify sources of network performance degradation
|D. Karger, et al. Web Caching with Consistent Hashing. Proceedings of the Eighth International Conference on World Wide Web, 1999. 16 pages.
|D. Karger, et al.. Consistent Hashing: Load Balancing in a Changing World. Jul. 1, 1997. 38 pages. http://theory.Ics.mit.edu/~karger/Papers/Talks/Hash/.
|D. Karger, et al.. Consistent Hashing: Load Balancing in a Changing World. Jul. 1, 1997. 38 pages. http://theory.Ics.mit.edu/˜karger/Papers/Talks/Hash/.
|Distributed Load Balancer (MS1-4809US), Aug. 11, 2008.
|E. Vaynshteyn. Applications of Consistent Hashing. World Wide Web Consortium Working Draft, last viewed on Jul. 22, 2004. http://www.mit.edu/people/ecoder/hashing-report.pdf.
|E. Vaynshteyn. Applications of Consistent Hashing. World Wide Web Consortium Working Draft, last viewed on Jul. 22, 2004. http://www.mit.edu/people/ecoder/hashing—report.pdf.
|R. Kleinberg and T. Leighton. Consistent Load Balancing via Spread Minimization. Proceedings of the 35th Annual ACM Symposium on Theory of Computing, pp. 565-574, 2003.
|System Method for Performing Client-Centric Load Balancing of Multiple Globally, Nov. 16, 2000.
|Citing Patent
|Filing date
|Publication date
|May 15, 2009
|Sep 20, 2011
|International Business Machines Corporation
|Controlling access to a destination in a data processing network
|Aug 2, 2011
|Jan 15, 2013
|International Business Machines Corporation
|Controlling access to a destination in a data processing network
|Aug 27, 2012
|Aug 18, 2015
|Microsoft Technology Licensing, Llc
|State maintenance as a service
|Jul 13, 2012
|May 3, 2016
|Freescale Semiconductor, Inc.
|Core-affine processing on symmetric multiprocessing systems
|Nov 18, 2014
|Sep 6, 2016
|Microsoft Technology Licensing, Llc
|Synchronizing state among load balancer components
|Sep 8, 2015
|May 30, 2017
|Microsoft Technology Licensing, Llc
|Proxy-based cache content distribution and affinity
|Jan 9, 2012
|Jun 13, 2017
|At&T Intellectual Property I, L. P.
|Methods and apparatus to route message traffic using tiered affinity-based message routing
|Dec 15, 2005
|Apr 9, 2009
|Anders Danne
|Method and Apparatus for Load Distribution in Multiprocessor Servers
|May 15, 2009
|Nov 26, 2009
|International Business Machines Corporation
|Controlling Access to a Destination in a Data Processing Network
|Aug 11, 2008
|Feb 11, 2010
|Microsoft Corporation
|Distributed load balancer
|Jul 13, 2012
|Jan 16, 2014
|Rekesh John
|Core-affine processing on symmetric multiprocessing systems
|Aug 27, 2012
|Feb 27, 2014
|Microsoft Corporation
|State maintenance as a service
|Nov 18, 2014
|Mar 5, 2015
|Microsoft Technology Licensing, Llc
|Synchronizing state among load balancer components
|U.S. Classification
|370/229, 709/217
|International Classification
|Cooperative Classification
|European Classification
|May 19, 2005
Owner name: MICROSOFT CORPORATION, WASHINGTON
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ACHLIOPTAS, DIMITRIS;HOUSE, SEAN B.;REEL/FRAME:016036/0270;SIGNING DATES FROM 20050331 TO 20050413
Owner name: MICROSOFT CORPORATION,WASHINGTON
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ACHLIOPTAS, DIMITRIS;HOUSE, SEAN B.;SIGNING DATES FROM 20050331 TO 20050413;REEL/FRAME:016036/0270
|Sep 25, 2013
|Fee payment
Year of fee payment: 4
|Dec 9, 2014
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034543/0001
Effective date: 20141014


text:
Got tracking for mine sent from TA_Planet. I got the pixel perfect version $370 isn't that much. Actually I cancelled the 2720mdp after reading the review which was $500. I got the pixel perfect version and a Spyder4Express for the same price. I've already calibrated my laptop it was a huge difference and suprised it had 97% sRGB (Alienware M17xR3).
post #3591 of 4784
4/28/12 at 5:25am


text: Salicornia is a tasty little succulent plant that grows in salt marshes. Its flavor is reminiscent of a salty asparagus with maybe a little spinach thrown in. It's also known as picklewort, glasswort, sea beans or samphire. I had it in one of the most amazing and memorable salads I've ever had -- it was paired with quail eggs, mache and mulberries, if I'm remembering correctly. Has anyone seen it for sale anywhere -- farmer's markets, specialty stores, anywhere? Please help.


text:
Fish 'n Flush
Against my better feelings, I feel I must recommend this, if only to further my fascination with the wondrous things our society can produce. This would make a great gift for that certain type of person who likes these types of things (know what I mean??) I just feel bad for the fish - can you imagine? "Uh-oh, here he comes again, looks like he's a little sick..."
How fun is this?
an elegant way to greet your guests at the entry, or as a chic accessory to your boudoir. i love the mirrored frame. very pretty.Tags: home, home decor, home accessories, mirrors, air, bathroom decor, wall mirrors, boudoirs, bathroom mirrorAdded 9 Years Ago from West Elm
If you have no soap then you will really smell!Tags: gifts, bathroom, soap, exfoliators, air, spices, citrus, Burts Bees, bathroom decorAdded 10 Years Ago from Vitacost
Beautiful photo. Great color!Tags: gifts, bathroom, bath, green, photography, plants, wall arts, air, wall decorAdded 10 Years Ago from Pottery Barn


text:
Know someone who is the epitome of an achieving woman? It's time to nominate her for the 2012 WOMEN OF ACHIEVEMENT Awards—a St. Louis tradition that honors volunteers who, through their dedication and leadership, have made our community a better place. Nomination forms are available online at woastl.org, or by sending a SASE to Women of Achievement, c/o MAC Meetings & Events, 801 N. 2nd St., Ste. 302, St. Louis, MO 63102. The deadline for submitting nominations is Jan. 13. The 2012 luncheon will be held on Thursday, May 17, at The Ritz-Carlton.
New Cardinals skipper MIKE MATHENY is among 12 St. Louisans being honored as a 'Character of Courage' by Metro Theater Company in recognition for the positive impact he and his Catch-22 Foundation have had on the community. During the three-week run of Metro Theater's production of the Civil War drama, Battledrum, at the Missouri History Museum, tributes will be held to Matheny and others, including teen volunteer SIMONE BERNSTEIN, The Learning Tree Intergenerational Center's SHIRLEY BLACK, Maplewood-Richmond Heights schools superintendent LINDA HENKE, Cultural Leadership's KAREN KALISH, education and social justice activists TRESA and FRANKLIN MCCALLIE, Missouri State Representative JEANETTE OXFORD, Pace Properties executive RICK RANDALL JR., Ranken Jordan topper LAURIE TANNER, Kwame Building Group's TONY THOMPSON, Saint Elizabeth, Mother of John the Baptist Church and St. Louis Catholic Academy pastor FR. JEFF VOMUND and University City Children's Center's STEVE ZWOLAK. Battledrum, which runs Jan. 10 to 28, tells the story of three young boys who take a risk to help others.
OPERA THEATRE OF SAINT LOUIS' 2012 spring gala is in the works. 'La Habanera,' a Carmeninspired costume ball, will take place May 5. In the Opera Theatre tradition of elaborately themed balls, this one will channel gypsies, bandits, soldiers, toreadors, mantillas and lace—all in celebration of the magic of Seville. The event venue will be announced at a later date.
Some 150 European paintings, sculptures, drawings and prints will soon call the SAINT LOUIS ART MUSEUM home, thanks to donors MARK and PHOEBE WEIL. The Weil Collection, acquired in the past 25 years, includes some noteworthy works such as ANDREA MANTEGNA's Entombment and ALBRECHT DURER's Adam and Eve. Also part of the collection are BENEDETTO DA MAIANO's sculpture, Saint John the Evangelist, and a stunning impression of REMBRANDT's most famous print, Christ Preaching. Mark Weil is an art historian and former director of the Mildren Lane Kemper Art Museum; his wife, Phoebe, pioneered the field of sculpture conservation in St. Louis in the 1970s.
Armenian scientist DR. ARAM AGHASYAN is the recipient of this year's Saint Louis Zoo Conservation Award for his role in helping to establish parks and sanctuaries in his home country to conserve Armenia's wildlife. In addition, longtime zoo leader MAHLON WALLACE III was honored with the Individual Award for his decades of service; the Corporate Award recognized PEABODY ENERGY for its contributions to The Living Promise campaign and the Zoofari and A Zoo Ado events, among others; and the Foundation Award went to INTERCO CHARITABLE TRUST for its advocacy and generous donations over the years.
WASHINGTON UNIVERSITY's Olin Business School has received two significant gifts totaling $25 million. They are from longtime benefactors CHARLES and JOANNE KNIGHT ($15 million), and GEORGE and CAROL BAUER through the Bauer Foundation ($10 million). The money will support the university's plans to build two new facilities for graduate education.