text:
Enjoy this article? You may also be interested in these related items from our partners
Jonas Brothers: the Concert
Experience Extended - 3-Disc Set
Kingdom Hearts HD II.5
ReMix Limited Edition
Fascinating Walt Disney-
Midwestern Prologue- The Artist Part 1
Casting notices have gone up seeking saxophone, clarinet and percussionists with 'energetic and outgoing personalities' for a new entertainment experience at the Magic Kingdom. There is no indication where the performances will be held, but the theme park has recently put out casting calls for circus acts and other improv groups believed to be used as part of the New Fantasyland Expansion at the Magic Kingdom.


text:
Email Sign Up
Katja Kassin Videos Favorite
Picture courtesy of
New Sensations
Born September 24, 1979 in Germany, pornstar Katja Kassin (aka Katja Kasin, Katya Kassin and Katia Kassin) has one of the best backsides in porn. In September 2000 Katja started working as a nude model and made her way into adult entertainment in 2003, working as a Spiegler Girl for over three years. She has since starred in more than 400 adult movies. Katja Kassin's 36D breasts are spectacular on her elfin 5'1" frame, but it's her curvaceous derriere that made her reputation and her fortune. Katja's specialty is rough anal sex, including double anal, but she's also up for DP, Inter-Racial, Orgy, Domination, Facials, All-Girl and Gang Bang.
Scenes with Katja Kassin: View All Scenes

subset: commoncrawl


text:
Outside My Window – March 1, 2015
Earlier in the afternoon today, a little bit of sunshine came through and even though it was cold out there, brave souls came out and frolicked on the ice. On the beach. Outside my window.
The snow stuck! – January 3, 2015
Woke up to fog this morning and first significant snow of the winter. After brunch, I decided to walk on the beach. Great beach weather today! I forgot to put on my sunscreen though. In my dreams, right?
Lake Michigan from Thorndale to a little bit beyond Hollywood – June 10, 2006
Photo stroll with Bernie (fotoFANATIK on Flickr) along the beach. She had a patient in my building so after her session seeing the patient, she called me … and off we went photographing. My neck of the woods I can't wait Cosmos The Lifeguard and the Photographer The Beach


text:
Is your resume in need of a little disruption?
Startups are all about
making things work better.
Our resume writing service is all about doing that for your CV.
StartUpHire is the place to find Interactive-Designer jobs in and near . Review the list below of Interactive-Designer jobs. These Interactive-Designer jobs represent some of the most rewarding careers available. You can refine your job search for Interactive-Designer jobs in by adding a job title, company name, or keyword to the search box above … or click the "Advanced Search" link for more options. Don't see the employment opportunity you're looking for? Check back soon for more Interactive-Designer jobs in , or sign up for an email alert (link in upper left) to receive periodic updates on new Interactive-Designer jobs in , .
AppNexus is looking for an Associate Instructional Designer to join a growing and evolving Client Education & Development team. This role offers the unique opportunity to influence and shape the way AppNexus educates clients on our product and industry.As an Associate6/15/2015 New York, NY Save This Job
Palo Alto Houzz is seeking extraordinary people to transform the home remodeling industry. Located in downtown Palo Alto, we are revolutionizing the way people envision, design and build their homes. We are looking for a talented Brand and Marketing Designer who is passionate5/30/2015 Palo Alto, CA Save This Job
at Houzz Palo Alto Upgrade your career at one of the hottest startups in Silicon Valley! If you are great at what you do, thrive in a highly collaborative environment, and don?t mind the occasional meeting in the bathroom (don?t worry, it?s just one of our conference rooms), we5/30/2015 Palo Alto, CA Save This Job
Upgrade your career at one of the hottest startups in Silicon Valley! If you are great at what you do, thrive in a highly collaborative environment, and don?t mind the occasional meeting in the bathroom (don?t worry, it?s just one of our conference rooms), we want to hear from5/30/2015 Palo Alto, CA Save This Job
The Opportunity Gilt Groupe is looking for a Senior Graphic Designer to join our fun and innovative team. The Sr. Graphic Designer will work within the marketing team to conceptualize, design and execute on-site and off-site creative. This individual is a strong leader, a fast5/25/2015 New York, NY Save This Job
Location: Cleveland, OHDepartment: User ExperienceType: Full TimeMin. Experience: Experienced Summary & Purpose: Explorys is seeking an innovative Interaction Designer with a portfolio combining strong interaction and interface design skills. This interaction designer is5/6/2015 Cleveland, OH Save This Job
Common information: We are currently seeking senior-level User Experience Designers for perm, full time opportunities to join our Conshohocken, PA area team of professionals. Our ideal candidate has exposure with all aspects of the user-centered design process, from user5/3/2015 Conshohocken, PA Save This Job
WHAT WE DO Zumobi is a leading mobile media and technology company that partners with top brands and publishers to provide integrated app content and advertising experiences on smartphones and connected devices. WHAT YOU?LL BE DOING We are looking for a Design professional5/23/2015 Seattle, WA Save This Job
Senior UX Designer Who we are We create events for companies to WOW their clients. Kapow Events started in 2012 and has been revolutionizing the event space. Combining the ease of an online marketplace showcasing hundreds of venue and service partners with the knowledge of5/8/2015 Chicago, IL Save This Job
Location: Deerfield, IL Responsibilities: Conduct user research, testing, and feedback sessions with users Analyze data from different sources and methods to develop a holistic picture of the problems and opportunities Summarize results into actionable recommendations that7/2/2015 Deerfield, IL Save This Job
This position is located in San Francisco, CA. Nerdwallet is seeking a Lead Product Designer who will push boundaries, influence strategy, and have a strong impact on product execution and quality. We?re intent on raising the bar in the financial technology space with smart,6/22/2015 San Francisco, CA Save This Job
What you?ll do As a User Interface Designer, you will lead the visual design of user interfaces for specific Belkin software products and product families. You will perform visual design and production duties for interfaces to appear on phones, tablets, and computers running the5/31/2015 Los Angeles, CA Save This Job
What you?ll do As a User Experience Designer for WeMo, you will be focused on creating and delivering fantastic user experiences across multiple touchpoints for the WeMo eco-system, one of the leading brands in the emergent ?Internet of Things? space. You will use your unique5/31/2015 Los Angeles, CA Save This Job
As a Technical Designer, you will be joining our exciting Creative team to help support and drive our Playable ad creation lifecycle, often from start to finish. Mobile applications are becoming more complex, and as such, we need someone who can help push our products to the next5/11/2015 San Francisco, CA Save This Job
DESCRIPTION We?re an early stage startup seeking a highly creative and energetic Front-End Designer to join our Creative team. The Designer will be responsible for front end development projects using HTML, XHTML, CSS, XML, RSS, JavaScript and other related interactive5/30/2015 Chicago, IL Save This Job
Responsibilities Get to know, understand and breathe Outbrain?s customers? needs and serve as their advocate throughout the product development lifecycle and the product pillars. Create and maintain user experience documentation such as competitive site analyses, user6/24/2015 New York, NY Save This Job
Job Id: 117 As a member of the Visually product team you will be responsible for delivering clean, informed designs in both a consumer facing and B2B space. You?ll employ creative problem solving to translate wireframes and requirements into beautiful but simple interfaces. You6/8/2015 San Francisco, CA Save This Job
EPAM, recently named the #1 small tech company in the US by Forbes, is currently seeking a Visual Designer to join their San Francisco area team. The Visual Designer contributes to concept development and is responsible for the execution of creative deliverables. As a member of a6/2/2015 San Francisco, CA Save This Job
EPAM, recently named the #1 small tech company in the US by Forbes, is currently seeking a Visual Designer to join their Philadelphia area team. The Visual Designer contributes to concept development and is responsible for the execution of creative deliverables. As a member of a6/2/2015 Conshohocken, PA Save This Job
Job Description Position Summary: The Graphic Designer is responsible for creating visually compelling designs that support the delivery of our messages. With a scope that spans both traditional and interactive (digital) design, the designer will improve our user experience by5/20/2015 Cincinnati, OH Save This Job


text:
Famous Smoke Shop Premium Cigars at Discount Prices! Order Toll Free: 1-800-564-2486 Contact Us
Quesada Tributo Alvaro Reviews
average customer review
from 0 customers
Purchased this product? Share you thoughts with other customers:
write your own review
write your own review
There are no reviews for this item.
recommended for you based on Quesada Tributo Alvaro
39% OFFLa Aurora 1495 Belicoso 5 Pack Pack of 5 $44.10 $26.99
25% OFFLa Gloria Cubana Serie R Belicoso 5 Pack Pack of 5 $37.45 $28.08
40% OFFLa Gloria Cubana Artesanos De Tabaqueros 652B 5 Pack Pack of 5 Only 3 left! $52.45 $31.21


text:
By clicking the "Submit" button, you agree to Avvo's Terms of Use. Please see our Community Guidelines and Privacy Policy for information on posting to Avvo and how we collect, use, and share information you provide to us.
16 W Main St Ste 736Rochester, NY 14614
Licensed for 34 years
Go back to Robert's profile.


text:
The Borg
View Single Post
October 11 2013, 06:46 AM
Location: Tactical withdrawl along the Klingon border
Re: The Borg
They were wrong, very wrong, to include it.
I get it that there was only a handful of Borg, but they were 24th century Borg, surely they would have done far more damage and had a much greater effect on the 22nd century. Also, having Phlox's "immunity" to assimilation was a joke and far too convenient (plus had the boffins at Starfleet Intelligence, who obviously kept this whole thing quiet, knew the Denobulans had such an advantage over the Borg why weren't they developing that immunity for two hundred years, ready to provide every Starfleet ship/base with a hypo to prevent assimilation when the Borg finally showed up again?).
It's a poor example of taking an established Trek race and cramming them into the new series without any thought or finesse, as they did with the Ferengi and Klingons. I'm surprised they never brought Q in for another appearance.
Avatar: Captain Naya,
U.S.S. Renown
NCC-1415 [Star Trek: Four Years War]
Manip by: JM1776 (STPMA.net)
View Public Profile
Send a private message to Bry_Sinclair
Find More Posts by Bry_Sinclair


text:
Ships Free to USA*!
Electric Upright Glass Washer from Admiral Craft. Bronze bearings, stainless steel construction. Enclosed motor with sealed lower bearing and splashguard canopy. ETL approved to UL, CSA and NSF standards. Adcraft has been a leading manufacturer of restaurant equipment and supplies since 1952. In those 60 plus years, with superior customer service and innovation, Adcraft's product line has grown to include more than 2000 quality commercial kitchen products. We have proudly distributed these products for 24 years.
Other Items You May Be Interested In
Bar Maid Submersible Glass Washer
Libbey Impressions 8 Ounce Old Fashioned Glass - One Dozen
ABestKitchen is the original online restaurant supply and kitchen equipment company. Selling online since 1998, we stock thousands of commercial quality foodservice products. We're not the biggest, but we're the best.
Need help? Our experienced staff is available if you need help selecting the best product to suit your needs.


text:
386 Geary Street
San Francisco, California 94102
Get Directions
About Hotel G San Francisco
Property Location
A stay at Hotel G San Francisco places you in the heart of San Francisco, steps from Alcove Theater and Union Square. This hotel is close to Lombard Street and Fisherman's Wharf.
Make yourself at home in one of the 153 air-conditioned rooms featuring iPod docking stations and flat-screen televisions. Your room comes with a pillowtop bed. Complimentary wireless Internet access keeps you connected, and digital programming is available for your entertainment. Private bathrooms have designer toiletries and hair dryers.
Rec, Spa, Premium Amenities
Take advantage of recreation opportunities such as a fitness center, or other amenities including complimentary wireless Internet access and tour/ticket assistance.
Enjoy a meal at a restaurant or in a coffee shop/café. Or stay in and take advantage of the hotel's 24-hour room service. Quench your thirst with your favorite drink at a bar/lounge.
Business, Other Amenities
Featured amenities include a computer station, complimentary newspapers in the lobby, and dry cleaning/laundry services.
The Fine Print
Hotel Cancellation Policy
The room rates listed are for double occupancy per room unless otherwise stated and exclude tax recovery charges and service fees. The actual booking amount will be charged in USD. See FAQs for more details.
Book this hotel on Groupon and you will receive a Groupon Bucks credit worth 5% of the price before taxes and fees. We will deposit this credit into the Groupon account used to make the purchase within 24 hours of hotel check-in, and it will expire within 180 days. Please note: At this time, promo codes and Groupon credit (i.e., Groupon Bucks, incentives, or gift cards) cannot be applied to Market Pick hotel reservations.
- Room service (24 hours)
- Complimentary newspapers in lobby
- Bar/lounge
- One meeting room
- Free Wi-Fi
- Restaurant
- Fitness facilities
Recent Reviews from TripAdvisor
What Makes This Getaway Great
Best Ratings & Value
All Getaways Market Pick hotels boast classifications of 3 stars and above, high customer ratings, the most desirable locations, and great value.
5% Groupon Bucks Credit
Book a Market Pick hotel on Groupon and get a Groupon Bucks credit worth 5% of the room rate for every night you stay. It's our way of ensuring you get the best value around.


text:
NuSil Technology Continues Expansion With Fourth BuildingNovember 28, 2012 12:30 pm | News | Comments
"The expansion at Bakersfield is a long-term strategic decision for NuSil to help facilitate and maintain the growth of the company," said Scott Mraz, President and CFO of NuSil Technology LLC. "Our goal is to increase capacity with the best available manufacturing technology, and we have made significant investments in new energy-efficient buildings and processes to accomplish this goal."
Plastics Plant Reopening In KentuckyOctober 31, 2012 8:17 am | News | Comments
A plastics company that has plants in four other Kentucky cities plans to reopen a facility in western Kentucky, creating 400 new full-time jobs in Madisonville. Berry Plastics Corp. announced Tuesday it will begin reconfiguring the Madisonville plant, with production expected to begin next year.
PolyOne To Buy Spartech Corp.October 25, 2012 8:38 am | News | Comments
Chemical company PolyOne Corp. reported an 11 percent increase in its third-quarter net income on Wednesday and disclosed it has agreed to buy plastics products maker Spartech Corp. for $246 million in stock and cash. PolyOne makes resins used in plastic pipe and other products, and said the earnings increase reflected an improved product mix.
Myers Industries Completes Jamco DealOctober 2, 2012 11:07 am | News | Comments
Myers Industries Inc., which makes plastic products for industrial, agricultural, automotive markets, said Tuesday that it completed its acquisition of Jamco Products Inc. Financial terms were not disclosed. Jamco designs and manufactures a variety of heavy-duty industrial steel carts and safety cabinets.
Study Suggests Tie Between BPA And Child ObesitySeptember 18, 2012 11:01 am | by Mike Stobbe, AP Medical Writer | News | Comments
While most people have traces of the plastics chemical in their bodies, the study found that children with the highest levels in their urine were twice as likely to be obese as those with the lowest. There are other factors that could explain the results, and many reasons why children gain too much weight, the researchers said.
Indian Capital To Ban Use Of All Plastic BagsSeptember 12, 2012 11:05 am | by The Associated Press | News | Comments
The government of India's capital is hoping that a strict ban on plastic bags will help the environment. An engineer with Delhi's government says that starting next week, the manufacture and sale of all types of plastic sheets and bags will be banned in the city, citing their environmental dangers.
Ill. Governor Rejects Plastic Bag Recycling BillAugust 27, 2012 9:47 am | by The Associated Press | News | Comments
Gov. Pat Quinn vetoed a bill Sunday that would have required plastic bag manufacturers to set up collection and recycling programs, calling it a "roadblock" for local communities to make their own choices. The proposed law says manufacturers must set up recycling programs with the goal of increasing the recycling rate by 12 percent by 2015 and to eventually make bags from at least 30 percent recycled content.
Will.i.am, Coca-Cola Try To Make Recycling HipJuly 31, 2012 3:17 pm | News | Comments
Should headphones made out of recycled trash ever cost $349? Perhaps, if they're designed by musician will.i.am.
Eastman Chemical Joint Venture To Make China PlantJuly 31, 2012 1:49 pm | News | Comments
Sinopec Yangzi Petrochemical Co. will help Eastman gain more access to fast-growing consumer packaged goods markets in Asia.
Pimp My ProstheticJuly 9, 2012 11:54 am | by Meaghan Ziemba, Associate Editor, PD&D | Articles | Comments
Bespoke Innovations uses 3D scanning to capture and accurately recreate to produce a custom form that changes the way the world views prosthetics.
LIMS Technology In Synergy At NPE 2012June 26, 2012 3:17 pm | Manufacturing.net | Product Releases | Comments
Shin-Etsu Silicones of America recently premiered advanced technical demonstrations with industry leading equipment partners, featuring Shin-Etsu's KEG-2000 LIMS (Liquid Injection Molding System) products' advanced handling and molding properties being run with a variety of integrated LIMS machinery and equipment.
Silicone Extrusion Technology Delivers High-Precision ProductsJune 8, 2012 12:42 pm | News | Comments
Saint-Gobain Performance Plastics (SGPPL) HealthCare Markets, a leading supplier of high-performance medical device components, introduces Compass Technology, a new co-development for silicone extrusion.
Iowa Tire Plant Getting $74M ExpansionJune 8, 2012 9:01 am | News | Comments
Bridgestone Americas plans to invest $74 million more through next year to expand its Firestone tire factory in Des Moines.
LA Ban Could Spell End For The Plastic BagMay 31, 2012 2:24 pm | by John Rogers, Associated Press | News | Comments
Environmentalists believe a ban in the city of 4 million will have a huge impact and could even influence the rest of the country to follow suit.
Plastics Recycler Building Plant In TennesseeMay 22, 2012 2:38 pm | News | Comments
Plastics recycler The Highlands Group says the new facility will have 34 manufacturing jobs.
Cytec Industries Sells Adhesives Unit For $105MMay 11, 2012 12:50 pm | News | Comments
The purchaser, Henkel AG, said the specialty adhesives, which are used in industrial and consumer products, will complement its current lineup.
Automakers Avoid Crisis After Scramble For ResinApril 27, 2012 3:03 pm | by Tom Krisher, AP Auto Writer | News | Comments
The auto industry may have steered its way around another crisis, as Ford said it doesn't expect the shortage of PA-12 to affect production.
Damaged Resin Plant Will Be Down At Least 6 MonthsApril 26, 2012 11:59 am | by Tom Krisher, AP Auto Writer | News | Comments
A shortage of the resin the Evonik Industries plant produces is threatening to cut car production just as auto sales are accelerating in the U.S.
4 Auto Parts Makers Downplay Resin ShortageApril 25, 2012 9:35 am | by Tom Krisher, AP Auto Writer | News | Comments
The resin, called PA-12, is most critical in fuel and brake lines because it can carry gasoline and other fluids without deteriorating.
Michelin Recalls 77,000 TiresApril 23, 2012 9:12 am | News | Comments
Michelin is recalling more than 77,000 bus tires because sidewall defects can cause them to lose air rapidly, increasing the risk of a crash.
Chemical Plant Shutdown Could Cut Auto ProductionApril 17, 2012 9:20 am | by Dee-Ann Durbin and Tom Krisher, AP Auto Writers | News | Comments
An explosion at Evonik Industries in Germany killed two workers and damaged a factory that makes a component of plastic fuel and brake lines.
Del-Nat Pulls 2,700 Tires Because Of BlisteringApril 9, 2012 9:16 am | News | Comments
Del-Nat Tire Corp. says it will pull 2,663 Chaparral Radial XT tires off the market because of a defect that could increase the chances of a crash.
Printing Prosthetics, Engines, And JetsMarch 27, 2012 10:00 am | by Joel Hans, Managing Editor, Manufacturing.net | Videos | Comments
Additive manfuacturing is an industrial form of 3D printing that can produce large, complicated designs with only a fraction of the waste as seen in the typical machining process.
Kyowa America Announces New $12M Plant In Tenn.March 23, 2012 2:26 pm | News | Comments
The automotive supplier specializes in plastic injection molding and will create 160 jobs with the investment.
Dart Container To Buy Solo Cup For $315M In CashMarch 22, 2012 1:54 pm | News | Comments
The maker of the ubiquitous red plastic cups will be folded into Mason, Mich.-based Dart, so it can broaden the range of materials it uses.


text:
Westwood High School's Student Council is named the 2015 AAAA Student Council of the Year! Congratulations also to Nick Hooks for being inducted into the Terry McCoy Hall of Fame and to Ekklesia Hawkins for being recognized as the SC District 4 Chair for 2014-15.
Nick Hooks was elected President of the Student Body and Megan Knight was elected Vice President for the 2015 - 16 School Year. This year the election was done differently with the President and Vice-President running as a team. Both look forward to working with the student body to continue to build a great Student Government and an awesome school.


text:
Emissions Monitoring Companies in Dominican Republic
Category: Air & Climate ×
Subcategory: Emissions Monitoring ×
Industry: Energy ×
Location: Latin America ×
Location: Dominican Republic ×
Environmental Source Samplers, Inc.
based in Wilmington, NORTH CAROLINA (USA)
Environmental Source Samplers, Inc. (ESS) has a long and proven track record. We have been providing air quality related services since 1979. Headquartered in Wilmington, North Carolina, ESS has provided stack testing services in most of the fifty states and internationally. ESS serves clients in many industries including: Aluminum, Asphalt, Automobile, Carbon, ...
CEMS Experts - a division of GK Associates, Inc.
based in Simi Valley, CALIFORNIA (USA)
CEMS Experts offers a wide range of custom solutions for CEMS (Continuous Emissions Monitoring Systems), environmental consulting, CEM rentals, sales and service. We are committed to providing our customers with courteous, professional service at cost-effective prices.
Lowe Engineering Limited
based in Leeds, UNITED KINGDOM
Lowe Engineering has emerged as a progressive and dynamic organisation operating successfully in a number of related markets. Lowe is internationally recognised as a leading authority in the design and manufacture of sample conditioning systems for boiler water, steam, process liquids and gases in both power plant and process plant applications. Many years of close ...
National Physical Laboratory (NPL)
based in Teddington, UNITED KINGDOM
The National Physical Laboratory is the UK's leading metrology institute and a world leader in the science of measurement. NPL Environmental Measurements Group is a highly qualified team of scientists dedicated to providing high quality, cost effective pollution monitoring and analysis services. As an internationally recognised National Measurement Institute we offer ...
Procal - Parker Hannifin Corp
based in Peterborough, UNITED KINGDOM
Procal design and manufacture continuous emissions monitors suitable for applications that come under the European Large Combustion Plant and Waste Incineration Directives. These monitors are also used on US EPA approved sites. Certified under MCERTS for the measurement of NO, SO2, CO, CO2, H2O, and providing information as required for compliance with EN14181, Procal ...
Sierra Instruments, Inc.
based in Monterey, CALIFORNIA (USA)
Sierra Instruments is a global leader in fluid flow measurement and control for nearly 40 years and is known for its high performance flow instrumentation for gas, liquid and steam applications commonly found in the semiconductor, environmental, scientific research, petrochemical, energy, aerospace, and general manufacturing industries. With over 150 locations in over 50 ...
Codel International Ltd
based in Bakewell, UNITED KINGDOM
CODEL are world leaders in the design and manufacture of systems for continuous emissions gas monitoring (cems) and combustion control analysers, such as in-situ and extractive flue gas emissions analysers (analyzers), pollution sensors, dust monitor/particulate monitors, opacity/smoke monitors, flue gas analysers (analyzers), road tunnel atmosphere sensors and velocity ...
Cerex Monitoring Solutions, LLC.
based in Atlanta, GEORGIA (US) (USA)
Cerex specializes in the manufacture of FTIR and UVDOAS multi gas analyzers for Continuous Emission Monitoring, Process Monitoring, Ambient Air Quality Monitoring, Indoor Air Quality Monitoring, Leak Detection and Hazmat First Response. Cerex uses solid spectroscopy techniques to simultaneously monitor multiple gases in real time. Our devices are highly sensitive, ...
Opsis AB
based in Furulund, SWEDEN
Opsis AB is the leading supplier of open-path monitoring systems for air quality monitoring, continuous emissions monitoring (CEM) and process control. Opsis offers total monitoring solutions and the product line includes both monitoring hardware and data handling software. The DOAS technology, developed by Opsis, has received international approvals and provides ...
Ecotech Pty Ltd
based in Knoxfield, AUSTRALIA
Ecotech is an Australian owned company with more than 35 years experience in servicing, maintaining, designing, manufacturing, installing and commissioning ambient air and emission monitoring systems around the world. Ecotech is ISO17025 accredited for the operation and maintenance of ambient air and continuous emission monitoring systems and was first accredited in 1999. ...
Need help finding the right suppliers?
Let the XPRTs do the work for you


text:
BWG Is doing our first giveaway promotion via Instagram. Click the link to enter to win this bottle of Russel's Reserve Single Barrell Bourbon. Cheers and good luck!
Visual Recap: Spring 2015
Spring was shorter than usual this year in Western NY. Winter didn't loosen its White Walker like clutch on our souls until the last week of April. Now a week into June, Summer is already in its full stampeding stride. Festivals, events, concerts, bbq's and vacations are booking every weekend up well into September. After the worst winter of all time Spring was seemingly overlooked — but BWG most certainly didn't overlook enjoying delicious libations as we dreamt of the warm summer to come…
Follow BWG on Instagram, Twitter and Tumblr for more pics of whiskey, beer and spirits.
Thomas H. Handy Sazerac Rye 2014
Thomas H. Handy Rye (most simply referred to as "Handy" by whiskey enthusiasts) is part of the annually released Buffalo Trace Antique Collection (BTAC). BTAC whiskies consist of famous brands of the whiskey world including George T. Stagg, William Larue Weller, Sazerac 18, Eagle Rare 17 and the aforementioned Handy. This high-end rye whiskey is named after a New Orleans bartender who first used rye whiskey in the now famous Sazerac Cocktail. Handy is unfiltered and bottled at cask strength. The 2014 release clocks in at a monstrous 64.2% abv. Opposite of our last review of the entry-level Old Overholt Rye, Handy Rye is considered among the top echelon of American rye whiskies and is highly limited in distribution and availability in NY. Handy is incredibly vibrant on the nose. Sharp notes of vanilla spice, rye, and a bouquet of fresh herbs and spices. One of the best features of this whiskey is how well masked the high alcohol content is. As Handy sits in the glass it gains more and more complexity, the alcohol always a second fiddle to the flavor. On the palate we get some heat, but it compliments the spicy rye characteristics wonderfully. Handy's finish is epically warm and long. It stays with you a good while, leaving pungent traces of herbal flavors that weave themselves back through the nose as you begin your next sip. This is an excellent whiskey, but it's not obtained easily. Best local bet for a bottle of Handy is to get on as many liquor store call and waiting lists as you can and hope they get a good allotment. The 2015 Handy release should get to NY sometime between October and December. BWG rating: 94/100.
Buffalo Canalside
Couldn't agree more with our brother-blog Buffalo Dandy. #LocalPride indeed.
Originally posted on Buffalo Dandy:
This weekend the Dandy family along with some friends (and most of Buffalo it seemed) headed downtown to enjoy some fun in the sun, and finally check out the Buffalo Canalside. To say we were amazed is an understatement. The area is incredible. Full of activities for people of all ages from bike rentals, paddle boats, remote controlled sailboats, or just enjoying the waterfront, the space is immaculate and shows a truly transformed Buffalo. This is perhaps, in my opinion, one of the best examples of the "New Buffalo" everyone has been talking about and we could not be more excited for not only ourselves but for future generations of Buffalonians like our son who will hopefully have this park to enjoy for years to come. With the warmer weather finally here, we encourage everyone to head down to Canalside and enjoy everything this amazing space has to offer.
Old Overholt Rye
Old Overholt Straight Rye Whiskey is one of the least expensive whiskies on the market. It has been around for over a century (literally) and due to its relative value, is the resident house rye at most cocktail bars. Old Overholt is part of the "Old Whiskey" trio, composed of itself, Old Crow and Old Gran-Dad. These entry level whiskies are owned by the Jim Beam Brands Company. Old Overholt is a simple whiskey with rich, spicy rye grain flavors on the nose and palate. It has a bit of sweet corn and cherrywood. It's mild, wistfully smooth on the tongue and glides into medium-long finish. There's absolutely nothing outstanding about this whiskey, almost to the point of being boring to the practiced whiskey drinker. But Old Overholt's mildness and spiced flavors lend themselves very well to rye-centric cocktails — and its price point makes it an unbeatable beginner level foray into the American rye whiskey style. Available online and in store at Gates Circle Liquors for $20. BWG rating: 82/100.
Woodford Rye
BWG has been delving into rye whiskies lately and we've steadily been gaining an affinity for the spicier, more herbal characteristics of the style. Woodford Reserve is best known for bourbon but they have a mid-level rye whiskey that is recent to our market. Woodford Reserve Straight Rye is composed of 53% rye, 14% malted barley, and 33% corn. The nose starts us off with a more bourbon-like vanilla extract and oak. Some pleasant potpourri spices come through as the whiskey is allowed to open up. The palate gives lots of herbal, peppery tang — and there's a lovely maple syrup, candied bacon flavor that fades into a long peppercorn finish. This is a solid, well structured whiskey. It's not overly complex or unique, but it displays all the flavors one would look for in a nice rye. Locally available at Gates Circle Liquor for $45. BWG rating: 88/100.


text:
Visit our other Wisconsin Historical Society websites!
Refine or narrow your results by clicking any term in the gray boxes below.
Expand your results by removing any breadcrumb in a gray outlined box above. Click on the blue "X" to remove the term.
Narrow your results by entering a keyword into the search box above. Then click the "Search Within" checkbox, then click on the orange "Search" button.
If you did not find what you are looking for or you did not get any results, try changing your search term(s). See our Search Help page for more tips.
Date: 1964-08-22
Description: The Hansen monument in a Washington Island cemetery. The monument is surrounded by a cluster of Tiger Lillies which are no longer blooming.
Date: 1964-08-22 ca.
Description: A grove of cedar trees on Schoolhouse Beach. Lake Michigan is visible through the trees and a picnic table is placed near the shoreline.
Date: 1908 ca.
Description: Portrait of Senator Robert M. La Follette, Sr.
Date: 1964-08-24 ca.
Description: A line of laundry dries in the wind outside the former cottage of noted economist Thorstein Veblen. He used the small building to the right of the house as...
Community: Washington
County: Vilas
Historic Name: Everett Resort - Auto Garage
Reference Number: 146850
Description: Tree-lined entrance to the D.S. Funk property, which is visible through the trees. Two rustic fences and a lamppost flank the driveway. A sign hanging abov...
Historic Name: Anvil Lake Campground Shelter
Reference Number: 44254
Date: date unknown
Description: A studio portrait of a young Eliza Ruhamah Scidmore in a light-colored dress. She is leaning against a wood railing.
Historic Name: Everett Resort - Gas Pumphouse
Reference Number: 146851
Description: View from shoreline of two ferry boats in Lake Michigan. These two boats were once used as Mackinac Straits ferry boats and were later used as potato packi...
Description: Vice President Richard M. Nixon in a buffet line in Washington, D.C., with a group of dignitaries behind him. Immediately behind Nixon's right shoulder is...
Date: 1986-02
Description: Secretary of State George Schultz during the Reagan Administration (first on the right) testifying before the Foreign Operations Subcommittee. Questioning...
Description: A close-up view of a cedar tree root knuckle, surrounded by grass. This gnarled root was found on Mr. Jack Myers' property on Washington Island.
Date: 1986-03-06
Description: Secretary of State George Schultz during the Reagan Administration (second from the left) testifying before the Foreign Operations Subcommittee. Leaning fo...
County: La Crosse
Historic Name: Victory School / District 3 School
Reference Number: 29126
Date: 1905-02
Description: Studio portrait of Slum-Xi-Sen-Pul (Bear Blood), called Pasquale Antoine, in native dress with headdress. Part of Salishan and Flathead Tribes.
Description: Full-length studio portrait of Alexander Gordon Jr., Company K, 7th Wisconsin Infantry. He has a sword on his left side.
Description: A Vietnam veteran takes a nap on outdoor concrete stairs. He is at an anti-war demonstration and took part in the Winter Soldier Hearings.
Date: 1990
Description: Congressman David R. Obey of Wisconsin confers with Leon Panetta, Congressman from California and head of the Budget Committee (facing away from the camera...
Description: A view of the staff at the dining hall of the marine barracks. The staff wears white aprons and prepares to serve food, and tables are set on either side...
If you didn't find the material you searched for, our Library Reference Staff can help.
Call our reference desk at 608-264-6535 or email us at:
© 1996-2015 Wisconsin Historical Society, 816 State Street, Madison, WI 53706


text:
The impact of rebate freezes, the need for improved access to dental care and medicines and the expansion of private health insurance into primary health care have all been ignored in this Budget as part of the Government's covert goal to destroy Medicare and reinstate a two-tier health system, according to Tim Woodruff, Vice president, Doctors Reform Society. *** Tim Woodruff [...]READ MORE
Federal Budget Preview
With the 2015-16 Federal Budget only hours away, Croakey has prepared the following summary of the announcements already made in the health portfolio and compared these against the recommendations made by key organisations in their Federal Budget submissions. While some of the requests from the sector have already fully or partially met (indicated below) through [...]READ MORE
Are co-payments dead? Or is that the sound of Medicare's last gasps?
Dr Tim Woodruff, vice president of the Doctors Reform Society, writes: The GP co-payment is dead. So said the Prime Minister as he made his third retraction on the issue six weeks ago. However, co-payments are far from "dead, buried and cremated" – and analyses of recent policy announcements suggests we are heading ever faster [...]READ MORE
The debate we're yet to have about private health insurance
In the final instalment of The Conversation's series Private health insurance in Australia, Lesley Russell, Adjunct Associate Professor at the Menzies Centre for Health Policy at the University of Sydney, asks whether Australians need private health insurance, and what a two-tiered system means for quality, access and equity: The six previous papers in this series highlight [...]READ MORE
Private health insurance 'carrot and stick' reforms have failed – here's why
Anne-marie Boxall writes: If your workplace is anything like mine, this week's private health insurance premium increases might prompt conversations that go something like this: Can you believe our private health insurance costs $421 a month – and we are all really healthy! Some people baulk at the cost of private insurance – especially the [...]READ MORE
Why Private Health Insurance won't improve our health
As highlighted by the Parliamentary Library, the introduction of private health insurance (PHI) into primary health care raises some significant policy and regulatory challenges. In the following piece Dr Tim Senior expands on some of these issues and outlines implications of this move for population health, equity, health care costs and ultimately the future of our public health system. [...]READ MORE
Private Health Insurance in Primary Care: an overview of the issues
The GP co-payment may be 'dead, buried and cremated' but controversy over its primary health care policies is not over for the Abbott Government. The introduction of private health insurance into the primary health care sector was flagged by previous Minister for Health Peter Dutton in a number of speeches and media interviews and was [...]READ MORE
Primary Health Networks: key questions and challenges
While the media has been focussed on the Government's proposed GP co-payment, there have been potentially bigger changes to primary health care progressing behind the scenes. The transition from Medicare Locals to Primary Health Networks (PHNs) may not appear significant, as the functions and objectives of both organisations are very similar. However, there are some [...]READ MORE
Health policy reform: principles for reform & how to get there
It's hard to pick where health policy will go now under the Federal Government, with a new Minister and the likely shift (mark 3?) on a GP co-payment – if not because they accept the evidence against it but because of the politics. But there was an interesting intervention in the lead up to this [...]READ MORE
Why reform is difficult. Health ministers are in office but not in power.
Hopefully, someone, somewhere – whether in the Federal Coalition or in the ALP – is putting some serious thought into community-centred health reform. As health leaders have suggested, it's well past time to move past "thought bubbles" masquerading as policy. In the article below, experienced policy analyst John Menadue (pictured below) identifies some of the challenges [...]READ MORE


text:
Long Live Queen Hamilton
Originally Published: July 15, 2003By Dean Turcol | EXPN.com
She burst onto the scene in 1997 and was quickly dubbed the "Queen of Wakeboarding." She's the most decorated female rider ever, with credentials any action-sport athlete would love to hang around their neck. But looking ahead to the X Games, 21-year-old Tara Hamilton is dealing with a strange phenomenon for her: She has only one major victory in nearly three years.
That title at the 2002 World Championship would make most riders' careers. For Tara, it pales in comparison to the stunning ...
Read full story
You must be signed in to post a comment
You are fully responsible for the content you post. Content that includes profanity, personal attacks or antisocial behavior (such as "spamming" or "trolling"), or other inappropriate content or material will be removed. We reserve the right to block any user who violates our terms of use, including removing all content posted by that user.


text:
VIEWS: 25 PAGES: 15 CATEGORY: Computers & Internet POSTED ON: 2/1/2010 Public Domain
Ptran II - A Compiler for High Performance Fortran M. Gupta S. Midki E. Schonberg P. Sweeney K.Y. Wang M. Burke fmgupta,midki ,schnbrg,pfs,kyw,email@example.com I.B.M. T.J Watson Research Center T.J. Watson Research Center, Hawthorne P O Box 704, Yorktown Heights, NY 10598 USA Abstract ibility and e ciency are generally contradictory goals, New languages for distributed memorycompilation, and may not be achievable for the same compilation including High Performance Fortran (HPF), provide instance. However, we believe that exibility will be- user directives for partitioning data among proces- come an issue for eventual usability of this technology, sors to specify parallelism. We believe that it will so that it important to learn how to provide exibility be important for distributed memory compilers for without sacri cing too much e ciency. This exibility these languages to provide not only e ciency close is, in fact, required to support the full HPF language. to a hand-written message passing program, but also This paper describes the architecture of the Ptran exibility, so that the user is not always required to II compiler. Some of the features described (in the provide directives nor specify the number of physical present tense) have already been implemented. Oth- processors. We describe distributed-memory compi- ers described (in the future tense) are not yet imple- lation techniques for both optimization and exibil- mented. The next two subsections give background ity. on HPF, and an outline of the compiler architecture. 1.1 HPF Features 1 Introduction In HPF, the distribution and alignment of data ob- jects is speci ed with a two level mapping. At the low- The Ptran II compiler is a prototype compiler for est level of this mapping, templates1 and arrays can High Performance Fortran (HPF) 6] that we are de- be distributed onto the processor grid. Block, cyclic, veloping as a testbed for experimenting with distrib- block-cyclic and collapsed distributions are supported. uted memory compilation techniques, including auto- If the distribution for a dimension d of the template matic data partitioning and parallelization, cost mod- or array A is not collapsed, then the dimension is said eling, and global communication optimizations. Our to be partitioned. design builds on the existing work in distributed mem- Let Ak be the k'th dimension of A, and AK be the ory compilation, including 20], 13, 12], 18], 2], 16] dimension of A that is mapped onto the k'th dimen- and 7]. Our input language can be either Fortran sion of a a processor grid or template. For simplicity, 77 or Fortran 90, after all array and forall con- we assume that arrays, templates and processor grids structs have been scalarized into Fortran 77. HPF are 0-origined. The function f(AK ; i) returns the in- directives provided by the user guide the partition- dex into dimension k of the processor grid that element ing of data and computation. The compiler pro- i of AK is mapped onto. For block distributions, this duces SPMD node programs, which contain explicit function is: message-passing communication primitives. Our aim is to provide both e ciency and exibil- f(A ; i) = i K bs ity. By exibility we mean being able to compile when the number of processors, array bounds, or reaching 1 A template is an abstract index space, and can be thought data distributions is not known at compile-time. Flex- of as an array without storage the chain of alignments which map A onto its ultimate DIM B(99,49), C(49,2:100), D(100,100) alignment target. PROCESSOR P(4,5) Depending on user-supplied directives, there are TEMPLATE T(100,100) situations where precise information about the lay- DISTRIBUTE T(BLOCK,BLOCK) ONTO P out of data on processors is known, and therefore, ALIGN B(I,J) WITH T(J+1,2*I+1) the compiler should make full use of this information ALIGN C(I,J) WITH B(J-1,I) to produce e cient programs. There are also situa- DISTRIBUTE D(CYCLIC,CYCLIC) ONTO P tions, however, where little or no information is avail- able, and the compiler should not unacceptably de- Figure 1: An example of chains of alignment grade performance. Imprecise compile-time informa- tion arises from the following situations. Alignment and distributions are dynamic { they can change during program execution. Because align- where bs is the blocksize of AK , i.e. the number of ment and distribution may change, many operations elements of AK on each element of dimension k of (e.g. communication analysis and computation parti- the processor grid. In the example of Figure 1, the tioning) that are desirable to perform during compila- blocksize for T1 is 25, and the blocksize for T2 is 20. tion may sometimes have to be postponed until run- For block-cyclic distributions, time. Therefore, a run-time library to query alignment and distribution information, and act upon that infor- f(A ; i) = i mod N K bs K mation (perform communication, selectively execute a statement depending on whether the left hand side of where NK is the extent of dimension k of the processor the statement resides on the processor) is necessary. grid. In Figure 1, N1 for P is 4 and N2 is 5. Cylic Also, alignments and distributions need not be distribution is a special case of the above, where bs is1. speci ed. When not speci ed the compiler must pro- At the next level of the two level mapping, arrays vide an alignment and distribution for the object. If can be (optionally) aligned with arrays and templates alignment and distribution information is speci ed, it that have been distributed. Alignment chains can be is advisory only and may be ignored by the compiler. arbitrarily long, e.g. in Figure 1, C is aligned with B This allows the compiler to improve upon the pro- which is aligned with T which is distributed onto P. grammer's directives, and allows sophisticated opti- The e ects of alignment can also be speci ed as mizations of the data layout to be performed without the alignment function g(BK ; i) = c i + O , where violating the language standard. k is the dimension of the alignment target that BK is Finally, the distribution of a data object may not be aligned to. Thus, for B, g(B2 ; i) = 2 i + 1; that is, known at compile-time because the bounds of arrays every element i of the rst dimension of B is mapped or processor grids may not be known until run-time. to element 2 i + 1 of the second dimension of T. 1.2 Architecture of the compiler Thus, an array dimension is either collapsed or dis- The major phases of the Ptran II compiler are tributed over a processor grid dimension. The distri- shown in Figure 2. The rst phase builds distribu- bution may consist of tion and alignment information. As mentioned above, 1. replication; the distribution of a data object may be de ned via a long chain of alignments. This phase collapses this 2. mapping to a constant processor position; chain into a canonical form, and stores the resulting information in the HPF symbol table. The canonical 3. or partitioned. form for partitioned data is the distribution function If partitioned, the mapping can be speci ed with a discussed in Section 1.1. mapping function of the form Next loop transformations are performed. Loop transformations enable more parallelism to be uncov- ered and to allow more e cient communication and + f(AK ; i) = s i bs O mod NK ] computation partitioning to be performed. This phase is discussed in Section 6. The term mod NK is needed only for the cyclic and The next phase performs automatic data partition- block-cyclic distributions. The term s i+O for array ing. This must be performed for scalars and all arrays A is found by composing the alignment functions of for which distribution information was not speci ed. ing is assumed to occur in the global data space. The front end program is translated into the local address space very late in the SPMDization process. This is to simplify Build analysis { for example dependence analysis must be performed in the global space. Distribution Information Loop Transformations 2 Automatic Data Partitioning While HPF assigns the primary responsibility of de- termining data distribution to the programmer, we be- Cost Data HPF lieve a good HPF compiler should also perform auto- Estimator Partitioning Symbol Table matic data partitioning for the following reasons: 1. It is the compiler which generates and optimizes Communication communication, hiding those details from the pro- grammer. It should warn the programmer if cer- Analysis and Code Generation tain directives are expected to lead to bad per- formance. Our experience has shown that a com- Computation piler can often make intelligent decisions about Partitioning data partitioning for programs with regular com- back end putations 10, 7]. 2. The programmer may not specify partitioning for Figure 2: The architecture of the Ptran II compiler all arrays, since the directives are optional. The compiler must map those arrays which are not partitioned by the directives. A cost estimator is used to guide the selection of var- 3. The compiler must always determine the distrib- ious parameters of data partitioning. This phase is ution of array temporaries generated during the covered in more detail in Section 2. compilation process. The communication analysis phase selects the com- munication operation to be used (e.g. many-to-many In addition, the compiler is required to assign owner- multicast or send/receive), and determines at what ship to each scalar variable. level in the loop nest the communication operation can 2.1 Distribution of Arrays be performed. Numerous communication optimiza- For partitioning arrays, Ptran II will use an ap- tions are also performed by this pass. This phase is proach similar to the Paradigm compiler 10, 7], discussed in Section 3. which we shall describe brie y. The data partition- Next, communication code generation inserts code ing decisions are made in a number of distinct passes for allocating bu ers and performing communication through the program, as shown in Figure 3. The operations into the intermediate form of the program align dimension pass determines the mutual align- maintained by the compiler. Also, this phase com- ment among array dimensions, and aligns each ar- poses communication operations across a processor ray dimension with a unique dimension of a template. grid's dimensions. This phase is discussed in Section The align stride-offset pass determines the stride 4. and the o set in each alignment speci cation, i.e., for At this point, the program is converted into an a speci cation of the form \A(i) ALIGNED WITH SPMD program by the computation partitioner, which T(c i + O )", where A is an array and T is a tem- is guided by the owner computes rule. A combination plate, it determines the values of c and O . The of modifying the bounds of loops and the insertion block-cyclic pass determines for each template di- of statement guards is used to restrict execution of mension, whether it should be distributed in a blocked statements to the appropriate processor. This pass is or cyclic manner. The num-procs pass determines the discussed in greater detail in Section 5. number of processors across which each template di- During most of compilation, addressing and index- mension should be distributed. Align Dimension Detector Computational Cost Estimator Align Stride-Offset Driver Communication Cost Estimator Solver Block/Cyclic Num-Procs Figure 3: Overview of automatic data partitioning in Ptran II In each pass, the detector module rst analyzes We shall modify the above procedure for automatic array references in each statement, and identi es de- data partitioning to incorporate the data distribution sirable requirements on the relevant distribution pa- directives supplied by the user. Such directives may rameter of each array. These desirable requirements be regarded as constraints with very high values of are referred to as constraints 8]. In order to han- quality measures, so that they are always honored. dle con icting requirements on data distribution from We shall also provide an option where the compiler di erent parts of the program, the driver module de- might warn the programmer about expected bad per- termines a quality measure for each constraint. The formance resulting from a directive, if the program is quality measure captures the importance of each con- amenable to static analysis. straint with respect to the performance of the pro- gram, and is obtained by invoking the communication 2.2 Distribution of Scalars cost estimator and/or the computational cost estima- Many distributed memory compilers simply repli- tor. Once all the constraints and their quality mea- cate scalars on all processors. However, that can po- sures have been recorded, the solver determines the tentially lead to a great deal of unnecessary commu- value of the corresponding data distribution parame- nication for a scalar assignment. Ptran II chooses ter. Essentially, the solver obtains an approximate between replication and non-replicated alignment of solution to an optimization problem, where the objec- a scalar variable. The ownership information for the tive is to minimize the execution time of the target second kind of scalar is speci ed in terms of an align- data-parallel program. ment relationship with a non-replicated variable. This For example, in the block-cyclic pass, the subsumes privatization, and also includes the possibil- detector records a constraint favoring block distri- ity of ownership by a single processor. Ptran II uses bution of a template dimension if an array dimension the static single assignment (SSA) representation 4] aligned to it is involved in a nearest-neighbor commu- to associate a separate ownership decision with each nication. It records a constraint favoring cyclic dis- assignment to a scalar. tribution if the computations across an aligned array Initially, all scalars are assumed to be replicated. dimension are better load balanced with cyclic distrib- For each scalar de nition, the compiler checks if the ution. In the above two cases, the communication and scalar can be privatized with respect to the innermost the computational cost estimators are respectively in- loop in which the assignment appears. If the analy- voked to record the quality measure, which is the dif- sis shows that the scalar can be privatized and if the ference between the estimated costs when the array given de nition is not live outside the loop, the scalar dimension is given a block (cyclic) distribution and variable is marked as a potential candidate for non- when it is given a cyclic (block) distribution. Once replicated alignment. If such a scalar is required to these constraints are recorded, the solver compares be available on all processors (for which the compiler the sum of quality measures favoring block distrib- checks if that variable is used as a loop bound or in- ution and those favoring cyclic distribution for each side a subscript expression for an array reference), the template dimension, and chooses the method of parti- variable is actually replicated. Otherwise, the com- tioning corresponding to the higher value. piler identi es a reference corresponding to a parti- !HPF$ DISTRIBUTE (BLOCK) :: A !HPF$ PROCESSORS P(10,10) DO I = 1, N !HPF$ ALIGN B(I) WITH A(I,1) ::: !HPF$ DISTRIBUTE (BLOCK, BLOCK) :: A T = A(K) DO I = 1, N A(K) = A(I) S = B(I) : 1 s A(I) = T DO J = 1, N ENDDO S = S + A(I,J)) : 2 s ENDDO B(I) = S : 3 s Figure 4: Non-replicated alignment of a scalar ENDDO Figure 5: Scalar assignment in a reduction computa- tioned variable in the given assignment statement to tion record the alignment information for the scalar. For example, in the program segment shown in Figure 4, the variable T is aligned with A(K). Clearly, if all data items on the rhs of the assignment are themselves reference. Later, we shall describe some global opti- replicated, the scalar is e ectively replicated too. mizations that can be used to reduce the overall cost Any scalar computed in a reduction operation (such of communication. as sum) being carried out across a processor grid di- 3.1 Analysis for Single Reference mension is given special treatment. The de nitions For each read reference in the program, the com- corresponding to its initialization and the reduction munication analysis module rst determines whether computation are handled together, and there is an- it requires interprocessor communication. If commu- other copy created of that variable, that yields two nication is needed, it determines (i) the placement of versions. The rst version is privatized, and repre- communication, and (ii) the communication primitive sents the local reduction computation performed by that will carry out the data movement in each grid each processor. The second version stores the result dimension. The code generator then identi es the of global reduction, and may either be replicated or processors and data elements that need to participate given a non-replicated alignment based on the proce- in communication, and composes the primitives over dure discussed above. di erent grid dimensions. For example, consider the program shown in Fig- In order to facilitate communication analysis, ure 5. Ptran II creates an additional version of S, Ptran II classi es each array subscript into one referred to as $S. Due to reduction taking place across of the following types: constant, single-index the second processor grid dimension, $S is privatized (a ne function of a single loop induction variable), along that dimension, and aligned with A(I,J). The multiple-index (a ne function of more than one scalar S is determined to be privatizable with respect loop induction variable), or complex. Information is to the I-loop, and is aligned with B(I). This enables also kept about its variation-level, i.e., the inner- statements s1 and s3 to be executed without any com- most loop in which the subscript changes values. We munication, and the use of a reduction communication refer to a subscript associated with a distributed di- primitive in the second grid dimension. mension in an array reference as a sub-reference. 3.1.1 Placement of Communication 3 Communication Analysis Consider a read reference appearing in a statement Ptran II uses analysis based on 9] to determine the inside one or more loops. If possible, the communi- communication required for each reference in the pro- cation for that reference should be moved outside of gram. We have extended the analysis described in 9] loops to enable message vectorization, which combines to handle scalars, and to incorporate additional possi- a sequence of messages on individual array elements bilities regarding the distribution of array dimensions, into a single message 13, 20]. The compiler exam- namely, replication and mapping to a constant proces- ines all incoming ow dependences to the reference, sor position. We shall rst describe how communica- and the innermost loop carrying any of those depen- tion requirements are determined individually for each dences identi es the loop from which communication cannot legally be moved outside. Thus, using depen- varying in a communication-independent loop. When dence analysis, the compiler determines the outermost both the sub-references are of the type single-index, loop level at which communication may be placed for Ptran II performs certain tests that are described be- a given reference. low. The extent to which messages are actually com- bined can next be controlled by stripmining loops, Synchronous Properties of Sub-references whenever necessary. If any sub-reference is of the Consider two sub-references, s1 = 1 i + 1 , and type complex, the compiler does not allow commu- s2 = 2 i + 2 . Let the distribution functions nication to be taken outside the loop correspond- of the corresponding array dimensions be f(d1 ; i) = ing to the variation-level of that sub-reference, to avoid excessive and potentially extraneous communi- b c1 i+1O 1 c modNK ], and f(d2 ; i) = b c2 i+2O 2 c mod b b cation. Future versions of Ptran II will use the run- NK ]. time compilation techniques developed by Saltz et al. The sub-reference s1 is said to be strictly synchro- 19]. In the presence of a sub-reference of the type nous with s2 , with respect to their common loop of multiple-index as well, Ptran II does not combine variation, if both sub-references are mapped to the communication with respect to all the loops. Tech- same processor position for all iterations of the loop. niques like tiling can be used in such cases to allow This property is satis ed if either of the following sets greater combining of messages, without the overhead of conditions hold (the proof is given in 7]): of extraneous communication 9]. (i) (c1 1 )=b1 = (c2 2)=b2 , and 3.1.2 Identi cation of Communication Primi- (ii) (c1 1 ? O 1)=b1 = (c2 2 ? O 2 )=b2, tive or We rst describe how for each grid dimension, com- (i) b1 = m c1 1, (ii) b2 = m c2 2, and (iii) munication primitives are identi ed for a rhs refer- b(c1 1 ? O 1)=(c1 1)c = b(c2 2 ? O 2 )=(c2 ence in an assignment statement. By the owner com- 2)c, putes rule, data is sent to the owner of the lhs refer- where m is a positive integer. ence. The rst step is to identify the pairs of sub- references in the lhs and the rhs reference corre- The sub-reference s1 is said to be k-synchronous sponding to aligned dimensions. For a pair of aligned (k being an integer) with s2 , with respect to their sub-references, a comparison of variation-levels common loop of variation, if at every iteration point identi es the innermost loop in which data move- where s1 crosses a processor boundary, s2 crosses the ment for that grid dimension takes place. If it is a next kth processor boundary. The conditions to check communication-independent loop, i.e., if communica- for this property are obtained in a similar manner, as tion is placed outside that loop, then data movement shown below: over the grid dimension for the entire loop can be im- plemented using a collective communication primitive. (i) (c1 1 )=b1 = k ((c2 2 )=b2), and Otherwise, the primitive used is a send-receive. The (ii) (c1 1 ? O 1 )=b1 = k ((c2 2 ? O 2 )=b2)+l, use of collective communication, such as broadcast and where l is an integer, reduction, can signi cantly improve the performance or on a massively parallel machine, and also leads to more concise and high-level code. (i) b1 = m c1 1 , (ii) b2 = k m c2 2, The choice of collective communication primitive and (iii) b(c1 1 ? O 1)=(c1 1)c = b(c2 2 ? depends on the type of sub-references and on the dis- O 2 )=(c2 2 )c + m l, tribution parameters of the array dimensions. We where m and l are integers, and m > 0. shall only describe the most common case when the aligned array dimensions are distributed in a block, The \boundary-communication" test helps detect cyclic, or block-cyclic manner. Using similar analysis, nearest-neighbor communication. It checks for the fol- Ptran II is also able to handle other cases, where lowing conditions: either of the array dimensions may be replicated or mapped to a constant processor position. Table 1 lists 1. c1 1 =b1 = c2 2=b2. the selected communication primitive, given a pair of sub-references of type constant or single-index, 2. j(c1 1 ? O 1 )=b1 ? (c2 2 ? O 2)=b2 j 1. LHS RHS Conditions Tested Commn. Primitive single-index constant default broadcast constant single-index 1. reduction op reduce 2. default gather single-index (s1 ) single-index (s2 ) 1. s1 strictly synch. s2 no communication (same level) 2. s1 1-synch. s2 (parallel) send-receives 3. boundary-comm check shift 4. default sequence of send-receives Table 1: Collective communication for single pair of sub-references Coupled Sub-references In the presence of cou- pled sub-references (more than one sub-reference of !HPF$ ALIGN B(I) WITH A(I) an array, varying in a single loop), the analysis shown !HPF$ ALIGN C(I) WITH A(I) in Table 1 is not su cient. In such cases, the com- !HPF$ DISTRIBUTE (BLOCK) :: A piler has to identify the relationship between simulta- DO I = 1, N neous movement of sub-references in each of the grid IF (C(I) .NE. 0.0) THEN dimensions. An extension of the analysis shown here A(I) = A(I) / C(I) helps detect when there are non-overlapping groups of B(I) = 2 * B(I) processors (spanning multiple grid dimensions) over END IF which collective communications may be carried out ENDDO 9]. However, due to the complexity of forming proces- sor groups, Ptran II currently uses send-receive Figure 6: Communication analysis for IF statement between processor pairs in those cases. 3.1.3 Conditional Statements 3.2 Global Optimizations A naive distributed memory compiler would force the So far we have described the communication analy- execution of a conditional statement on every proces- sis for a single reference, and various optimizations sor. In such a case, data needed for evaluating the that are performed with regard to that communica- condition has to be broadcast to all processors, if it is tion. We now describe some global optimizations that not already replicated. As an optimization, Ptran II help reduce the overall cost of communication for the examines all statements inside the scope of the condi- program. tional, and obtains a pseudo-lhs reference which \cov- ers" all other lhs references in terms of mapping to Message Coalescing processors. A test for checking whether lhs1 \covers" lhs2 is equivalent to checking whether an assignment Message coalescing 13] involves combining messages statement of the form lhs1 = lhs2 requires any com- between the same pair of processors, that correspond munication under the owner computes rule. In the to di erent references. Ptran II can detect the pos- worst case when the conditional statement has to be sibility of coalescing messages corresponding to refer- executed on all processors, the pseudo-lhs reference ences even in di erent statements by symbolic analy- is to a dummy scalar variable replicated on all proces- sis before the actual sets of processors and the array sors. Once the pseudo-lhs reference is obtained for a sections involved are identi ed. Consider two sets of conditional statement, communication analysis is car- communications that are placed at the same program ried out using exactly the same procedure as that for point (for instance, in the preheader of a loop). Let the an assignment statement. communications correspond to rhs references r1 and For example, in Figure 6, Ptran II obtains \A(I)" r2, appearing in statements with lhs references l1 and as the pseudo-lhs reference for the if statement, and l2 respectively. Clearly, all messages corresponding to further analysis shows that no communication is nec- the two references may be combined if r1 and r2, and essary for the reference to C(I). similarly l1 and l2 , are mapped to the same processors, for all instances of those references corresponding to The above sets are used to determine the communi- di erent loop iterations. The compiler detects this cation required between any arbitrary pair of proces- in constant time by carrying out an extended version sors, with the owner computes rule. The rst step is of the test for strictly-synchronous property between to determine the iterations in which communication all pairs of sub-references corresponding to aligned di- takes place. mensions. Send Iteration Set: SIS(l; r; L; p; q) { set of iter- Eliminating Redundant Communication ations of L for which the processor at position p has to send data to the processor at position q, Most distributed memory compilers generate commu- corresponding to the sub-references l (lhs) and r nication for data that is not owned by the processor (rhs). It is computed using the following equa- which needs it. If the data is already available locally tion: as a result of prior communication, and if it has not been modi ed by its owner, the compiler can elimi- SIS(l; r; L; p; q) = LIS(r; L; p) \ LIS(l; L; q) nate that communication, and make the processor use its local copy. Ptran II will use a data- ow analy- Receive Iteration Set: RIS(l; r; L; p; q) { set of it- sis framework that has been developed to keep track erations of L for which the processor at position of the availability of data at processors, at di erent p has to receive data from processor at position q, points in the program 11]. Such a framework can corresponding to the sub-references l (lhs) and r also be used to relax the \owner sends" rule, so that (rhs). It is computed using the following equa- any processor having a valid copy of data, not just its tion: owner, can be the sender. RIS(l; r; L; p; q) = LIS(l; L; p) \ LIS(r; L; q) 4 Communication Code Gener- The array subscript positions corresponding to the ation data being sent/received, referred to as data sets, can now be determined by applying the subscripting func- Because a program's data is distributed among proces- tion of the rhs sub-reference to the above iteration sors, data movement may be required to perform com- sets. putation. The communication code generation phase When the data and processor sets can be com- of the Ptran II compiler performs data movement puted at compile-time for a reference to a parti- by generating calls to communication library routines. tioned array, the communication code generator com- The input to the communication code generator is the poses the communication primitives across the corre- communication analysis information and an HPF pro- sponding processor grid's dimensions. Let the tuple, gram with data distribution annotations. The output (Ps; d(ps); Pr ; d(pr )), denote the information that we is the HPF program augmented with communication will use to compose communication where Ps and Pr code. are the send and receive processor sets and d(ps) and If possible, the communication code generator at- d(pr ) are the send and receive data sets for that di- tempts to determine, at compile-time, the actual data mension. For a two dimensional array reference, such items that need to be communicated between proces- as B in Figure 7 the composition would be as follows sors, and for each data item the processor that owns if communication in the rst dimension occurs rst: it and the processors that need it. This can be ac- 1:(hPs1; Ps2i; d(hp1; p2i); hPr1; Ps2i; d(hp1; p2i)) s s r s complished by computing the following information for each dimension of an array reference. These com- 2:(hPr1; Ps2i; d(hp1; p2i); hPr1; Pr2i; d(hp1; p2i)) r s r r putations are a simple extension of Koelbel's work 16]. where P n refers to the nth dimension of P. This ap- Processor Set: PS(sr; L) - set of processor po- proach works for any multi-dimensional communica- sitions along a grid dimension traversed by the tion. sub-reference sr in loop L. Although executing the communication operations in any order provides correct results, the ordering may Local Iteration Set: LIS(sr; L; p) { set of itera- have signi cant impact on the cost of communication. tions of loop L in which the sub-reference sr is The communication code generator orders communi- mapped to processor position p. cation primitives to rst reduce message length and REAL B(3,18) Processor Grid !HPF$ PROCESSORS P(3,3) !HPF$ DISTRIBUTE B(BLOCK, BLOCK) DO I = 1, N B(1,1) = B(1,1) + B(N,I) END DO Figure 7: An example to illustrate communication composition across a processor grid's dimensions. send receive then to reduce the number of processors involved in the communication 7, 18]. We now illustrate the com- position procedure for an example that is presented in Figure 7 to demonstrate how the order of communi- cation primitives e ects the cost of communication. The array reference B(N; I) requires communication reduce along both dimensions of the processor grid P. The processor and data sets for each dimension are: Figure 8: Communication composition (2; B1(N); 0; B1(1)) for the send-receive communication in the rst dimen- sion, and were to be executed before the reduction, the mes- sage length for the send-receive communication would (0 : 2; B2(p2 BSB2 + 1 : (p2 + 1) BSB2 ); 0; B2(1)) increase to BSB2 , and the number of (parallel) send- for the reduction communication in the second dimen- receives would increase to the extent of a row in the sion, where pn is a processor's identi cation number in processor grid. the nth dimension of the processor grid, and BSB2 is This approach of composing communication opera- the blocksize of the 2nd dimension of B. The compo- tions across a processor grid's dimensions is conceptu- sition when communication in the second dimension ally clean, and helps to simplify communication code occurs rst is: generation. Furthermore, our table-driven implemen- tation will allow this approach to be easily modi ed (h2; 0 : 2i; B(N; p2 BSB2 + 1 : (p2 + 1) BSB2 ); to optimize for other characteristics, such as reducing network tra c. h2; 0i; buf(B(N; 1))) When the actual data items that need to be com- for the reduction, and municated or the processors that are involved in the communication are not known at compile-time due (h2; 0i; buf(B(N; 1))i; h0; 0i; B(1; 1)) to missing information, such as an unknown distri- bution of an array reference or a complex subscript, for the send-receive where buf(B(N; 1)) denotes the this information must be computed at run-time. In bu er that contains the value generated by the re- this case, the communication code generator gener- duction. The communication code generator allocates ates calls to a di erent set of communication library and manages communication bu ers. Figure 8 illus- routines that perform the computations at run-time trates the data movement along the dimensions of the and then execute the appropriate communication 5]. processor grid P. Currently, these library routines do not provide in- Choosing to execute the reduction before the send- spector/executor functionality; however, in such cases, receive communication in the program segment pre- tests for the ownership of data are used to ensure the sented in Figure 7 results in unit message length for correct communication of data. In addition, Ptran the send/receive communication and requires only one II relies on run-time support to handle block-cyclic send-receive to occur. If, instead, the send-receive distributions. 5 Computation partitioning that P(u; v) owns must map onto both a row and col- umn owned by P(u; v). Stated succinctly, the values of When compiling any language for a parallel machine, a i that accesses these elements of A are the intersection problem that arises is what processors should perform of the sets of values that access a column and a row. what operations. Solving this problem for a particular Thus, the LIS of the I loop for processor P(u; v) is program is the task of the computation partitioner. In solving the problem, the computation partitioner L = LIS(A1 ; I; u) \ LIS(A2 ; I; v) (1) is guided by the owner computes rule: the processor 5.2 Multiple Statements in the Body of that owns the modi ed data item (e.g. the owner the Loop of the left hand side of an assignment statement) in- If multiple statements are present in a loop, then volved in an operation performs the operation. The the loop must execute all iterations needed by every input to the computation partitioner is an HPF pro- statement. This is the union of the iteration sets of gram with data distribution annotations, the output each statement in the loop intersected with the origi- is a Single Program Multiple Data (SPMD) program, nal iteration set of the loop. Thus, if the statement: where each node processor executes only those opera- tions in the original HPF program that it should fol- B(3*I - 2, I) = : : : lowing the owner computes rule. In converting the is added to the above loop, where B is aligned with A, HPF program to an SPMD program, a major goal is the LIS for the I loop would be to allow innermost loops to run e ciently on a node processor. L (LIS(B1 ; I; u) \ LIS(B2 ; I; v)) A brute force method of performing computation partitioning is surrounding every statement in the pro- using L from Equation 1. gram with a test to determine if that processor owns 5.3 References that have an identical ef- the left hand side reference, i.e. an IOWN( ) test. fect on the loop bounds This strategy has the advantage of needing to know Consider the loop: nothing about the distribution and alignment of the , but every processor must execute the entire loop, PROCESSOR P(4) and every iteration of the loop contains an additional DISTRIBUTE A(BLOCK) branch. Intelligent computation partitioning can be ALIGN B(I) with A(I-1) thought of as an attempt to replace IOWN tests with DO I = 2, 100 ::: less expensive techniques. A(I) = ::: One way to do this is by shrinking the bounds of the B(I+1) = END DO loop 16] using the LIS components ilow and ihigh (Sec- tion 4). Loop bound shrinking allows the IOWN test Naively applying the strategy above would require the to be implicitly enforced by executing on each proces- iteration set for the I loop to be the union of the LIS's sor only those iterations of the loop whose statements for both references. Because the subscripts for A and B would be executed with using the IOWN test. are strictly synchronous (see Section 3.1.2), the LIS's 5.1 Coupled Subscripts for A and B are equal. Therefore the LIS for either A Coupled subscripts are subscripts for di erent di- or B can be used to specify the iteration set for the I mensions of an array that reference the same loop in- loop. dex variable. The e ects of all subscripts in the refer- 5.4 The e ects of communication on com- ence must be taken into account when shrinking the putation partitioning loop. Consider for example: In general, computation partitioning only looks at the left hand sides of statements, since these contain PROCESSOR P(4,4) the modi ed operand. However, it is necessary that DISTRIBUTE A(BLOCK,BLOCK) statements to send data also be executed. For exam- DO I = 2, 100 ple, in the loop nest: A(I+1,I-1) = ::: END DO DISTRIBUTE A(BLOCK) ON P(1024) DISTRIBUTE B(BLOCK) ON P(1024) A processor P(u; v) owns an element A(r; c) if and only DO I = 1, N if owns the part of column c that contains row r. This A(I) = A(I-1) + ::: implies that the values of i that access elements of A END DO elements of the A array must be communicated. Ex- 5.6 Fusing loops and guard conditions actly what iterations are needed is determined by One condition (in the Ptran II compiler) for fus- the communication code generation phase and is ex- ing a pair of adjacent loops is that the bounds of the pressed as an LIS. This LIS is unioned with the LIS loops be identical. The bounds that result from com- of the left hand side reference to A formed using the putation partitioning can be very complicated even if owner computes rule. The union of these two LIS's is the original loop bounds were constant expressions. the the iteration set of the I loop. It is therefore desirable that the computation parti- 5.5 When guard statements are needed tioner identify the bounds of loops (formed by com- putation partitioning) that are identical. To do this As mentioned previously, loop bounds shrinking is requires the same analysis of the synchronous property an optimization to reduce the number of guard state- used in Section 3.1.2 to determine if two statements ments that must be used within a loop. If the bounds have the same iteration sets. If the bounds of two or of a shrunk loop do not precisely specify the iteration more loops are identical, the upper and lower bounds set of a reference, then it will still be necessary to add will be found and saved to compiler temporaries, and guard statements. This occurs whenever: these compiler temporaries will then be used as the upper/lower bounds expression for each of the loops. The LIS for the loop nest is not equal to the LIS For example, the loop: for the reference; PROCESSOR P(4) The LIS for the statement cannot be accurately DISTRIBUTE A(BLOCK) computed because of a complicated subscript or ALIGN B(I) WITH A(I-1) distribution. DIMENSION A(100) DO I = 2, 100 If the iteration set for the statement can be determined A(I) = ::: exactly, it is only necessary to determine if the current B(I+1) = ::: loop iteration is a member of the iteration set for the END DO statement. If the bounds for the statement cannot be after loop distribution and computation partitioning determined exactly (e.g. for non-linear subscripts) an will become: IOWN test is necessary. We attempt to move guards statements to the out- lb = max(pid*25+1, 2) ermost loop where the guard is invariant. For example, ub = min((pid+1)*25, 100) consider the loop nest: DO I = lb, ub A(I) = ::: DISTRIBUTE A(BLOCK,BLOCK) ONTO P(1024,1024) END DO DO I = 1, N DO I = lb, ub A(5,B(I)) = ::: : 1 s B(I+1) = ::: END DO END DO Statement s1 should only be executed on proces- and the bounds will cease to be a concern as far as sor P(u; v) if the mapping functions (Section 1.1) loop fusion is concerned. f(A1 ; 5) = u and f(A2 ; B(I)) = v. The rst test is Similarly, if adjacent statements have identical it- invariant to the loop, and is hoisted out of the loop, eration sets, the guard conditions for the statements whereas the second test, being dependent on the value are identical, and the guard IF statements surround- of I, must remain in the loop. This results in the code: ing each of the statements can be replaced by a sin- gle guard IF surrounding all of the statements. This f( ; 5) = u IF ( A1 ) THEN transformation will enable better uniprocessor opti- DO I = 1, N mizations to take place. IF ( A2f( ; B(I)) = v) THEN A(5,B(I)) = END IF ::: : s 1 6 Loop Transformations END DO We believe that good HPF compilers need loop trans- END IF formations to enhance performance. Some of the loop transformations will be performed before automatic data partitioning and communication analysis, while DO J = 1, N others will be performed just before code generation, DO I = 1, N as explained below. In contrast to the SUIF 1] com- A(I, J) = A(I-1, J) + D(I) piler, we have made the following design decisions: END DO END DO We assume that uniprocessor optimizations to improve cache locality are performed by the Better Loop Ordering: back-end, after the SPMD program is produced. Such uniprocessor optimizations may include ad- DO I = 1, N ditional loop blocking, interchange, and loop fu- DO J = 1, N sion. This division of functionality leads to a sim- A(I, J) = A(I-1, J) + D(I) pler, more modular design than if both uniproces- END DO sor and multiprocessor optimizations are consid- END DO ered together. However, it is important that the SPMDizer does not inhibit subsequent uniproces- sor optimization. This requires some care in code Figure 9: Example Program For Message Vectoriza- generation, so that opportunities for such trans- tion formations as loop fusion are not inhibited. We do not use any loop skewing transformations Each assignment statement requires a separate guard, in the SPMDizer. Skewing is useful for the fol- and there is a communication inserted between the lowing purposes: two statements. Loop distribution results in the code: 1. Wavefronting. 2. To achieve outer-loop parallelism. DO I = 1,N A(I) = ::: However, for distributed-memory SPMD pro- END DO grams, wavefronting is achieved through data <communication> communication, so that loop skewing is not DO I = 1, N necessary. Furthermore, outer-loop parallelism B(I+1) = A(I) ::: achieved through loop skewing results in non- END DO rectilinear data partitions, which are not cur- rently in HPF. Therefore, we do not consider loop skewing to be particularly useful in our current In the resulting program, each loop is completely par- system design. allel, with a single communication between the two loops. Further, the guard needed for each of the state- Our SPMDizer will include loop distribution, inter- ments is easily eliminated by the computation parti- change, and blocking to both increase parallelism and tioning module. reduce communication costs. Even though the SPMDizer performs full loop dis- 6.1 Loop Distribution tribution, no loop fusion is performed. We rely on Full loop distribution will be performed before au- subsequent uniprocessor optimizations to reassemble tomatic data partitioning. Loop distribution maxi- the program loop structure, according to locality con- mizes the number and nesting depth of perfect loop siderations. nests, thereby exposing more opportunities for op- 6.2 Loop Permutations and Stripmining timization. For distributed memory compilation, In addition to loop distribution, loop permutations loop distribution is especially useful for optimizations can also be used to enable vectorization at the out- like message vectorization and eliminating statement ermost legal loop levels. For example, in the code of guards. For example, consider the following program Figure 9, if A is partitioned (BLOCK, *), di erent it- segment, where A and B are aligned with each other: erations of the I loop require communication, which DO I = 1, N cannot be moved out of the loop. However, if I and A(I) = ::: J are interchanged, the J loop is communication-free, B(I+1) = A(I) ::: and each communication operation can consist of an END DO entire row of A. This reduces the number of messages. Preliminary steps for permuting loops to vectorize messages include: DO J0 = 1, N, b DO I0 = ; myloA myhiA b0 ; 1. Identify loops which carry communication- (Vectorized Communication Level) inducing dependences. ; ( DO I = I0 min myhiA I0 b0) ; + ; DO J = J0 min N J0 + b) (; 2. Move parallelizable loops inwards to enable mes- A(I, J) = A(I-1, J) + D(J) sage vectorization. END DO When the distribution of arrays is not speci ed by user END DO directives, we assume that each array dimension has END DO been distributed for this step by the compiler. In general, there may be more than a single legal loop con guration with inner parallelizable loops, and Figure 10: The Canonical Loop Form there should be an algorithm for selecting the best con guration. To illustrate, consider the program: resulting sectioning J0 loop is interchanged with the I DO J = 1, N loop: DO I = 1,N A(I, J) = f(A(I-1, J-1)) DO J0 = 1, N, b END DO = DO I myloA myhiA; END DO (Vectorized Communication Level) DO J = J0 , min(N, J0 + b) If A is partitioned (BLOCK, *), there is a A(I, J) = A(I-1, J) + D(I) communication-inducing dependence from A(I, J) to END DO A(I-1, J-1), which has dependence vector (1; 1) car- END DO ried by the J loop. Communication can be moved END DO outside the I loop, but there is no signi cant vector- ization. Because of the way that A is partitioned, there The blocksize b must be chosen so that the cost of the will be only a single element of A communicated at computation for the inner loop is balanced with the each iteration of the J loop. If the J loop and I loop (vectorized) communication overhead. Note that the are interchanged, the J loop is parallelizable, and an the message length is proportional to b. entire row of A can be communicated at a time. The al- So far, we have not considered the interaction of gorithm for determining the best loop ordering should both cache locality and message vectorization trans- therefore compare legal loop permutations, where the formations on the loop nest of Figure 9. In this loop ability to perform message vectorization is the crite- nest, the I loop carries locality, so that it is important rion for choosing the best. for uniprocessor performance for the I loop to be in Message vectorization is not unconditionally good, the innermost position. Therefore, for the canonical since it sometimes can prohibit parallelism, when a loop form (Figure 10), the I loop is also stripmined, processor executing a later iteration must wait for an so that subsequent uniprocess cache optimizations are earlier iteration to nish. Therefore, after the above not inhibited by the generated communication code. loop permutation transformation has been performed, loop blocking should be performed so that the compu- tation can be pipelined. For this transformation, the inner parallelizable loops, which enable the message 7 Related work vectorization, are stripmined, and the resulting sec- There are several groups which have looked at the tioning loops are permuted outside the loop carrying problem of compiling for distributed memory. A sum- the communication-inducing dependence. mary of this work can be found in 21]. Consider again the interchanged form of the loop The SUPERB Compiler 20] and Fortran D 13] nest in Figure 9. Any processor that depends on re- share some of the goals of our project. All accept dis- ceiving values of A(I-1, J) must wait until all itera- tribution and alignment information from the users, tions of the J loop complete, so that there is no par- all target distributed memory machines, and all auto- allelism. The parallel J loop is stripmined, and the matically generate communication. SUPERB is the earliest of these and di ers from formance and uniprocessor performance in di erent Ptran II in that only block distributions are sup- phases. SUIF also performs loop skewing optimiza- ported. Message vectorization, and overlay areas are tions, which are not included in the current PTRAN supported, as in Ptran II. Elimination of identi- II design. cal redundant communication is also supported. The Ptran II project is investigating more general opti- mizations for the elimination of redundant communi- 8 Conclusion cation. The second generation SUPERB compiler will use expert system techniques to do automatic data The design of a SPMDizer for HPF has been described. partitioning. The current implementation status is that programs Fortran D allows cyclic and block cyclic distribu- with block distributions are being successfully com- tions. Currently, partitioning of only a single dimen- piled and executed. The compilation does not depend sion of an array is supported, although plans call for on the number of processors. However, if the number supporting the partitioning of more than one dimen- of processors is known at compile-time, better code sion. As with SUPERB, message vectorization, over- will be generated. With this base system in place, lay areas and elimination of some redundant commu- we plan to incorporate the automatic data partition- nication is supported. Fortran D converts addressing ing and communication optimizations that have been into local space very early in the compilation process, presented. and all analysis is performed using local space. Ptran II stays in global space as long as possible { doing so allows symbolic information to be maintained that is References true across the processor grid, information that can 1] S. Amarasinghe, J. Anderson, M. Lam, and be resolved at run-time. Ptran II allows arrays to be A. Lim. An overview of a compiler for scalable partitioned on all dimensions. parallel machines. In Proc. Sixth Annual Work- Our use of the iteration sets for generating commu- shop on Languages and Compilers for Parallel nication and partitioning the computation draws on Computing, Portland, Oregon, August 1993. the work of Koebel on the Kali project 16]. Li and Chen have introduced the component a n- 2] Z. Bozkus, A. Choudhary, G. Fox, T. Haupt, ity graph framework 17] for aligning array dimensions, and S. Ranka. A compilation approach for For- which we use with improved cost estimation. They tran 90D/HPF compilers on distributed memory have also described techniques for automatically gen- MIMD computers. In Proc. Sixth Annual Work- erating communication. We have extended that work shop on Languages and Compilers for Parallel by incorporating dependence information, and by in- Computing, Portland, Oregon, August 1993. troducing the notion of synchronous properties be- tween sub-references, that allows our compiler to per- 3] S. Chatterjee, J. R. Gilbert, R. Schreiber, and form a more accurate communication analysis. S.-H. Teng. Automatic array alignment in data- Knobe, Lukas and Steele's 14, 15] preferences are parallel programs. In Proc. Twentieth Annual similar to the constraints of Section 2, but our use ACM Symposium on Principles of Programming of sophisticated cost measures should allow these con- Languages, Charleston, SC, January 1993. straints to be used more accurately in determining a data partitioning. 4] R. Cytron, J. Ferrante, B.K. Rosen, M.N. Weg- Chatterjee et al. 3] describe a framework for auto- man, and F.K. Zadeck. E ciently computing sta- matic alignment of arrays in data-parallel languages. tic single assignment form and the control de- They do not provide experimental results, and do not poendence graph. ACM Transactions on Pro- yet deal with the problem of determining the method gramming Languages and Systems, 13(4):451{ of partitioning and the number of processors in di er- 490, October 1991. ent mesh dimensions. 5] K. Ekanadham and Y.A. Baransky. PDM: Parti- The SUIF compiler, 1] performs loop transfor- tioned Data Manager. IBM T.J. Watson Research mations both for increasing parallelism and reducing Center, 1993. communication costs, and to enhance uniprocesor per- formance. Ptran II di ers from SUIF by performing 6] High Performance Fortran Forum. High Perfor- loop transformations related to parallel program per- mance Fortran language speci cation, version 1.0. Technical Report CRPC-TR92225, Rice Univer- 17] J. Li and M. Chen. Index domain alignment: sity, January 1993. Minimizing cost of cross-referencing between dis- tributed arrays. In Frontiers90: The 3rd Sympo- 7] M. Gupta. Automatic data partitioning on dis- sium on the Frontiers of Massively Parallel Com- tributed memory multicomputers. PhD thesis, putation, College Park, MD, October 1990. University of Illinois, September 1992. 18] J. Li and M. Chen. Compiling communication- 8] M. Gupta and P. Banerjee. Demonstration of e cient programs for massively parallel ma- automatic data partitioning techniques for par- chines. IEEE Transactions on Parallel and Dis- allelizing compilers on multicomputers. IEEE tributed Systems, 2(3):361{376, July 1991. Transactions on Parallel and Distributed Sys- tems, 3(2):179{193, March 1992. 19] J. Saltz, K. Crowley, R. Mirchandaney, and H. Berryman. Run-time scheduling and execution 9] M. Gupta and P. Banerjee. A methodology for of loops on message passing machines. Journal of high-level synthesis of communication on multi- Parallel and Distributed Computing, 8:303{312, computers. In Proc. 6th ACM International Con- 1990. ference on Supercomputing, Washington D.C., July 1992. 20] H. Zima, H. Bast, and M. Gerndt. SUPERB: A tool for semi-automatic MIMD/SIMD paralleliza- 10] M. Gupta and P. Banerjee. Paradigm: A compiler tion. Parallel Computing, 6:1{18, 1988. for automatic data distribution on multicomput- ers. In Proc. 7th ACM International Conference 21] H. Zima and B. Chapman. Compiling for on Supercomputing, Tokyo, Japan, July 1993. distributed-memory systems. Proceedings of the IEEE, 81(2):264{287, February 1993. 11] Manish Gupta and Edith Schonberg. A frame- work for exploiting data availability to optimize communication. In Proc. 6th Workshop on Lan- guages and Compilers for Parallel Computing, Portland, OR, August 1993. 12] S. Hiranandani, K. Kennedy, C. Koelbel, U. Kre- mer, and C. Tseng. An overview of the Fortran D programming system. In Proceedings of the Fourth Workshop on Languages and Compilers for Parallel Computing, Santa Clara, CA, August 1991. 13] S. Hiranandani, K. Kennedy, and C. Tseng. Compiling Fortran D for MIMD distributed- memory machines. Communications of the ACM, 35(8):66{80, August 1992. 14] K. Knobe, J. Lukas, and G. Steele Jr. Data opti- mization: Allocation of arrays to reduce commu- nication on SIMD machines. Journal of Parallel and Distributed Computing, 8:102{118, 1990. 15] K. Knobe and V. Natarajan. Data optimization: Minimizing residual interprocessor data motion on SIMD machines. In Proc. Third Symposium on the Frontiers of Massively Parallel Computation, October 1990. 16] C. Koelbel. Compiling programs for nonshared memory machines. PhD thesis, Purdue Univer- sity, August 1990.
Pages to are hidden for
"- A Compiler for High Performance Fortran"Please download to view full document