text:
Announcing Bosun
Kyle Brandt
Imagine if alerting was what you wanted it to be:
- Every alert you received was actionable, and there were few false alerts
- Notifications were actually informative
- You received alerts in time to fix problems before they impacted your users
This isn't the world we live in…
- We accept lots of notifications from our alerting system that are not actionable
- The notifications don't tell us about the problem
- We get paged when stuff is dead and not when it is sick
In order to resolve the dissonance between reality and what alerting should be we need:
- An expressive way to evaluate alert conditions that isn't a 1:1 mapping to the metrics
- Alerts backed by time-series and not just recent values
- A way to to make rich notifications that include useful information
- A way to iterate fast with alert design so that our alerts are continuously improved
A little less than a year ago, Matt Jibson and Kyle Brandt set out to create a system to solve this and other problems in monitoring; we call it Bosun. Our belief is that achieving excellence in alerting is a complex problem and requires a powerful and flexible platform to design alerts. Therefore, Bosun's strategy is to provide a framework that enables the operator to create intelligent and informative alerts. We believe that you are smarter and more creative than any monitoring system can be when it comes to your environment.
In order to achieve that, at the highest level Bosun provides:
- An expression language (a small domain-specific language) designed to allow for the creation of highly flexible and specific alerts
- Notification templates that allow you to include whatever information you think is relevant
- A web interface the provides a workflow for more rapid iteration with improving and creating alerts: Graph -> Expression -> Rule + Template -> Test the rule over history
The Expression Language
We believe that every alert requires action. An alert asks for your attention, and human attention and time is a valuable asset. So alerting is about owning the operators attention. Taking action with alerts practically means one of two things. If the alert was accurate, then you fix the issue that triggered the alert. If the alert was a false positive, then the alert should be tuned in a way that the false positive won't trigger the alert. This is where things tend to fall down because alert evaluations are not powerful enough to be tuned. With Bosun's expression language, you can tune alerts in the following ways:
- Alert thresholds based on history vs static thresholds (or both combined)
- Statistics functions: Min, Percentile, Median, Deviations, Forecasting. You can change the duration that these evaluate over (i.e. 5 minutes, 1 hour, 1 week?)
- Scope-aware: How should components in your environment be grouped? By Host, subsystem, cluster, a combination of those things
- Boolean conditions: The interaction of multiple components
These possibilities, when applied selectively by a skilled operator, provide ample ways to reduce alerting noise.
Notification Templates
Once you have someone's attention with a valid alert, you need to direct them to the problem as accurately as possible. Our notification templates use the Go template language, which means they can be quite flexible. Notifications in Bosun allow you to:
- Include breakdowns of information related to your alert as embedded graphs, html tables, or whatever else you think makes sense
- Include information that wasn't directly related to the alert: i.e. CPU of a host even though it was a memory alert
- Generate links to your dashboards or other sources of information
- Includes notes about why you created that alert, caveats, and other information the person being notified should be aware of
The Workflow
One of the main issues with alerting is that there is so much friction to tuning alerts that it doesn't get done. One of Bosun's goals was to provide a faster iteration cycle for creating and tuning alerts by making the web interface an alerting IDE: Graphs in Bosun's interface link to expressions, which then link to alert rules and templates. You can then test alerts before implementing; the results of a rule and template can be tested in the interface. You can test how they will behave currently, how they might have behaved at a past time, or generate a timeline of how they might have behaved over a range of time.
This means that your alert tuning doesn't need to be totally reactionary. You can test alert changes and see how and when they would have triggered over the past weeks (or longer, if you are patient). This results in less alert noise being sent to operators.
But wait! There's more!
Bosun has also attempted to make some problems in monitoring easier:
- Getting data into the system: our agent (called "scollector") runs on Windows and Linux and starts sending data to Bosun
- Applications can push metrics to the system via JSON API calls
- Human maintenance: Properly designed alerts will apply to new systems, and services are auto-discovered by scollector. This means you don't have to remember to update your monitoring most of the time when a new services and hosts are deployed (as long as scollector is pushed out via your build or configuration management process)
We hope you go try this out. We have a docker image that has everything you need—just follow the getting started guide. We hope Bosun is useful to the community. We need your creativity and ideas to continue to grow it (and some contributors would be nice too!). We owe a special thanks to everyone else at Stack Exchange for:
- Contributing to scollector – Greg Bray has been working hard to fill out our Windows metrics, and Sam Torno did the same for Linux
- Getting a docker build – Peter Grace (who also did a lot of the dogfooding)
- Manning the front lines to keep the site up while we built this – the rest of the SRE Team
- Feature ideas and monitoring concepts – Tom Limoncelli and his monitoring chapters in The Practice of Cloud System Administration
- Letting Matt and I go tilting at windmills – Stack Exchange, Inc.
The Site Reliability Engineering team at Stack Exchange has a new addition – Greg Bray!
Greg joins us as our new Windows-focused generalist (in case you missed it, our friend Steven Murawski moved on recently), though he's happy to work on whatever technology we throw at him. He's a software developer turned sysadmin, and he'll be assisting us in our quest to automate our infrastructure until it achieves sentience.
Greg is a participant on a number of the sites in the Stack Exchange network (since the early days in 2008!), a University of Utah Computer Engineering graduate (where he worked on a FPGA-based tester for NAND flash storage [pdf link]), and an occasional blogger. He has a knack for finding obscure bugs.
Greg lives in Salt Lake City with his wife and 4 year old cat named Kitty, and he'll be working remotely from there. When he's not working with technology, he enjoys biking, camping, and golfing.
Join me in welcoming Greg to our team!
Welcome Shane Madden, ServerFault Valued Associate #0000007!
Tom Limoncelli
The ServerFault Systems Administration team continues its growth with the addition of sysadmin Shane Madden. Shane lives in Denver and will work out of the Denver office on days he wants lunch.
You may know Shane already from his contributions in the open source world. He's an avid Puppet programmer and Python coder. Shane's very active on Server Fault… he has over 69k reputation points which makes him the 7th highest rep user there!
Shane's hobbies include skiing, board games, video games and hockey. He has many pets including a mantis shrimp.
Please join me in welcoming Shane to the team!
The Importance of Observability
Kyle Brandt
Site reliability engineers, in the most general sense, are charged with a clear mission: efficiently keep the sites reliable. Reliability can be broken down into two main facets: availability and performance. This is about where it stops being straightforward and everything becomes nuanced. This is because you have to start defining what availability and performance means for your systems (which is generally driven by the mission of your organization and how your systems fit into that). Even more complexity comes into play when you consider all the activities an SRE team engages in to achieve these things. For example: configuration management, capacity planning, restores, fault tolerance, and security to name some of them.
How you define availability and performance in your organization is a topic worthy of its own set of posts; and the details of all the activities an SRE team participates could fill a library. An SRE team needs to start somewhere and have a strategy to tackle all of this. There is no one answer, but achieving a high level of observability needs to be a key strategic component for any SRE team.
Observability is the Foundation
Observability is the degree and facility in which your team can gain insight into the behavior of your systems. It is worth noting that the scope of your systems is likely quite broad; it includes the obvious things like your applications and hosts, but also includes things like processes, workflows, and team dynamics. Having insight in your systems means:
- Questions operators have about their systems can be quantifiably answered with minimal effort
- Operators have rich mental models of how their systems function
When you have to decide something you can either guess or use "the science." Without a set of systems for observability in place you will end up guessing (not the educated kind) or be terribly inefficient. A good understanding of how systems work is what allows operators to be effective and avoid disastrous mistakes: observability can drive that.
Decision Making and Incident Preparedness
Observability is key to the strategy for an SRE team because it informs and impacts nearly every other activity that team engages in. I've written about the OODA loop before which stands for Observe, Orient, Decide, Act (You can think of Orient as "Analyze.") It is a military strategy that suggests you can be successful when you can rapidly and successfully iterate through this loop quickly. It is also a tool that is useful for thinking about site reliability operations as well.
OODA is carried out at both the macro and micro levels (planning and incidents) by SRE teams. As an example, we can imagine what making system design decisions as a team is like without good observability (and since we have likely all been there, you can probably just remember.) The observation phase will be based on people's memory and is frequently skipped. Orienting or analyzing that information as a group will have conflicts because people don't agree on what the facts are. This can result in arguments about the person's recollection of the facts instead of the issue at hand. Decisions end up being prolonged and half hearted because of the uncertainty of their basis. Lastly, action will be hindered because a strong consensus hasn't been reached because people don't trust the baseless decision. Even worse, people question if this is even the system they should be working on at all.
Many have also probably been through outages when observability is lacking. Lots of time is lost trying to figure out what is even going on. Orienting is difficult because operators lack the internal model of the system that observability provides over time. As a result of these things decisions and actions are chaotic. Or more simply put, it's amateur hour.
In contrast, the picture is entirely different with a solid foundation in observability because everything becomes data informed. This is different from "data driven" because you can trust people's intuition. Due to good observability they have developed keen instincts about systems over time. When it comes to system design decisions you are in a much better position because chances are you are designing the right thing in the first place. Team members will bring their observations to the discussion. If there are questions about the facts, instead of arguing then you can just look them up. Decisions will be made with more confidence and faster because they are based on evidence. Lastly, action will have more consensus behind it, even if people didn't agree they at least know the choice was based on something.
You never know what the next incident will be, but if you have good observability then your operators will have a deeper understanding of the system and will be far more prepared for the unknown.
Other Benefits
Observability positions a team to do more capacity planning by enabling them to see constrained resources and forecast growth. This can help reduce the vicious cycle of fire fighting that many SRE teams are locked into.
Since observability leads to insight, team members are learning more about their systems which generally is a common source of fulfillment for engineering types.
Convinced? 5 Steps to Achieving Good Observability:
In order to achieve good observability an SRE team (often in conduction with the rest of the organization) needs to do the following steps.
- Instrument your systems by publishing metrics and events
- Gather those metrics and events in a queryable data store(s)
- Make that data readily accessible
- Highlight metrics that are, or are trending towards abnormal or out of bounds behavior
- Establish the resources to drill down into abnormal or out of bounds behavior
Each of these steps largely depends on the previous step to be successful.
1. Instrument your Systems
Brainstorm what key and useful metrics exist for your system. Make those metrics easily accessible (i.e. standard APIs like json via REST or by providing a destination to push to) and document what they are and what the implications of those metrics are. This largely falls on the developers of systems, and DevOps culture can go a long way encourage application developers to empower the operations side of things by doing this. At the highest level you can break metrics and events into two categories:
- Objective Oriented: These metrics reflect the mission of your organization. For example they include client facing measurements like response time, availability, error codes, items sold, number of users, number of active users and rate of content created.
- Diagnostic Oriented: These measure aspects of the system that allow you to achieve your objects. These include system measures such as OS, network, hardware, middleware, cluster, and application metrics. These also include response time and availability metrics but they measure components and parts of the pipeline that contribute to your objectives.
Good Metrics also tend to have these properties:
- High Resolution: "High" is qualitative, but a higher frequency of data collection means you will have more insight into the shape of your data (i.e. is it bursty)
- Lossless: This means that there isn't missing information from your metric. This can often be achieved by publishing counters instead of rates and letting the display side of things calculate a rate from that information. Also not pre-aggregating things into averages can be useful (or if you are going to do that also aggregate the data into multiple percentiles)
- Specific: More specific metrics can often be more useful to understanding a system and drilling down into a problem. For example, with something like CPU utilization it is better to report something like %user, %system CPU time breakdowns and let something later in the pipeline aggregate them.
It is also worth making a point to instrument your own internal "meta" systems such as bug tracking and documentation.
2. Gather those metrics in a queryable data store(s)
This is a key intermediate step to making this data accessible. Data generally needs to be stored over time in order to give it context (although the time of each datapoint isn't always important for things like histograms when it is processed later). Having this step enables things like:
- Building dashboards
- Enabling capacity planning
- Allowing operators to explore the data and learn
- Allowing people to invent cool stuff you didn't anticipate
As a rule of thumb, less data stores are better because it makes it easier to work with the data (although specialized databases for things like time series might be worth the tradeoff because of features and scalability.) For time series data in particular, a couple of useful qualities are:
- Scalability: This enables one to collect a lot of metrics, at high resolution, and high retention
- Aggregation: This encourages a shift from host/process oriented views to cluster and service oriented views
3. Make that data Readily Accessible
If there is a lot of friction to view the data then people won't have time or energy to do it. This is why it is important to have good dashboards and APIs to allow easy access for your operators. Good dashboards tend to have the following attributes:
- A fast responsive UI to allow for operators to drill down and explore easily
- Enables operators to create their own dashboards and graphs
- Highlight problems
4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior
Ideally a team ends up collecting a lot of data. This means humans can't process it all and therefore your systems need to ask for operator attention. Essentially this is alerting. However it is important to understand that alerting doesn't always mean "emailing". It can also mean things like publishing something to a dashboard or logging it.
Traditionally alerting has been done on current values, but anomaly detection and forecasting are becoming a reality thanks to some work done at Etsy.
Alert noise / desensitization is a plague in our field, my belief is that future systems will allow for more carefully crafted and adjustable rules to reduce the noise. Keeping this under control is also largely about discipline and remembering that every alert requires action.
5. Establish the resources to drill down into abnormal or out of bounds behavior
The above steps are a gateway to observability. This is because the nature of collecting metrics is resource constrained. You can only collect so much information without noticeably impacting what you are trying to observe. Eventually you are going to need to drill down into problems or explore further why metrics are behaving in a certain way. There are three common activities for this:
- Log analysis: Digging into your system logs for information. System logs can also be a powerful source of metrics (especially things like web logs) if you parse them and feed the results into your monitoring systems
- Profiling: This the activity of sampling programs to figure out what they are doing – generally at a much higher resolution than collecting metrics (computer time (sub 1ms) instead of human time)
- Tracing: Collecting every single thing a system is doing (i.e. strace or DTrace)
Although my path to observability puts an emphasis on collecting metrics and events, this step is also crucial to observability.
Use the science, Luke
If observability is one of the key components of the strategy for your team, then it sets the tone and foundation for everything else. It can create a culture of constant learning as it provides a medium for learning about your systems and proves a source of information for productive analytical arguments. Whatever your strategy is, you need to consider what role observability plays in your team. And remember: Use The Science.
Homegrown DevOps Tools at Stack Exchange
Kyle Brandt
A lot of tools available in IT/Sysadmin/Ops/DevOps are disappointing:
- They don't fit your environment. They lack features or our designed for a different sort of environment (i.e cloud vs hardware, Linux vs Windows, distributed vs centralized etc)
- You can't interact with them programmatically
- They cost too much
- They are not customizable enough, or require too much customization to get off the ground
- Feel kludgy, unreliable, outdated, or like the programmers were stoned
- Don't fit with your company's culture (i.e. Enterprise vs Agile)
In short a lot of stuff is too expensive, isn't a good fit, or is simply bad software. This ends up leaving an ops team with two options. They can whine about it, or create their own tools. So at Stack Exchange we build our own DevOps tools.
Nick Craver's baby, which we just call "Status" is at first glance a monitoring dashboard, but is essentially a collection of tools that filled various needs:
- An Overview of CPU, Memory, and Network utilization for all our servers as well as a detailed view. Done with responsive and interactive D3 graphs as well as sparklines it helps compensate for Solar Wind's terrible interface.
- SQL Server monitoring. SQL's built in Clustering views are deeply flawed. If a node loses connectivity, it stops updating remote nodes status, so it could show everything as connected and fine, even if there is no connectivity. We also get to see the most expensive queries, active queries utilizing whoisactive, current connections, and which DBs are on which server
- HAProxy Monitoring and Administration: With multiple instances of HAProxy we needed a single view instead of HAProxy's built-in display. Also, this gave us a nice web interface to take servers out of rotation
- Redis: A nice presentation of Redis Info across all instances and all servers. Also a display that shows what is slaved to what in at a quick glance
- Elastic Search: Health overview of or clusters (as well as index and shard data)
- A dashboard of all the exceptions generated by our applications
Status is C# / .NET app. It polls data from various sources – sometimes the system directly and other times it gets it from Orion. There is a lot more to status that makes it awesome. The real accomplishment is that status enables us to see the general health of our main infrastructure at a glance.
Web Logging
If you business is creating and running websites, your web logs are gold. We use the logs generated by our load balancer, HAProxy, as our canonical web logs. In their raw text format, web logs are often not that useful (this is particularly true with over 100 million records a day). However we parse and structure our web logs in a few different ways:
- We have C# service that Jarrod Dixon wrote that inserts them into SQL so we can query them. In order to query them we use an instance of Data Explorer, SQL management studio, and also have certain lookups directly from our sites
- Displaying realtime graphs of various log information with Realog, a system I created with Go, Redis, and NVD3.js so we could view activity live without having to write queries
One of the interesting things we do with our weblogs is to add extra information by adding headers inside the app and striping them from the response at HAProxy. For example, we capture how many Redis and SQL queries were involved in that request and how long they took.
Patch Dashboard
OS updates can be a bit tedious, even more so in a mixed Windows and Linux environment. Steven Murawski and George Beech created a dashboard that allows us:
- View the outstanding patches and patch count for both Linux and Windows
- Trigger updates on either Linux or Windows
- Schedule time frames for automatic Linux updates
What's Next
If you want to learn more about these tools and DevOps at Stack Exchange, come see George, Nick, and Steven present "Building for Operations" at Velocity.
Keeping all this stuff to ourselves feels a bit greedy. However, for something open sourced to be very useful it usually needs to be made a bit more generic which takes time. We also want to build a lot more. Our inventory system Racktables lacks an API so we need a new one or a way to extend it. We want to build our own monitoring system (likely on top of OpenTSDB). In order to create more, and open source it we need help. So we are looking a full time developer with ops experience to join our SRE team. So if you are awesome, want to build awesome ops stuff and open source it, come join us!
Keeping Your Cool
Steve Murawski
I'm not talking about data center cooling here…
I was recently listening to The Ship Show podcast titled "Keep calm and PROD on". In this podcast, the hosts were discussing whether or not all devs and operations personnel should have production access.
The conversation really hit home with me when it changed from having access to production to how people handle dealing with outages/incidents. The hosts asked for some feedback on the topic on Twitter, but I have just a few more than 140 characters of thoughts on the topic.
The Scenarios
The hosts outlined a couple of scenarios for discussion. They didn't use these exact terms, but it helps me to group them under certain archetypes (it must be all those role playing years catching up with me..). In no particular order (going from memory and making up the classifications here), we have the "cowboy" response, the "neophyte" response, and the "deer-in-the-headlights" response. I'll provide a quick synopsis of these responses.
The "Cowboy" Response
This incident response archetype is very identifiable in many environments. Picture it, the year is 2010, it's 2 AM and your cell phone is blowing up with monitoring alerts. The website is down and everyone is waking up. You log in to your VPN and start gathering information. Everything goes dark, and then everything comes back up, seemingly magically fixed. It comes to light that one of the developers, on his own volition, decided to take drastic actions to restore service.
This response highlights the lack of communication and command and control that is typical of frantic incident response in many IT departments I've observed over my years in IT.
The "Neophyte" Response
This incident response archetype is your typical "newbie". The neophyte does not have to be new to the field, just new to high pressure outage scenarios. Taking the scenario above, the neophyte may be taking his first turn at the on-call rotation or this may be the first time a particular problem has cropped up. The neophyte might not be comfortable with getting a more senior engineer out of bed, or getting some developers involved. Another pitfall with the neophyte is that they might not be willing to take charge as others begin to respond to the incident, meaning that the incident response can be confused with no coordinated direction.
The "Deer-In-The-Headlights" Response
The last one I remember the podcast covering was the "deer-in-the-headlights" response, where regardless of a person's experience level, they just don't respond well in a crisis. Many of the downsides of the neophyte are felt here too.. The primary responder may not reach out for help or may not be able to control other responders. This leads to a fractured response, where people may be working at cross purposes.
What Should We Do?
What I've Learned In Previous Careers
Training To Be A Cop
You may not know this about me, but before I got into IT, I was training to be a cop. I went to school and did all the fun training on how to drive fast, arrest people, and shoot guns. A big part of that training also included how to respond to medical situations (trauma or illness) or hazardous materials situations. (Despite what you may think.. watching re-runs of Cops is not adequate training – except as what not to do!)
In that training, we drilled how to respond to emergencies as individuals and as part of a team. Each course drilled scenarios, but our firearms course drilled scenarios most heavily and over time included skill sets learned in other classes. Nothing wakes you up to a training scenario like walking into a situation with a vague description of a problem and, as you start to gather information and stabilize the scene, you get drilled in the head by several rounds of simunitions. Even though you know that you are using training rounds, it shakes you to your core to realize you could be dead in seconds.
Part of the training regimen includes working past the failures and mistakes. In that first scenario, even though every recruit gets ambushed and shot in the head, our instructors make us follow through with the techniques we've drilled in the classroom and on the range. We are coached to get to cover, return fire, and call for backup. We can't dwell on our mistakes (there will be time for that later.. everyone is videotaped for review with the whole class afterwards), we have to follow our training and deal with the problem and stabilize the scene.
Over time, we progressed through a number of other scenarios, some by ourselves, and some with other recruits. In all cases, we were responsible for communicating status back to our dispatch center, requesting resources as needed, and dealing with any subjects in person that the scenario called for. We were tested with a variety of actions, all potentially threatening, but each requiring evaluation for how we could respond and always under the watchful eyes of our instructors and the unblinking eye of the video camera. This training reinforced our more static, isolated drills, allowing us to respond to dynamic situations with a combination of intentional action and reflexive reaction based on our other drills. Training helps minimize the "deer-in-the-headlights" and the "neophyte" responses and allows people who would experience those responses to fall back on training.
Working In A Public Safety Agency
I also worked as a dispatcher, clerk, and auxiliary officer before and alongside my role as IT guy for a local police department. Over that time, I observed, interacted with, and sometimes responded with our emergency responders as they dealt with life and death situations. I observed the growth in how the police and fire agencies learned to respond to incidents together, using the National Incident Management System (NIMS), the emphasis on which grew after 9/11/2001.
Under the NIMS model, the first responding public safety officer is the incident commander. As the situation develops, the incident commander role can change, based on who's best suited to deal with the incident. For example, in the case of a fire, the police officer first on scene will be the incident commander, until a fire department official is set up and ready to take over command. (In my experience, this is after police officers have saved all the people in the building. ). Let's look at another scenario: officers respond to a report of a burglary in progress. The first officer responding is the incident commander. As the scene develops and a perimeter is established, the officer in command is responsible for requesting the resources needed and beginning to stabilize the situation. Next a sergeant, lieutenant, or captain arrives on the scene and takes over the coordination of the perimeter, allowing the officer to focus on his area of responsibility.
NIMS also defines several other key roles for the command staff in an incident, a Public Information Officer (PIO), a Safety Officer, and an Liaison Officer. The PIO's responsibility is to keep the stakeholders and public in general informed as to what is going on. Part of the role is determining what information is helpful to share and what should not be disclosed. The Safety Officer monitors conditions and ensures the safety of all incident personnel. The Liaison Officer is responsible for dealing with all the coordinating agencies. Defining these roles can help deal with the "cowboy" response. If incident response is structured, the right resources can be directed to a problem and a sustainable fix is more likely an outcome, versus "just get x involved, he fixed it last time."
How We Should Respond
These two experiences provide some basic thoughts into how we can approach incident response.
Just because we are sysadmins, site reliability engineers, devops engineers, etc. doesn't immediately grant one the intrinsic knowledge and skill necessary to deal with an outage, especially if you are a specialist and the outage deals with technology you are less familiar with.
For operations personnel (developers, dba's – yes them too – and sysadmins), this is critical.
For someone with deep intuition about their environment, the answers are easier than for newer or more narrowly focused personnel. Guess what? The knowledgeable guy isn't always around when things go ill.
Focused drills, around dealing with one sort of problem or technology, as well as combined drills with multiple components are vital. Not only do you need to be familiar with the systems you are responsible for, but also everything they interact with, internally and externally. Quick – what do you do when your CDN stops serving content? Have you drilled that scenario? If you haven't where do you even begin?
This is something we are going to be focusing more on here at Stack Exchange and I'm super excited about that. We've decided that this is a priority for our organization and we'll be dedicating time to this.
In the police department where I worked there was a definite command structure, but individual officers had a great deal of latitude to respond to most situations. The latitude enjoyed by officers in that agency is similar to the latitude I have as a sysadmin on my team. Certain situations we can just deal with and not need to involve others. If I need backup (additional resources), I can request those, but if a situation escalates, it's time to bring in more support. We don't have a strict command and control environment; as we grow our technical staff I think that'll be more defined.
When it comes to incident response though, the cops and my fellow sysadmins have a bit of a different experience. Since the officers drill a variety of scenarios and have those drills and training reinforced by continual engagement with an unpredictable public, their escalations from situation to incident are much more fluid, as are their transitions of the incident commander role. We currently don't have anything defined like that, though I hope as we start to drill these scenarios more, I'll be able to lobby effectively for the establishment of at least two of the NIMS roles, the Incident Commander and the Public Information Officer.
In the podcast, one of the themes discussed included how detrimental demands for status updates and presence on conference bridges could be. By assigning (and training) someone (and at least one alternate) to fill the PIO role, the remainder of the technical staff is freed up to deal with the issue.
In our case, I can envision that as having our PIO designate one chat room or google hangout as our internal status update location and during the incident he/she'd make regular updates to those areas. In addition, the PIO would be responsible for updating our status Twitter account and status blog to keep the public informed as needed.
As for the Incident Commander role, we'd need to train all of our on-call personnel, as well the rest of the technical staff, so everyone is on the same page as to who is in charge and directs the resolution of an incident. While we don't have this defined yet, we had a short outage a few weeks back that illustrates how this can work.
- About 9 AM UTC, while I was dreaming of servers with 4TB of RAM and many multi-core processors, my phone began to blow up with alerts from Pingdom. I wasn't the on-call person, but I always monitor for severe external alert failures.
- I rolled out of bed and
ranstumbled down to my office and got online to start investigating. - First order of business, check our chat site.. oops! It's down! Normally our chat servers are in the opposite data center from where our Q&A sites run, but we are preparing for maintenance in our secondary data center, so it's running in New York with the rest of our infrastructure.
- Next up, VPN.. which connects.. that means internet to our data center in NY is still good.
- I jump in a Google Hangout (
fortunatelyunfortunately for my co-workers, I look as good as I do right out of bed as I do after getting ready in the morning) that we have set up for our site reliability team. - Chat comes back online. I dropped a note in chat that I'm in the SRE hangout and troubleshooting.
- I'm soon joined by Geoff Dalgas, one of our Core Q&A developers.
- He and I discuss the situation and began validating the different bit of our infrastructure.
- We determined that we were seeing an issue with keepalived on our load balancers.
- Just then, Tom Limoncelli, another of our SRE team joined us to help with the issue.
- We also had several more developers pop in and see if they could offer any help.
- We determined a course of action to remediate the problem and began to implement it.
- Soon, Stack Overflow (and the rest of the network) was back online and the twitters began to calm themselves.
In this situation, I was first on-scene and acted as incident commander. If the situation had continued to develop into something more complex involving the load balancers, I might have had to defer to one of our other engineers or Geoff.
Open Line
So, what do you do in your organization? Do you drill and train for failures? Do you test your backups? Do you prepare your operations personnel for how to respond in an incident?
Configuration Management is Hard – Responses
Steve Murawski
I've been getting some great response to my previous post, and I wanted to make a few things very clear.
What I'm not doing -
- I'm not advocating or dismissing any particular configuration management tool.
- I'm not discounting the tough work done by companies and community projects that have created abstractions on managing disparate systems.
What I am attempting to do -
- Highlight the challenges of cross-platform management and application management.
- Show one of the efforts in providing a standards based management abstraction.
- Offer my thoughts on why I see value in that direction and what challenges I see.
Managing the Operating System vs. Managing Applications
There is definitely some confusion around using CIM to manage the OS vs managing applications.
- CIM Classes can be used to model applications as well as OS resources.
- Most of the "applications" that are packaged as roles and features in Windows Server expose a CIM management API.
- WMI is an implementation of the CIM standard and starting with Server 2012 and the Windows Management Framework V3, CIM is exposed via WSMAN.
Hyper-V, File Shares, Clustering, IIS, and others all offer CIM based management models. Other applications can expose a CIM management model as well. As long as the host CIM server (WMI on Windows and OMI or OpenPegasus or ???? on Linux based Operating Systems) is operational, applications can also offer their configuration and status via that channel by creating a provider. To do this on Windows, there is some documentation to get you started:
- http://blogs.msdn.com/b/wmi/archive/2013/05/16/introducing-new-management-infrastructure-mi-api.aspx
- http://blogs.msdn.com/b/wmi/archive/2013/05/08/implementing-mi-provider-1.aspx
- http://blogs.msdn.com/b/wmi/archive/2013/05/08/implementing-mi-provider-2.aspx
- http://blogs.msdn.com/b/wmi/archive/2013/05/11/implementing-mi-provider-3-generate-provider-skeleton.aspx
To do this with OMI, you can find some documentation and source at https://collaboration.opengroup.org/omi/documents.php.
The Current State – Reprise
But, but, tool {fill in your favorite tooling here} already does THAT!!!
There are a number of tools that valiantly strive to provide cross-platform management. I mentioned several of them in my last post, but there are a number of others.
Yep, it does. Until…
things change. The challenge these tools have is that they have had to implement their abstractions against very different implementations. The problem there is that these things are not stagnant. The management APIs can change over time and since there is not a standard description of the API or underlying configuration.
If CIM were the standard API exposing the configuration, the underlying implementation details can change, but configuration management and monitoring tooling don't have to care about that. The tool vendors and community projects can focus on other value adds for their particular tooling, rather than being forced to continually update the basics.
The Next Steps
We are still in the early stages of the push for CIM and WSMAN. We'll have to see how adoption picks up. The continuing work around OMI holds promise, but it needs a deployment or integration story for various Linux distros and more public providers for managing components of the Linux OS and attendant applications.
There have been some interesting announcements at TechEd in relation to Windows Server 2012 R2 (watch this video and pay attention around 49 minutes in). I'm going to talk more about this feature and it's implications and my plans with it in the very near future.
Cross Platform Configuration Management is Hard
Steve Murawski
Configuration management today is mess if you work in a heterogeneous platform.
There is tooling that takes a stab at it, and is getting better (from the *nix world – Puppet, CFEngine, and Chef and from the Windows world – System Center Configuration Manager, Group Policy, among other third party application deployment platforms). These tools are all well and good, but they fall down when reaching across the OS divide. Puppet, Chef, and CFEngine (there are others as well, but these are some of the more popular) all have some cross platform support, but it feels unnatural (especially in module or recipe development).
Why is this a mess?
Windows is traditionally described as having an API oriented management model, whereas *nix has a document based management model.
Well, that's a load of crappy, crap, crap. What does that actually mean?
It means that the two operating systems offer two different management models. The two different models have different abstractions and idioms for operating system constructs. Let's look at a concrete example, setting a static IP on a network interface (just the rough strokes.. I'm not going to spend too much time on the minutia). As I stated before, Linux uses a document oriented management model, so to configure my network interface, I'll edit a document or two.
The Linux (Centos) example:
- Find the correct interface file under /etc/sysconfig/network-scripts
- Open it in your text editor of choice
- Edit it to contain your desired settings for the network interface and save the file
- If you need to add/modify DNS servers, find /etc/resolv.conf
- When done, you can bring your interface online with a command line call to
ifup eth0
You'll have something that looks like this for your network configuration file:
DEVICE=eth0 IPADDR=192.168.10.80 NETMASK=255.255.255.0 NETWORK=192.168.10.0 BROADCAST=192.168.10.255 GATEWAY=192.168.10.1 ONBOOT=yes BOOTPROTO=none USERCTL=noAnd something like this for your resolv.conf:
domain serverfault.com search serverfault.com nameserver 192.168.10.60 nameserver 192.168.10.61That wasn't so bad, and as an added benefit, they are just text files, so I could check them in to a revision control system (Versioning FTW!). Now, let's look at what we'd need to do on the Windows side. Since this is a blog for a community of "professional" systems administrators, we are going to dispense with any GUI example for doing this.
The Windows Server (2008 R2) example:
- Use WMI to retrieve the network adapter interface index.
- Use WMI to retrieve the network adapter configuration by the index.
- Set the desired IP address, gateway, and DNS servers and suffix against the WMI object.
You can use the following PowerShell commands to make those changes:
$NetworkAdapter = Get-WMIObject Win32_NetworkAdapter -filter "NetConnectionID = 'Local Area Connection'" $NetworkAdapterConfiguration = Get-WMIObject Win32_NetworkAdapterConfiguration -filter "InterfaceIndex = $($NetworkAdapter.InterfaceIndex)" $NetworkAdapterConfiguration.EnableStatic('192.168.10.80','255.255.255.0') $NetworkAdapterConfiguration.SetGateways('192.168.10.1','1') $NetworkAdapterConfiguration.SetDNSServerSearchOrder('192.168.10.60','192.168.10.61') $NetworkAdapterConfiguration.SetDNSSuffixSearchOrder('192.168.10.60')and
The Windows Server (2012) example:
- Set the desired IP address and gateway based on the interface name.
- Set the DNS servers and with a few more PowerShell commands.
You can use the following PowerShell commands to make those changes:
$IPAddressParameters = @{ IPAddress = '192.168.10.80' InterfaceAlias = 'Local Area Connection' AddressFamily = 'IPv4' PrefixLength = 24 DefaultGateway = '192.168.10.1' } Set-NetIPAddress @IPAddressParameters Set-DNSClientServerAddress -InterfaceAlias 'Local Area Connection' -ServerAddresses '192.168.10.60','192.168.10.61'Both of these examples are interactive commands, but I could easily save them in a file and place that under version control (and I should).
So what?
The examples don't look all too different, but they do illustrate the difference between similar operations. In both examples, I end up with an artifact, but one is for a one time application of the setting (the Windows side) and the other is the setting storage location (the Centos example).
On the Centos box, we had to edit a file where the configuration was read from. On the Windows servers, we updated settings via a WMI API (in both cases.. on Server 2012 there are more built in cmdlets, but many of them are thin wrappers over the WMI APIs) and not the actual end storage location.
This is what
Any configuration management tool that works in a cross platform capacity needs to understand these distinctions and check based on OS type which implementation to use when configuring a system. This means for most configuration types, you'd have a big "IF" block where *nix based OS's follow this line of processing and Windows based machines follow the other line of processing. This can become a maintenance nightmare as OS versions change the API on the Windows side or modify location and or layout of the configuration files on the *nix side.
And it's even worse…
Now, what happens when you have a model that doesn't translate across both worlds?
For example, how do manage file permissions?
Posix style permissions (used on most *nix variants) assigns permissions are nowhere as discreet as NTFS file permissions. In addition on Windows, the file system auditing is also configured via the permissions configuration. In the reverse, on *nix files can be set as executable, where that is handled by file type mappings based on file extensions in Windows. This fragmentation leads to more complex implementations on the side of configuration management software developers or missing feature coverage. In either case, this is a loss for the sysadmin who maintains cross platform environments.
But what if….
there was a common method of interacting with operating systems, regardless of what was running underneath? What if this method used a common transport (open standard) and communications were defined by an open standard? This is the direction Microsoft is taking with CIM and WS-Management.
CIM (Common Information Model) is a DMTF (Distributed Management Task Force) standard for describing management information for systems, networks, applications, and services.
WS-Management is another DMTF standard for management communication, focused on CIM traffic.
Microsoft has contributed to an open source project hosted by the Open Group called OMI. OMI is a CIM server that communicates over WS-Management and is implemented to run on *nix based operating systems.
I'm personally interested in where this will go, given Microsoft's market power (Cisco and Arista are working on incorporating OMI into their network switches). The idea of a shared management model is very appealing to me, as I work in a cross platform environment. I'm responsible for our Windows infrastructure, but I have to be able to work with our *nix infrastructure as well. If I could use one model for interacting with both, that's a huge win for me and my team.
This wouldn't eliminate any domain specific knowledge on either OS side, as you'd still need to know what buttons to push and knobs to tweak to get things going and do some deep troubleshooting. It does, however, make the idea standardizing how various OS components can be accessed, making basic configuration, monitoring, and troubleshooting much easier.
I'm interested because…
this pushes the implementation down to the OS provider (or the CIM provider provider) and gives vendors one target to hit for configuration standards. In the Microsoft case, they can say "Follow this standard and any Windows system can manage you with minimal effort." If other OS's support CIM and WS-MAN as well, it becomes easy to offer management interfaces there as well.
Obviously this would be huge change to the existing way of doing things for OS and application developers, not to mention systems administrators that are invested in their existing ways of doing things.
I don't see another good alternative though, as the numbers and variety of systems continue to scale up and "cloud" becomes more of a factor in our environments, yet the number of admins is staying stagnant or being reduced. Simplifying the management and monitoring surface makes sense in today's and likely tomorrow's data center landscape.
It doesn't solve every problem and vendors can still implement vendor specific extensions (and we know how well that's worked with SNMP).
NOTE – Be sure to check out some clarifications and expanded discussion in my followup post.
Welcome, ServerFault Valued Associate #0000006!
Steve Murawski
The ServerFault Systems Administration team continues its growth with the addition of sysadmin icon Thomas Limoncelli. You may know Tom from his books, Time Management for System Administrators and The Practice of System and Network Administration, or from his many conference appearances at events like LOPSA-East, the Cascadia IT Conference, and USENIX LISA. Tom's also been a ServerFault user, though references to his books outnumber the number of direct answers he's supplied.
When we saw that Tom was just finishing up his time at Google as we were posting the next ServerFault opening, it was kismet. Tom is the quintessential systems administrator and a great fit for our team.
I've been lucky enough to know Tom for the past couple of years and I am very excited to be able to work with and learn from him (and hopefully teach him a trick or two on our Windows stuff). His The Practice of System and Network Administration was one of the first resources I had when I got started working as a systems administrator and really got my career headed in the right direction (well, if you consider getting to work with a small crack team of sysadmins and developers on one of the more dynamic environments out there "the right direction").
Tom is joining Bart and George in working out of our (newly constructed) New York headquarters. Tom hails from New Jersey and frequently attends local LOPSA meetings.
Join me in welcoming Tom to our team!
Windows Server 2012 – A New Hope or Just New Hype?
Steve Murawski
Hey everyone, I'm one of the newer ServerFault sysadmins and I've got the fun job of explaining why we have been moving to Windows Server 2012 just almost as fast as WDS can deploy images.
What's Wrong With Windows Server 2008 R2?
Where do I begin?
2008 R2 was a great operating system in its day. Automation was becoming a core part of the platform. PowerShell was shipped in the box, enabled by default, and supported several roles and features. Many of the kinks were worked out after the Vista/2008 release cycle and people were starting to acclimate to the User Account Control (or figured out how to turn it off).
Server 2008 R2 held up well for a long time, but was starting to show it's age with our SQL Server cluster. There are a bunch of hotfixes required to make clustering have a chance of working with SQL Server 2012 AlwaysOn in our environment. Brent Ozar, in part based on experiences helping us, now recommends not building AlwaysOn Availability Groups on Server 2008 R2.
Additionally, the management story for Windows Server up to this point has been very focused on single machines. Server Manager in 2008 R2 targets one machine at a time. Most of the MMCs deal with one machine at a time. PowerShell remoting was not on be default. It is definitely work to build a multiple machine management story.
So, how does Server 2012 help?
First of all, Windows Server 2012 helps address our major pain point of how SQL Server 2012 Availability Groups interact with Windows Clustering. Clustering received a lot of attention in Windows Server 2012. The addition of dynamic quorum helps maintain the cluster state in the case of site-to-site VPN interruptions. Clustering itself has just become more resilient and offers a wider feature set.
The second (and I think most important feature) that Windows Server 2012 offers is SMB3. SMB3 fixes many of the performance considerations with access to files stored centrally on a file share. SMB Multichannel allows SMB clients and servers to leverage multiple network connections, makes SMB transfers resilient to TCP disconnects (by having multiple streams per transfer, even with one NIC), and provides the foundation for scale out file servers, providing a way for multiple servers to share the same connection concurrently, isolating storage transactions for single server failures and allowing connections to scale across multiple servers. SMB3 provides a number of opportunities to radically change the structure of your application data access and I talked with Richard Campbell about this at length on RunAs Radio.
The third huge win is how Windows Server 2012 supports three levels of GUI-ness. You have the traditional full UI to support the traditional administrative experience, and since this is the same codebase as Windows 8, you get the Metro/Modern UI start screen as a "bonus". The next level of GUI-ness offered by Server 2012 is the minimal shell experience, or what I like to call "Server Core with Training Wheels". The minimal shell removes Explorer, Internet Explorer and most GUI apps, but leaves behind Server Manager and the MMCs. This provides a bit of a security blanket for those admins who aren't quite ready to give up managing a server locally, but want to break their reliance on the UI and grow up to be SYSTEMS administrators who can manage multiple systems. By removing explorer.exe and iexplore.exe, you've eliminated a huge source of patches, particularly the ones that require a reboot. Additionally, since there is no Explorer, the RDP experience is a command prompt and Server Manager – discouraging random RDP sessions. The final level of GUI-ness is full on Server Core. This is the DEFAULT install option for Windows Server 2012. Server Core lacks even the minimal GUI tooling provided by the minimal shell configuration. Server Core, as of Server 2012, is no longer a life choice. In Server 2008 and Server 2008 R2, when you selected Server Core as the install option, you were stuck in Server Core. With Server 2012, the GUI bits are features and can be added and removed through the standard feature management tooling. I talked a bit about this with Richard Campbell on RunAs Radio earlier this year.
Next up in the "pro" column for Windows Server 2012 is PowerShell V3 and the increased cmdlet coverage in the operating system. PowerShell v3 brings a number of great enhancements to the party like improved performance, resilient remoting sessions, commands for working with the web and REST endpoints, workflow, ISE enhancements and more. Windows Server 2012 also brought about 2400 commands across the in-box roles and features (up from under 500 in 2008 R2). This explosion in coverage was due to improvements in the WMI APIs and a new API in PowerShell for being able to take an XML mapping file and generate PowerShell commands from a WMI API. This got us tons of additional coverage, including the network stack. Since servers really aren't intended to be managed individually (despite what some sysadmins, product teams, and third parties software vendors believe), the increased PowerShell coverage makes that a much more reasonable task. If you are interested in further talk about PowerShell V3, you guessed it, there was a RunAs Radio talking about that as well.
The final consideration we had for choosing Server 2012 was in-box NIC teaming. Server 2012 offers OS level NIC teaming with a variety of teaming options. Given the problems we've faced with third-party NIC teaming (http://blog.serverfault.com/2011/03/04/broadcom-die-mutha/ and some other fun stories), I was gung-ho to get switched over to the native teaming features (which I've monitored in production environments for over a year at that point). Having the network load-balancing and failover (LBFO) teaming included in the OS is a major plus. Don't worry, I won't point you to yet another RunAs Radio show this time.
Where are we now?
I'm currently working with our team to continue the roll-out of Server 2012. Part of that process is updating our installation and configuration management plan. Nick Craver, one of our developers who likes to help with the sysadmin work and has been featured on this blog, and I have spent a good amount of time tweaking our deployment scripts (server and application deployment) to facilitate the roll out of new servers. We've got our SQL Server and web tier installs fully scripted, to the point where we can go from bare metal to a fully functional web server in about 3 hours (mainly OS install time and running Windows Updates).
We are still working out the rest of our deployment story, looking for integration points with our Puppet configuration management, making the configuration idempotent, and creating artifacts that can be stored in source control to version our environment. But each week sees more Server 2012 (and PowerShell use… Pete's started using a bunch of stuff I wrote and now I have bugs to fix… grrr.. ) in our environment.
Where do we want to be?
I would like to see our Windows deployments that are in a production capacity (internally or externally facing) under full configuration management control, with changes pushed via a configuration management infrastructure and configurations enforced regularly.
I would like to have most of our servers running Server Core (doesn't make much sense to do that for our management stations, but for most servers it works).
Most of all, I would like our server deployment to help serve as a template for a different way of Windows administration, one with rigid standards but quick to respond to changing needs. I'd like to see our infrastructure as agile as our code base, where changing our OS configuration is as easy as pushing a build is for our development teams. I'd like to be able to validate infrastructure changes quickly, closing the feedback loop and allowing worry free infrastructure changes.
You all have a front row seat to our environmental changes and I plan to blog extensively about my configuration management efforts in the near future, giving you an opportunity to learn from my mistakes.


text:
My daughter LOVES Miss Amy's classes! She is an amazing teacher who truly cares about the kids and creates such a wonderful experience for them!! We look forward to class every week!!! Thanks Miss Amy!!!! xoxox
Kelly Reamer O'Brien
***** My son and I both LOVE Miss Amy's classes. We started going to Crawl, Rattle 'N' Roll when my son was only 2 months old. My Husband was deployed, and the adult interaction was such a life saver for me. As my son grew, he made friends that we often see for play dates outside of class. Miss Amy cares so much for each and every parent and child that walks into each session, and even offers great art classes at her home. After a year of classes almost every week, I can say that Miss Amy and these classes are such a treasure.
Amanda Klutts
San Marcos Babies Classes
Has a New Little Sister...
Toy Store
Launch Signage (Back)
Our November 2014 Launch Signage (Front)
Christmas 2014 Street/Yard Signs (resolution was thankfully better on the actual signs)
Toddler "Move & Groove" Class
Toddler/Preschooler Messy Art Class
Baby "Crawl, Rattle 'n' Roll" Class
New! Toy Store!
Local Delivery or Pick Up. Lower 48 States by UPS Delivery.
...Kinda proud of Miss Amy's Toys' first graphics and signage!

subset: commoncrawl


text:
December 17, 2010
Seacology is a prime example of how a few people with a brilliant idea can do so much. I give sparingly to organizations because I like to feel like I know where my support is going. But I will always give to Seacology because their approach is so very different. What I love is that they look at the whole picture, not just the reef although that is important, but the people as well. They look at the needs of the local people as they try to find ways to save ecosystems. This organization does amazing things with a small staff: read their excellent newsletter and find out more! Hey, it's a great idea to give a donation to Seacology as a seasonal gift!
The Great!
I've personally experienced the results of this organization in...
Read the newsletter to find out the great projects this organization is doing!
Ways to make it better...
If I had to make changes to this organization, I would...
Increase the staff so more projects could be accomplished.
Donor & I am a classroom teacher and I heard about Seacology through a friend and decided to be a supporter.


text:
Black Ops II stays on top of UK chart; miserable start for PlayStation All-Stars Battle Royale
Call Of Duty: Black Ops II is number one in the UK all-formats software chart for a second week in succession, seeing off the challenge of Hitman: Absolution despite sales falling by 81 per cent week on week.
Absolution has to settle for a debut in second spot, despite outselling its 2006 predecessor Blood Money by almost three copies to one. FIFA 13 stands firm at number three for a third successive week, with Assassin's Creed III a non-mover at number four.
After Hitman, the next highest new entry is Lego Lord Of The Rings at number six – look out for a review of that on the site later this week. We're also taking a look at Family Guy: Back To The Multiverse, which debuts at number 35.
It's been a dismal launch weekend for PlayStation All-Stars Battle Royale, however. Sony's answer to Smash Bros, developed by US studio Superbot Entertainment, barely sneaks into the top 40, at number 38.
01. Call Of Duty: Black Ops (Activision)
02. Hitman: Absolution (Square Enix)
04. Assassin's Creed III (Ubisoft)
05. Halo 4 (Microsoft)
06. Lego Lord Of The Rings (Warner Bros)
07. Just Dance 4 (Ubisoft)
08. Skylanders Giants (Activision)
09. Need For Speed Most Wanted (EA)
10. Dishonored (Bethesda)


text:
Reviews: Videos
Fender American Vintage '65 Jazzmaster Electric Guitar Review
October 16, 2012
The Jazzmaster is relatively en vogue these days, thanks in no small part to once-fringe players like Nels Cline, Sonic Youth's Lee Ranaldo and Thurston Moore, and a host
Read More
TV Jones Spectra Sonic C Melody Electric Guitar Review
October 8, 2012
The TV Jones Spectra Sonic C Melody doesn't disappoint in any regard other than the semi-steep price.
Read More
Wampler Pedals Brent Mason Hot Wired v.2 Pedal Review
October 3, 2012
A smooth, 2-channel overdrive with incredible touch sensitivity.
Read More
Fender Select Stratocaster Review
July 25, 2012
With figured wood and premium features, the Select Strat is as close as you can get to Custom Shop at USA production prices.
Read More
Champions of Chime: The EL84 Roundup
July 9, 2012
Three studio pros test nine EL84-powered amps in the 15-watt ballpark.
Read More
Ibanez GB300 George Benson Signature Guitar Review
May 23, 2012
Outside of a few minor tweaks in body shape, neck size and pickups, the essence of the GB remains.
Read More
Reverend Sensei RA Electric Guitar Review
May 2, 2012
Sporting the company''s innovative new Railhammer pickups (half pole-pieces, half rail), the Sensei RA represents the heavier side of Reverend''s lineup.
Read More
PRS SE 30 Amp Review
March 9, 2012
It's hard to separate the name Paul Reed Smith from the image and sound of his guitars. Whether it's dreams of a deephued 10-Top or Carlos Santana squeezing the
Read More
Fender Machete Amp Review
March 7, 2012
For players used to really big sounds, compact high-gain tube combo amps sometimes seem to demand too many tradeoffs to make the convenience and portability worth it. If you're
Read More
Lollar Pickups El Rayo Humbuckers Review
February 10, 2012
Jason Lollar is one of the undisputed kings of aftermarket pickups. In his workshop not far from Seattle, he crafts some of the finest, handmade Gibson- and Fender-style pickups
Read More


text:
Gold Soundz (Music)
With Chris Bay
Fri Apr 20th 2012 7.00pm–9.00pm
R=listener request. N=music new at the station. L=local music. Your purchases using the Buy it! links
Time zone: central
7:00pm | Levon Helm "Wide River to Cross" from Dirt Farmer (Vanguard 2007)Buy it! |
7:05pm | R.E.M. "Nightswimming" from Automatic for the People (? 1992) —88.1 KDHX benefit and R.E.M. tribute next Friday at Off BroadwayBuy it! |
7:09pm | The Mountain Goats "You Or Your Memory" from The Sunset Tree (4AD 2005) —Playing tomorrow at the GargoyleBuy it! |
7:14pm | Cursive "The Sun And Moon" from I Am Gemini (Saddle Creek 2012) N Buy it! |
7:18pm | Cymbals Eat Guitars "Wavelengths" from Lenses Alien (Barsuk) —Tonight at Off Broadway, with CursiveBuy it! |
7:22pm | Two Gallants "The Hand That Held Me Down" from Two Gallants (Saddle Creek Records 2007) —At the Firebird on 5/19 with Mariachi El BronxBuy it! |
7:27pm | Josh Ritter "Open Doors" from The Historical Conquests Of Josh Ritter (Sony 2007)Buy it! |
7:31pm | John Doe "Giant Step Backward" from Keeper (Yep Roc www.yeproc.com 2011) —Headlining the final night of Twangfest on 6/9Buy it! |
7:37pm | Chris Whitley "Weightless (Daniel Lanois Single Mix)" from Long Way Around: An Anthology 1991 - 2001 (Sony 2002)Buy it! |
7:40pm | Danger Mouse & Sparklehorse Feat.Suzanne Vega "Man Who Played God" from Dark Night Of The Soul (CAPITOL 2010)Buy it! |
7:45pm | Nada Surf "Clear Eye Clouded Mind (Acoustic)" from The Dulcitone Files (Barsuk 2012) N —At the Old Rock House on 6/25Buy it! |
7:48pm | Teenage Fanclub "Ain't That Enough" from Songs from Northern Britain (Creation Records www.mybloodyvalentine.org 1997)Buy it! |
7:52pm | Dear John "Frustrated Conversation" (BDR Records 2011) L —Playing Vintage Vinyl tomorrow at 1p.m.Buy it! |
7:55pm | Counting Crows "Ooh La La" from Underwater Sunshine (Collective Sounds 2012) N —Faces coverBuy it! |
8:04pm | Feist "Islands In the Stream" from The Reminder (Deluxe Edition) (Arts & Crafts Productions Inc. 2007)Buy it! |
8:08pm | Megafaun "Hope You Know" from Megafaun (Hometapes 2011)Buy it! |
8:13pm | Sarah Jaffe "The Way Sound Leaves A Room" from The Body Wins (Kirtland 2012) N Buy it! |
8:19pm | Lee Ranaldo "Off The Wall" from Between The Times & The Tides (Matador / Hostess 2012) N Buy it! |
8:22pm | Maps & Atlases "Fever" from Beware And Be Grateful (Barsuk Records www.barsuk.com 2012) N Buy it! |
8:27pm | King Tuff "Bad Thing" from King Tuff (Sub Pop Records www.subpop.com 2012) N —Playing LouFestBuy it! |
8:32pm | Cloud Nothings "Stay Useless" from Attack on Memory (Carpark 2012) N Buy it! |
8:35pm | Screaming Females "Rotten Apple" from Ugly (Don Giovanni 2012) N Buy it! |
8:38pm | Dinosaur Jr. "Almost Ready" from Beyond (Fat Possum 2007) —Playing LouFestBuy it! |
8:42pm | Yann Tiersen "The Gutter" from Skyline (Anti- www.anti.com 2012)Buy it! |
8:46pm | Mogwai "Yes! I Am a Long Way from Home" from Young Team (Chemikal Underground 1997)Buy it! |
8:51pm | Bon Iver "Re: Stacks" from For Emma, Forever Ago (Jagjaguwar 2008)Buy it! |
Like everything else, times are approximate. Spinitron and this station are not liable for errors or omissions.


text:
Triple Points at Homebase PLUS 15% off Everything this weekend at Homebase using Nector
15% offCode: NEC15
The site has been opened in a new window for you. Simply copy the Nectar voucher codes, make your purchase and then enter the voucher codes in the box provided at the basket checkout stage.
Instructions: .
Looks like offer for only Nectar account holders.
I have received email. People can see this offer in your account on Nectar.com
Triple Point plus 15% Off Eveything.
TIP: Get the Best Deals In Your Inbox
orSearch for active hot Home Vouchers
Newest Nectar voucher codes


text:
SOURCE: AtlanticW
comments powered by Disqus
Serbian cave produces oldest human ancestor in this part of Europe
tags: Europe, Serbia, fossil, NeanderthalA fragment of lower jaw recovered from a Serbian cave has now been dated as the oldest hominin ancestor found in this part of Europe. The fossil was dated to between 397,000 and 525,000 years old, a time when distinctly Neanderthal traits began to appear in Europe. The evolution of these traits was strongly influenced by periodic isolation of groups of individuals, caused by glacial episodes.
According to research published in February 2012 in the open access journal PLoS ONE, the individual probably evolved under different conditions than populations who inhabited more western parts of the continent during the same time frame....
comments powered by Disqus
- U.S. Textbook Skews History, Prime Minister of Japan Says
- Recalling a Film From the Liberation of the Camps
- Skull Fossil Offers New Clues on Human Journey From Africa
- Are crude conspiracies right? Research shows nations really do go to war over oil
- Famed SC civil rights protesters have convictions erased
- Columbia University professors Eric Foner, Alan Brinkley, and Alice Kessler-Harris to retire
- A powerhouse appropriations subcommittee is now headed by a historian: Republican Rep. Tom Cole (OK)
- Slavic scholars divided over a scholarship sponsored (and withdrawn) by Stephen F. Cohen
- Claire Strom to Step Down as Editor of Agricultural History
- Joan Peters's legacy assessed by one of her fiercest critics, Norman Finkelstein


text:
Get 1 more page view just for clicking
to like us on Facebook
Globe-Gazette (Newspaper) - April 29, 1965, Mason City, Iowa CUBA DOMINICAN WHERE MARINES LANDED More Marines land in WASHINGTON er 100 US Marines were land ed in the strifetorn Dominican Republic Thursday to protect Americans as bloody fighting engulfed Santo Domingo None of the some 2000 US citizens remaining in the Carib bean country have been harmed according to the latest reports reaching Washington Evacuation was proceeding by helicopter and ship Meanwhile shooting among rival Dominican forces was re ported continuing in the capital city with at least 400 Domini can dead 1200 wounded and armed mobs up to several hun dred strong sacking and loot ing in residential areas President Johnson ordered the Marines ashore from a na val force late Wednesday after getting word of the breakdown of local governmental authority in the Caribbean country The Dominicans were unable to guarantee safety for the Ameri cans Johnson told the nation in an emergency radioTV appear ance About 400 Leathernecks land ed by helicopter on the outskirts of Santo Domingo under John sons order Another 100 from a US naval force were landed early this morning There has been no exchange of fire between the American See refugees story Page 2 8 A Marines and Dominican fac tions the US officials added As of the latest count 215 Americans had been evacuated since the Marines landed Another 1100 Americans were evacuated Tuesday In Washington the Council he Organization of American States OAS was summoned into emergency session at US request to get a report on the action and to consider possible peace making efforts to end the bloody civil war US officials said the United States was taking no sides in the political turmoil engulfing the republic They said the Marines will serve no cause except to protect Americans and will stay there until their mission is com pleted Johnsons order marked the first such use of Marines in Lai Dom in America in years US offi cials expected criticism from Latin Americans mindful of American gunboat diplomacy of a bygone era But under in ternational law they said a nation can senci its forces to an other land to protect its nation als when the local government breaks down Reports reaching the Presi dent told of gun battles in Santo Domingo looting Communist agitation the collapse of local authority and word from the Dominicans that they could not guarantee the safety of the Americans there i mean ac VOL 10S Australia joins war in Viet To provide a battalion CANBERRA Australia AP Prime Minister Sir Robert Mcnzics announced in the House f fleprcscnlalives Thursdaj Auslralia will provide an infan try ball a lion for service in South Viol Nam Robert said the decision hcicl been made on request of die South Vietnamese govern ment for further military as sistance Defense Minister Shane riflfle declined lo say when the battalion would leave and where It would be deployed in South Viol Government sources had re ported earlier thai Australia would provide up to 800 army troops for combat duty with US and South Vietnamese forc es An Australian contingent of 800 men would be Ihe second largest unit from any of Ameri cas allies in South Viet Nam South Korea has 2000 troops thnrc but they are primarily a work force to build and repair roads bridges and railways Government sources said the decision to send Australian combat forces is not a result of the visit to Canberra last week by US presidential envoy Hen ry Cabot Lodge The Australian Cabinet bad considered the idea of a more positive involvement in Viet Nam for some time be fore Lodges visit the sources Lodge is touring fnendly Far Eastern and Pacific nations lo drum up support for US policy in Viet Nam Australia already has in South Viet Nam about 100 jungle comearly Thursday in a Mason City a Alf hospital of injuries received in Italian Air 1orcc transport unit The jungle advisers like thciri3 onccar accidcnl Wedncsda US counterparts are nol clas US officials gave this count of the crisis action In attempts to return to mocracy after 34 years of Tru jilio dictatorship Santo Domin go has had 8 governments in last 4 years The country also has been in deep economic trou ble The American ambassador W TapJey Bennett was recalled to Washington last Friday to see what could be done about it On Saturday rebels favoring the return of former President Juan Bosch sought to oust the Donald Reid Cabral govern ment The newspaper th Home Edition MASON CITY IOWA THURSDAY APRIL 29 19iS 10c Taper Conslfils of Two Ono Associated PICKS Full Wirca No arthquake hits coast Quad Cities dikes weaken THK IKICSS Ihe churning Mississippi River swirled through the Quincy area in westcentral Illinois Thursday while 140 miles upriver weakened dikes were battered in the Quad Cities of Iowa and Illinois The record flood burst dikes and spread debris littered water over thousands of acres from Dubuciue to Hannibal Mo Minnesota girl dies of injuries Colleen Marie Willsc 23 died sified as combat troops Bui they have1 beenIn combat areas for months and have suffered casualties in aclion including driven by Roberta Lynn Roscoe nno Vi II nil n I i i i one killed The US ambassador to SouthjRoscoc at pm Wednesday Viet tVnm Maxwell D Taylorjosl control of the car on High told Italian newsmen earlier wav 9 6miles northeast of this month that aside from the United Slates the nationals of supporting nations in South Viet Nam number over 3000 He saitl 37 nations have provided support or arc about to give support to South Viet Nam in a Buffalo Center hospital in The Sydney Sun said New fair condition Floodwaters rolled over more than 5600 acres of land south of Quincy but most of the city of 45 000 residents is situated on bluffs and is safe More than 15 square blocks ol the business district were inun rlated in Hannibal a city of 20 000 The river burst a levee in the South Quincy drainage district and engulfed more than SSOO acres Some 150 persons were evacuated Illinois 57 south of Quincy was closed and most rail traffic was stopped Highway 36 Mark Twain Bridge danger of being closed Two National Guardsmen on Ihe South Quincy levee when it broke were rescued by boats About miles south of Quin cy Ihe 550 residents of Hull 111 were ready to leave Officials said the levees protecting the town would not hold In Hannibal 23 families were evacuated from low areas as the river reached ils highest level in history Hannibal has no night Miss Wiltse daughter of Mr and Mrs Jerome Wiltse of Blue Earth was a passenger in a car 19 also of Blue Earth Miss Bancroft The car went off the road onto the soft shoulder and rolled back onto the paving The car which was owned by Scot Jagcrson of Frost Minn was demolished Miss Roscoe is Zealand is reported also to have agreed lo provide further mill tary aid lo Soull1 Viet Nam Tho accident was in vest ipa led by Highway Patrolman Charles Bird Resignation of Fiala rejected The llirccmcmbcr Iowa Em ployment Security Commission voted unanimously Thursday in DCS Moincs to reject the resig nation of Frank F Fiala Jr wlm has been manager of its Mason City office Henry E Carter a commis sion member old the Globe Gaxcttc that Fiala then was of fered the position of methods and planning analyst in the un employment insurance divisions central office in Des Moincs said he has accepted the position and will report for work next Monday Kiala was removed as man ager of he Mason City office two weeks ago when he was giv en his choice of resigning or being dismissed by George W Moorn of Des Moines chief of the Iowa State Employment Service Moore had said the forced resignation resulted from com plaints abqul Fiala Fiala had this comment Thurs day I feel the unanimous ac tion by the commission inrefus ing lo accept my resignation is complete exoneration of any such complaints Rather than to return lo the Mason City office with its in ternal dissension f accepted the Des Moincs appointment Fiala said will maintain hit residence in Mason City for the action came after a weeklong investigation during which the commission received many communication from community and business leaders in Mason City that were highly compli mentary to Mr Fiala In another action Thursday the commission appointed Clyde Prier of Oskaloosa as manager of the Mason City office and the were in Some Seattle buildings fall SEATTLE A heavy earthquake struck tha Pacific Northwest at am Mason City time Thurs day Two men were killed in Seattle and scores of per sons were injured here and elsewhere Damage was widespread Three women died of heart attacks during tha iuake here one in Tacoma and one in Olympia Debris from a building on South King St in Seat Chinatown crushed Vocational school bill is okayed FOOD STUFFS SCATTERED IN QUAKE Grocery clerks clean up littered packages of food stuii at a Seattle Wash supermarket shortly after an earthquake measuring an intensity of seven on the Richter scale struck the Pacific Northwest Thurs day present time Carter said the commission Civil Defense workers used a lo patrol against looting in the flooded Hannibal business district The US Weather Bu reau predicted a 25Afoot crest Gunday Upriver the crested Missis sippi surged against soggy dikes n the Quad Cities of Moline Rock Island and East Moline til and Davenport The area las a population of more than 270000 The Mississippi crestcrl at 224 feet in Rock Island Wednesday nore than 7 feet above flood stage Thousands of volunteers con tinued an aroundtheclock sandbagging job This is the time you can get killed during a flood one US Corps of Engineers official said Theyre all out there piling sandbags while the rivers ris ing but the moment it starts to drop a tenth of a foot or two Carter said Pricr a nalive wanl lo home And these dikes the longer theyre up the soggier they get They can go just like that You dont even know what hap pened Huge areas of the Quad Cities and Bcttenclorf were under wa ter Estimates of the number of persons evacuated in Illinois and Iowa have been placed at 10000 Officials sandbagged sand boils at the vital Point Missis sippi Dike near Bettendorf when water began seeping behind homes A new flank levee was prepared in Muscaline a sec ondary defense line for 000 homes and million worth of factories The levee backs up primary dikes Upriver from the Quad Cities Fulton and Albany in Whiteside County 111 were surrounded by flood walera Up to 10 feel of water gushed The Globes lowo Line The rirluslvc telephone wire was ujtrt to ffttitr Ififormitlnn for hlji nrlfclc of Burlington has been local office manager for the Iowa Employment Security Commis sion in Oskaloosa for the last two years and previously had served four years in the Des Moincs local office Ht is a former school prin cipal in Ollic and is a past presi dent of the Junior Chamber of Commerce in Oskaloosa Prior is a very capable and personable young man said Carter L E Shaw a field representa tive of the Iowa State Employ ment Service has been acting manager of the Mason City of fice in Fiaias absence ih Carries an aid increase DES MOINES AP Oppo nents of local property taxes for schools and colleges von a negotiated settlement in he area schools bill passed 45 2 by the Iowa Senate Wednes jday afternoon The bill retains the property tax provisions but carries a boost in state aid to the pro posed schools Sen Vern Lisle RClarinda said the aid hike he sponsored could make property taxes unnecessary for operation of the schools The Senate reconsidered its Tuesday action eliminating the property lax provisions from the bill and retained them by a 30 27 vole Then it adopted Lisles amend ment tn raise from SI50 to a day the amount of state aid to through deserted Gulfport III be allowed student per day for each EARTHQUAKE SHAKES BRICKS A sharp earthquake shook loose bricks from the tops of stores m the Seattle business district Thursday I he stores were located near the King Street railroad station Similar damage was reported over a wide area as a result of the quake Gov Harold Hughes has asked the legislature to appropriate million to help pay financing costs for four of the area schools with a committee to decide which ones get the mon ey Critics said this would dis criminate against other areas which would receive only the regular state aid i The bill would itlow two or more counties with approval of the Slate Department of Public Instruction to form an area lo set up a technical vocational school a twoyear college or a combination of the two i Each county in the area would jelcct one member to an area governing board which could levy one mill of property taxes to operate the school and with approval of 60 per cent of the areas voters one additional mill to finance construction kl FOCUS on North A blueprint of Hawkeye Naturama By CHARLES W WALK North Iowa Director Although Iowas most amhi lious outdoor recreation proj ect is barely out of the embryo stage its scope staggers the imagination The Hawkeye Naturama was the brainchild of the Governors Committee on Conservation of Outdoor Re sources Called a master recreation outdooreducation development by the committee in a report made public in December the project has four major purposes They are To provide a wide choice of recreational activities for all ages from youth to senior citi zens To provide increased op portunity for outdoor education activities for all ages To provide training for good outdoor manners and to develop a sound conservation philosophy To attract tourists whose participation will help defray costs of the operation and whose expenditures will add to the economy of the state The size of the proposed proj ect is impressive The commit Ices report calls Jar an area covering between 5000 and 6000 acres This would make it more than twice as bigas the present largest state sauqua State Park in Van Buren County The cost of the project is estimated at million Although tHe Governors com mittee didnt offer a suggestion for the location of the project it FOCUS turn 2 and killed Aclolphus Lewis age undetermined A collapsing wall at the Fisher Flouring Mill fatally injured a warehouseman Ray Haughton 52 Three others in the mill including H state grain inspector were in jured Mrs James Johnson 56 died of a heart attack here while alking on the telephone when he quake hit In Tacoma Mrs Ethel Webb 75 under treatment for a heart ailment had a heart eizure and died At Olympia he heart attack victim was Senola Iorcnz a woman in her 0s who was found in her hotel m Steven Coughtry 21 was in serious condition in a Seattle hospital with a head injury aft a huge section of brick roof gable toppled on his car in the old part of the city His wife aulinc 27 suffered minor in juries Rescuers worked 15 min ites to free the trapped coupla rrom the flattened ear Coughtry said he had j u s t parked at the curb so his wife could apply for a job when Ins bricks came raining clown There were dozens of other injuries in Seatllc many in tha older sections near the water front where brick veneer walls peeled off and tumbled to the street The quake rated at an in tensity of more than 7 on tha Richler scale was felt from Coos Bay on the Oregon coast to southern British Columbia and east as far as Idaho The earthquake belt was at QUAKE turn to Page 2 Inside The Globe Editorials 4 Clear Lakt news 47 Society news 1314 Cub Gazette 15 Mason City news U1722 Latest markets Comics 20 Classified pages 2223 North Iowa 24 North Iowa Weather outlook Clear and warmer Thursday igM lows in the Fair Friday highs 4775 Weathar Details 1 SAM I its OATI1H41U r
Once upon a time newspapers were our main source of information. Now those old newspapers are a reliable source for hundreds of years of history and secrets of the past. Now you can search for people, places, and events without the hassle of sorting through mountains of papers!
Newspaper Archive is the world's largest online newspaper database featuring over 155+ million newspaper pages. Plus our database expands by one newspaper page per second for a total of around 2.5 million pages per month! The value of your membership grows along with it.
Those looking to find out more about their forefathers can empower their genealogy search with Newspaper Archive. Within our massive database, users can search ancestors' names for news stories and obituaries. We must understand our past to understand our future!
24 hours a day Monday-Saturday
Your full introductory membership payment will be credited toward the cost of full membership any time you choose to upgrade!
"It is amazing how easy and exciting it is to access all of this information! I found hundreds of articles about my relatives from Germany! Well worth the subscription!" - Michael S.
"I love this site. It's interesting to read articles about different family members. I've found articles as well as an obituary about an uncle who passed away before I was born, and another about a great aunt. It's great for helping with genealogy." - Patricia T.
"A great research tool. Allows me to view events and gives me incredible insight into the stories of the past." - Charles S.
Uncover 400+ Years
of Newspaper Archives
(1607 to today!)
Research Newspaper Articles from 19 Countries
& all 50 U.S. States
Explore 6,200+ Current &
Historical Newspaper Titles
and Counting!


text:
Mystery Solved: Gloucester Man Returns Ring To Owner
It was the Facebook message Daniel Roark has been waiting for.
Gloucester Scuba Diver Searching For Owner Of Ring Found In Mexico
An avid scuba diver who found a wedding ring turned to Facebook to locate its owner.
Travel To The Wonders Of The World
Here's a quick guide for families to the New Seven Wonders of the World and the only surviving original Seven Wonders of the World.
Most Popular Destination Wedding Locations
Planning a destination wedding offers couples the opportunity to plan a unique wedding where they can say "I do" in front of a breathtaking background. All of these locations offer unique venues, luxurious resorts and beautiful settings.
The World Cup Soccer Coach You Have To See To Believe
This year's World Cup has provided some incredible games along with some unbelievable moments including the bite felt around the world. There is one personality though who is winning the heart of soccer fans from […]
The 25 Best Beaches In The World
After months of polar vortexes it's finally almost time to lay out in the sun and sand. To help get the ball rolling, we rounded up the 25 best beaches in the world.
Boomers Should Be Thinking About Three Things
Baby Boomers need to be thinking about retirement and the when, what, and where.
Gov. Patrick Leading Weeklong Trade Mission To Mexico, Panama
Gov. Deval Patrick heads to Panama Monday with state officials and business leaders.
International Trips You Can Do In A Weekend
Many international destinations close to home may prove to be a better option than South Beach or Miami. Among the best international trips that can be done in a weekend are the Bahamas, Cabo San Lucas, Cancun, Montreal and Vancouver.
Bradley Jay: 'Join Me For The Finale Of Our Wild West Rail Adventure.'
This video is part two of the Jay Talking cross-country rail trip from New Orleans to Los Angeles.


text:
Celebrities at Comic-Con 2013 | Pictures
Comic-Con Is On! See All the Stars in San Diego
Comic-Con has wrapped up, and we're taking a look at all the famous faces to show up in San Diego. A-listers like Tom Cruise, Sandra Bullock, and Scarlett Johansson all popped up to promote their projects, and the cast of The Hunger Games: Catching Fire plugged the much-anticipated sequel with Jennifer Lawrence, Liam Hemsworth, and Josh Hemsworth reuniting on the red carpet to introduce the movie's full trailer. The stars of Divergent also hit the ground running as Shailene Woodley stayed close to her castmates Theo James, Zoë Kravitz, and Maggie Q during the film's panel, while Andrew Garfield and Jamie Foxx stopped in to talk about The Amazing Spider-Man 2. Meanwhile, Kristen Bell showed up in style to get the crowd pumped for the Veronica Mars movie, and the X-Men: Days of Future Past panel brought out iconic stars like Hugh Jackman, Halle Berry, and Ian McKellen. Another famous Ian — Somerhalder, that is — kept things festive before joining his castmates for the Vampire Diaries panel on day three. Click through to see all the stars at Comic-Con!
Shailene Woodley
Tom Cruise
Jennifer Lawrence
Kristen Bell
- Eli Manning's wife, Abby, gives birth to a baby girl — Us Weekly
- Kim Kardashian has really mastered one kind of selfie — HuffPost Celebrity
- Who is Rachel McAdams's mystery dinner date? — Lainey Gossip
- Iggy Azalea backtracks after a social media rant — Dlisted
- Rihanna strips down for a sexy spread — ET
- Why Reese Witherspoon dressed up as a guy — JustJared
- Hilary Duff is one hot mom — Hollywood Tuna
- Eddie Murphy shares exciting SNL news — Pink Is the New Blog
- Why Suge Knight is under arrest — The Superficial


text:
Complete statistics and more, for any player. Enter as much of the player's name as you know. |
Find statistics for every player in a single organization and navigate the farm system for any organization. Start by selecting an organization. |
View the statistic leaders for every league, individual and team and check to see the scoreboard for each league. |
Darrell Chaney, ss, Braves
Proper Name: Darrell Lee Chaney Born: March 9, 1948 in Hammond, Ind. High School: Ht.: 6-1 Wt.: 190 Bats: B Throws: R |


text:
You're leaving Wedding Paper Divas to see the Deal of the Day at our parent site, Tiny Prints.
You won't be able to dismiss this by usual means (escape or click button), but you can close it programatically based on user choices or actions.
"I personalized my response cards, but I wanted text that was not an "option" in the template. The design team added what I wanted and enhanced it in a way I did not think of. When I got the proof, I was extremely happy with the changes."
"The rsvp cards were beautiful. When they were ordered, I needed to make a few design changes. This was done by the next morning and I had a proof waiting for me in my email by the next morning. It came out exactly how I had hoped. This company is very good and my package arrived early!"
"I have already received compliments on how beautiful this and my matching invitations are from close friends and family that I have shared the invites and response cards with, as we haven't sent them out yet. But I am completely happy with how they turned out and I can't wait for everyone else to see them!"
"My fiance and I worked with WPD professional designers to create the perfect response card. We asked them to capitalize the first word in each statement, as well as to fit our food options (they were really long titles) in the space provided. They did both beautifully and the response cards turned out great! I would definitely recommend this product with its matching invitation!"
"Turned out great! Ordering was easy and I like the option to message the designer. I love weddingpaperdivas received several free samples which help me narrow down my choices. I also love their attention to detail, right down to packaging. I felt very well taken care of. Thank you!!!"


text:
This place is beyond basic. Just a basic bed and bathroom. The shower was so weak it may had well not worked. The cleaning products smelt awful and made me feel ill. To top it off the staff had no basic manors what so ever. You can definitely get a better stay for cheaper else where.
- Reservation Options:
- TripAdvisor is proud to partner with Priceline, Booking.com and Odigeo so you can book your Hostal Nicolau reservations with confidence. We help millions of travelers each month to find the perfect hotel for both vacation and business trips, always with the best discounts and special offers.
- Also Known As:
- Hostal Nicolau Ibiza, Spain


text:
Remember Me
Forgot your password?
Haven't registered yet?
1/1/2015 » 6/30/20152015 Human Resources Webinar Series
1/1/2015 » 6/30/20152015 Advanced ACH Specialist Webinar Series
1/1/2015 » 6/30/20152015 IRA Webinar Series
1/1/2015 » 6/30/20152015 BSA Compliance Webinar Series
1/1/2015 » 6/30/20152015 Required Compliance Webinar Series
The National Credit Union Association's (NCUA) Office of Small Credit Union Initiatives (OSCUI) is bringing its free workshop series to the Twin Cities. Topics to be covered during this training include:
Financial assistance to attend the training session is available to credit unions under $10 million in assets, chartered less than 10 years, or low-income designated. Email firstname.lastname@example.com to apply.
Time: 7:45 a.m.-4 p.m.Location: Minneapolis Airport Marriott, BloomingtonCost: The workshop is free. Room blocks are available at the Marriott for a special rate of $109 per night, if registred by April 20.