text:
Continuous Delivery Using Maven
I'm currently working on a continuous delivery system where I work, so I thought I would write something up about what I'm doing. The continuous delivery system, in a nutshell, looks a bit like this:
I started out with a bit of a carte blanche with regards to what tools to use, but here's a list of what was already in use, in one form or another, when I started my adventure:
- Ant (the main build tool)
- Maven (used for dependency management)
- CruiseControl
- CruiseControl.Net
- Go
- Monit
- JUnit
- js-test-driver
- Selenium
- Artifactory
- Perforce
The decision of which of these tools to use for my system was influenced by a number of factors. Firstly I'll explain why I decided to use Maven as the build tool (shock!!).
I'm a big fan of Ant, I'd usually choose it (or probably Gradle now) over Maven any day of the week, but there was already an existing Ant build system in place, which had grown a bit monolithic (that's my polite way of saying it was a huge mess), so I didn't want to go there! And besides, the first project that would be going into the new continuous delivery system was a simple Java project – way too straightforward to justify rewriting the whole ant system from scratch and improving it, so I went for Maven. Furthermore, since the project was (from a build perspective) fairly straightforward, I thought Maven could handle it without too much bother. I've used Maven before, so I've had my run-ins with it, and I know how hard it can be if you want to do anything outside of "The Maven Way". But, as I said, the project I was working on seemed pretty simple so Maven got the nod.
GO was the latest and greatest C.I. server in use, and the CruiseControl systems were a bit of a handful already, so I went for GO (also I'd never used it before so I thought that would be cool, and it's from Thoughtworks Studios, so I thought it might be pretty good). I particularly liked the pipeline feature it has, and the way it manages each of its own agents. A colleague of mine, Andy Berry, had already done quite a bit of work on the GO C.I. system, so there was already something to start from. I would have gone for Jenkins had there not already been a considerable investment in GO by the company prior to my arrival.
I decided to use Artifactory as the artifact repository manager, simply because there was already an instance installed, and it was sort-of already setup. The existing build system didn't really use it, as most artifacts/dependencies were served from network shares. I would have considered Nexus if Artifactory wasn't already installed.
I setup Sonar to act as a build analysis/reporting tool, because we were starting with a Java project. I really like what Sonar does, I think the information it presents can be used very effectively. Most of all I just like the way in which it delivers the information. The Maven site plugin can produce pretty much all of the information that Sonar does, but I think the way Sonar presents the information is far superior – more on this later.
Perforce was the incumbent source control system, and so it was a no-brainer to carry on with that. In fact, changing the SC system wasn't ever in question. That said, I would have chosen Subversion if this was an option, just because it's so utterly freeeeeeee!!!
That was about it for the tools I wanted to use. It was up to the rest of the project team to determine which tools to use for testing and developing. All that I needed for the system I was setting up was a distinction between the Unit Tests, Acceptance Tests and Integration Tests. In the end, the team went with Junit, Mockito and a couple of in-house apps to take care of the testing.
The Maven Build, and the Joys of the Release Plugin!
The idea behind my Continuous Delivery system was this:
- Every check-in runs a load of unit tests
- If they pass it runs a load of acceptance tests
- If they pass we run more tests – Integration, scenario and performance tests
- If they all pass we run a bunch of static analysis and produce pretty reports and eventually deploy the candidate to a "Release Candidate" repository where QA and other like-minded people can look at it, prod it, and eventually give it a seal of approval.
This is the basic outline of a build pipeline:
Maven isn't exactly fantastic at fitting in to the pipeline process. For starters we're running multiple test phases, and Maven follows a "lifecycle" process, meaning that every time you call a particular pipeline phase, it runs all the preceding phases again. Our pipeline needs to run the maven Surefire plugin twice, because that's the plugin we use to execute our different tests. The first time we run it, we want to execute all the unit tests. The second time we run it we want to execute the acceptance tests – but we don't want it to run the unit tests again, obviously.
You probably need some familiarity with the maven build lifecycle at this point, because we're going to be binding the Surefire plugin to two different phases of the maven lifecycle so that we can run it twice and have it run different tests each time. Here is the maven lifecycle, (for a more detailed description check out the Maven's own lifecycle page)
Clean Lifecycle
- pre-clean
- clean
- post-clean
Default Lifecycle
- validate
- initialize
- generate-sources
- process-sources
- generate-resources
- process-resources
- compile
- process-classes
- generate-test-sources
- process-test-sources
- generate-test-resources
- process-test-resources
- test-compile
- process-test-classes
- test
- prepare-package
- package
- pre-integration-test
- integration-test
- post-integration-test
- verify
- install
- deploy
Site Lifecycle
- pre-site
- site
- post-site
- site-deploy
So, we want to bind our Surefire plugin to both the test phase to execute the UTs, and the integration-test phase to run the ATs, like this:
<plugin> <!-- Separates the unit tests from the integration tests. --> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> -Xms256m -Xmx512m <skip>true</skip> </configuration> <executions> <execution> <id>unit-tests</id> <phase>test</phase> <goals> <goal>test</goal> </goals> <configuration> <testClassesDirectory> target/test-classes </testClassesDirectory> <skip>false</skip> <includes> <include>**/*Test.java</include> </includes> <excludes> <exclude>**/acceptance/*.java</exclude> <exclude>**/benchmark/*.java</exclude> <include>**/requestResponses/*Test.java</exclude> </excludes> </configuration> </execution> <execution> <id>acceptance-tests</id> <phase>integration-test</phase> <goals> <goal>test</goal> </goals> <configuration> <testClassesDirectory> target/test-classes </testClassesDirectory> <skip>false</skip> <includes> <include>**/acceptance/*.java</include> <include>**/benchmark/*.java</include> <include>**/requestResponses/*Test.java</exclude> </includes> </configuration> </execution> </executions> </plugin>
Now in the first stage of our pipeline, which polls Perforce for changes, triggers a build and runs the unit tests, we simply call:
mvn clean test
This will run the surefire test phase of the maven lifecycle. As you can see from the Surefire plugin configuration above, during the "test" phase execution of Surefire (i.e. this time we run it) it'll run all of the tests except for the acceptance tests – these are explicitly excluded from the execution in the "excludes" section. The other thing we want to do in this phase is quickly check the unit test coverage for our project, and maybe make the build fail if the test coverage is below a certain level. To do this we use the cobertura plugin, and configure it as follows:
<plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>cobertura-maven-plugin</artifactId> <version>2.4</version> <configuration> <instrumentation> <excludes><!-- this is why this isn't in the parent --> <exclude>**/acceptance/*.class</exclude> <exclude>**/benchmark/*.class</exclude> <exclude>**/requestResponses/*.class</exclude> </excludes> </instrumentation> <check> <haltOnFailure>true</haltOnFailure> <branchRate>80</branchRate> <lineRate>80</lineRate> <packageLineRate>80</packageLineRate> <packageBranchRate>80</packageBranchRate> <totalBranchRate>80</totalBranchRate> <totalLineRate>80</totalLineRate> </check> <formats> <format>html</format> <format>xml</format> </formats> </configuration> <executions> <execution> <phase>test</phase> <goals> <goal>clean</goal> <goal>check</goal> </goals> </execution> </executions> </plugin>
To get the cobertura plugin to execute, we need to call "mvn cobertura:cobertura", or run the maven "verify" phase by calling "mvn verify", because the cobertura plugin by default binds to the verify lifecycle phase. But if we delve a little deeper into what this actually does, we see that it actually runs the whole test phase all over again, and of course the integration-test phase too, because they precede the verify phase, and cobertura:cobertura actually invokes execution of the test phase before executing itself. So what I've done is to change the lifecycle phase that cobertura binds to, as you can see above. I've made it bind to the test phase only, so that it only executes when the unit tests run. A consequence of this is that we can now change the maven command we run, to something like this:
mvn clean cobertura:cobertura
This will run the Unit Tests implicitly and also check the coverage!
In the second stage of the pipeline, which runs the acceptance tests, we can call:
mvn clean integration-test
This will again run the Surefire plugin, but this time it will run through the test phase (thus executing the unit tests again) and then execute the integration-test phase, which actually runs our acceptance tests.
You'll notice that we've run the unit tests twice now, and this is a problem. Or is it? Well actually no it isn't, not for me anyway. One of the reasons why the pipeline is broken down into sections is to allow us to separate different tasks according to their purpose. My Unit Tests are meant to run very quickly (less than 3 minutes ideally, they actually take 15 seconds on this particular project) so that if they fail, I know about it asap, and I don't have to wait around for a lifetime before I can either continue checking in, or start fixing the failed tests. So my unit test pipeline phase needs to be quick, but what difference does an extra few seconds mean for my Acceptance Tests? Not too much to be honest, so I'm actually not too fussed about the unit tests running for a second time. If it was a problem, I would of course have to somehow skip the unit tests, but only in the test phase on the second run. This is doable, but not very easy. The best way I've thought of is to exclude the tests using SkipTests, which actually just skips the execution of the surefire plugin, and then run your acceptance tests using a different plugin (the Antrun plugin for instance).
The next thing we want to do is create a built artifact (a jar or zip for example) and upload it to our artifact repository. We'll use 5 artifact repositories in our continuous delivery system, these are:
- A cached copy of the maven central repo
- A C.I. repository where all builds go
- A Release Candidate (RC) repository where all builds under QA go
- A Release repository where all builds which have passed QA go
- A Downloads repository, from where the downloads to customers are actually served
Once our build has passed all the automated test phases it gets deployed to the C.I. repository. This is done by configuring the C.I. repository in the maven pom file as follows:
and calling:
mvn clean deploy
Now, since Maven follows the lifecycle pattern, it'll rerun the tests again, and we don't want to do all that, we just want to deploy the artifacts. In fact, there's no reason why we shouldn't just deploy the artifact straight after the Acceptance Test stage is completed, so that's what exactly what we'll do. This means we need to go back and change our maven command for our Acceptance Test stage as follows:
mvn clean deploy
This does the same as it did before, because the integration-test phase is implicit and is executed on the way to reaching the "deploy" phase as part of the maven lifecycle, but of course it does more than it did before, it actually deploys the artifact to the C.I. repository.
One thing that is worth noting here is that I'm not using the maven release plugin, and that's because it's not very well suited to continuous delivery, as I've noted here. The main problem is that the release plugin will increment the build number in the pom and check it in, which will in turn kick off another build, and if every build is doing this, then you'll have an infinitely building loop. Maven declares builds as either a "release build" which uses the release plugin, or a SNAPSHOT build, which is basically anything else. But I want to create releases out of SNAPSHOT builds, but I don't want them to be called SNAPSHOT builds, because they're releases! So what I need to do is simply remove the word SNAPSHOT from my pom. Get rid of it entirely. This will now build a normal "snapshot" build, but not add the SNAPSHOT label, and since we're not running the release plugin, that's fine (WARNING: if you try removing the word snapshot from your pom and then try to run a release build using the release plugin, it'll fail).
Ok, let's briefly catch up with what our system can now do:
- We've got a build pipeline with 2 stages
- It's executed every time code is checked-in
- Unit tests are executed in the first stage
- Code coverage is checked, also in the first stage
- The second stage runs the acceptance tests
- The jar/zip is built and deployed to our ci repo, this also in the second stage of our pipeline
So we have a jar, and it's in our "ci" repo, and we have a code coverage report. But where's the rest of our static analysis? The build should report a lot more than just the code coverage. What about coding styles & standards, rules violations, potential defect hot spots, copy and pasted code etc and so forth??? Thankfully, there's a great tool which collects all this information for us, and it's called Sonar.
I won't go into detail about how to setup and install Sonar, because I've already detailed it here.
Installing Sonar is very simple, and to get your builds to produce Sonar reports is as simple as adding a small amount of configuration to your pom, and adding the Sonar plugin to you plugin section. To produce the Sonar reports for your project, you can simply run:
mvn sonar:sonar
So that's exactly what we'll do in the next section of our build pipeline.
So we now have 3 pipeline sections and were producing Sonar reports with every build. The Sonar reports look something like this:
As you can see, Sonar produces a wealth of useful information which we can pour over and discuss in our daily stand-ups. As a rule we try to fix any "critical" rule violations, and keep the unit test coverage percentage up in the 90s (where appropriate). Some people might argue that unit test coverage isn't a valuable metric, but bear in mind that Sonar allows you to exclude certain files and directories from your analysis, so that you're only measuring the unit test coverage of the code you want to have covered by unit tests. For me, this makes it a valuable metric.
Moving on from Sonar now, we get to the next stage of my pipeline, and here I'm going to run some Integration Tests (finally!!). The ITs have a much wider scope than the Unit Test, and they also have greater requirements, in that we need an Integration Test Environment to run them in. I'm going to use Ant to control this phase of the pipeline, because it gives me more control than Maven does, and I need to do a couple of funky things, namely:
- Provision an environment
- Deploy all the components I need to test with
- Get my newly built artifact from the ci repository in Artifactory
- Deploy it to my IT environment
- Kick of the tests
The Ant script is fairly straightforward, but I'll just mention that getting our artifact from Artifactory is as simple as using Ant's own "get" task (you don't need to use Ivy juts to do this):
<get src="${artifactory.url}/${repo.name}/${namespace}/${jarname}-${version}" dest="${temp.dir}/${jarname}-${version}" />
The Integration Test stage takes a little longer than the previous stages, and so to speed things up we can run this stage in parallel with the previous stage. Go allows us to do this by setting up 2 jobs in one pipeline stage. Our Sonar stage now changes to "Reports and ITs", and includes 2 jobs:
<jobs> <job name="sonar"> <tasks> <exec command="mvn" args="sonar:sonar" workingdir="JavaDevelopment" /> </tasks> <resources> <resource>windows</resource> </resources> </job> <job name="ITs"> <tasks> <ant buildfile="run_ITs.xml" target="build" workingdir="JavaDevelopment" /> </tasks> <resources> <resource>windows</resource> </resources> </job> </jobs>
Once this phase completes successfully, we know we've got a half decent looking build! At this point I'm going to throw a bit of a spanner into the works. The QA team want to perform some manual exploratory tests on the build. Good idea! But how does that fit in with our Continuous Delivery model? Well, what I did was to create a separate "Release Candidate" (RC) repository, also known as a QA repo. Builds that pass the IT stage get promoted to the RC repo, and from there the QA team can take them and do their exploratory testing.
Does this stop us from practicing "Continuous Delivery"? Well, not really. In my opinion, Continuous Delivery is more about making sure that every build creates a potentially releasable artifact, rather that making every build actually deploy an artifact to production – that's Continuous Deployment.
Our final stage in the deployment pipeline is to deploy our build to a performance test environment, and execute some load tests. Once this stage completes we deploy our build to the Release Repository, as it's all signed off and ready to handover to customers. At this point there's a manual decision gate, which in reality is a button in my CI system. At this point, only the product owner or some such responsible person, can decide whether or not to actually release this build into the wild. They may decide not to, simply because they don't feel that the changes included in this build are particularly worth deploying. On the other hand, they may decide to release it, and to do this they simply click the button. What does the button do? Well, it simply copies the build to the "downloads" repository, from where a link is served and sent to customers, informing them that a new release is available – that's just the way things are done here. In a hosted environment (like a web-based company), this button-press could initiate the deploy script to deploy this build to the production environment.
A Word on Version Numbers
This system is actually dependant on each build producing a unique artifact. If a code change is checked in, the resultant build must be uniquely identifiable, so that when we come to release it, we know we're releasing theexact same build that has gone through the whole pipeline, not some older previous build. To do this, we need to version each build with a unique number. The CI system is very useful for doing this. In Go, as with most other CI systems, you can retrieve a unique "counter" for your build, which is incremented every time there's a build. No two builds of the same name can have the same counter. So we could add this unique number to our artifact's version, something like this (let's say the counter is 33, meaning this is the 33rd build):
This is good, but it doesn't tell us much, apart from that this is the 33rd build of "myproject". A more meaningful version number is the source control revision number, which relates to the code commit which kicked off the build. This is extremely useful. From this we can cross reference every build to the code in our source control system, and this saves us from having to "tag" the source code with every build. I can access the source control revision number via my CI system, because Go sets it as an environment variable at build time, so I simply pass it to my build script in my CI system's xml, like this:
cobertura:cobertura -Dp4.revision=${env.GO_PIPELINE_LABEL} -Dbuild.counter=${env.GO_PIPELINE_COUNTER"
p4.revision and build.counter are used in the maven build script, where I set the version number:
If my Perforce check-in number was 1234, then this build, for example, will produce:
And that just about covers it. I hope this is useful to some people, especially those who are using Maven and are struggling with the release plugin!
How Mature Is Your Continuous Integration?
As I'm sure I've
ranted about mentioned in the past, Continuous Integration is far more than just a collection of tools and scripts. It's "a practice", a way of doing something, and it has to be part of our working culture to be truly effective.
I've seen instances of CI implemented which are truly magnificent, using great tools, great architecture, very smart scripts and a good process, but I've also seen this system fail. Unfortunatley, it's all too easy to have your wonderful system, and then have it ignored unless there is the right level of buy-in from the people who this system is meant to cater for, nameley development, QA and Management.
The way I currently see it, is that there are a number of levels of CI maturity. I'll just call these levels "Level 1", "Level 2" and so on, rather than "Highly Immature", "Stroppy Teenager" etc:
Level 1
No CI tools to speak of, no CI process. I've been there. Builds take about a day to get working. It's a nightmare. I still shiver just thinking about it.
Level 2
We've got some CI tools like cruise control, repeatable builds, but no CI process. We've basically got a front-end to a system of chaos. Most of the builds are broken, but you now have a nice way of visualising that, and nobody cares. It's Level 1 with a pretty wrapper on it.
Level 3
We've got a system, but not the right tools. We've got a policy of running our tests locally before checking-in, and some poor soul somewhere is left with the task of making the "official" builds for passing to QA. These builds will usually fail and everyone will have to chip in to help sort out the mess until a build can be made. We desperately need a computer to do this build lark for us!
Level 4
We've got some rudimentary tools, like Cruise Control or Jenkins, but we're not using them to their full capacity, but we've got a build monkey! The build monkey does his or her best to make sure that people are aware that they've broken the build. The build monkey sets up the CI system and makes changes whenever necessary. The build monkey is the first to look into every build failure. The build monkey goes on holiday and the whole place grinds to a halt.
Level 5
We've got the tools, we've got the process, but nobody's listening to us!!! All the tools are in place, we've got a suitable CI system and maybe we're even trying to do continuous delivery. The build system is virtualised and we have a release engineering team (the collective noun for a group of build monkeys is a "release engineering"). The only trouble is, the unit test coverage is apalling and people don't fix their broken builds, despite the fact that we've got a nice shiny wiki page saying we should aim to have 95% unit test coverage and broken builds should be fixed within 3o mins.
Level 6
We've got the tools, the processes and we've got management buy-in! This is looking good, we now have a lovely system, which our team of build engineers looks after, and we have a semi-compliant dev team who get told off if they don't play ball! We're all heading in roughly the right direction
Level 7
We've got the tools, the process and the right culture! Everyone has buy-in to the build system. Developers and build enginners alike can be trusted to edit build files and even the CI configuration because we all clearly understand what we're trying to achieve. Best practices are being observed and so our build engineering team don't need to spend all day chasing people or working on trivial tasks. Our time can be better invested and productivity increased.
Ultimately we're all responsible for looking after the CI system – it's for our own benefit afterall. As a developer I want to make sure I have some fast and reliable feedback on the quality of my code changes. If I see my build has failed, I would actually want to find out why, rather than ignore it. As a build engineer I want our CI system to be providing useful feedback to our developers, and useful information to management – if it isn't, or if this "useful" information isn't being acted upon, then it's not really useful at all, and my job is less fulfilling! All of this means that we all have some responsibility to occasionally look under the hood and see what's going on, and try to figure out why the system is telling me that something isn't working quite right.
The hardest part to get right, particularly in distributed teams or in companies over a certain size, is the culture. You have to have a team of build engineers and developers who all understand the big picture. Developers need to understand that they are instrumental in making the system work – their input is vital, and they have to understand clearly what benefits they will personally get from this system, otherwise they'll ignore it. Build engineers in turn need to understand that the more you are able to devolve the ownership of the system, the better it can work, and the more buy-in you will get in return. The build system needs guardians, but it doesn't need treating like a holy relic.
Coping With Big C.I.
Last night I went along to another C.I. meetup to listen to Tom Duckering, a consultant devops at Thoughtworks, deliver a talk about managing a scaled-up build/release/CI system. In his talk, Tom discussed Continuous Delivery, common mistakes, best practices, monkeys, Jamie Oliver and McDonald's.
Big CI and Build Monkeys
First of all, Tom started out by defining what he meant by "Big CI".
Big CI means large-scale build and Continuous Integration systems. We're talking about maybe 100+ bits of software being built, and doing C.I. properly. In Big CI there's usually a dedicated build team, which in itself raises a few issues which are covered a bit later. The general formula for getting to Big CI, as far as build engineers (henceforth termed "build monkeys") are concerned goes as follows:
build monkey + projects = build monkeys
build monkeys + projects + projects = build monkey society
build monkey society + projects = über build monkey
über build monkey + build monkey society + projects = BIG CI
What are the Issues with Big CI?
Big CI isn't without its problems, and Tom presented a number of Anti-Patterns which he has witnessed in practice. I've listed most of them and added my own thoughts where necessary:
Anti-Pattern: Slavish Standardisation
As build monkeys we all strive for a decent degree of standardisation – it makes our working lives so much easier! The fewer systems, technologies and languages we have to support the easier, it's like macro configuration management in a way – the less variation the better. However, Tom highlighted that mass standardisation is the work of the devil, and by the devil of course, I mean McDonald's.
McDonald's vs Jamie Oliver
Jamie Oliver might me a smug mockney git who loves the sound of his own voice BUT he does know how to make tasty food, apparently (I don't know, he's never cooked for me before). McDonald's make incredibly tasty food if you're a teenager or unemployed, but beyond that, they're pretty lame. However, they do have consistency on their side. Go into a McDonald's in London and have a cheeseburger – it's likeley to taste exactly the same as a cheeseburger from a McDonald's in Moscow (i.e. bland and rubbery, and nothing like in the pictures either). This is thanks to mass standardisation.
Jamie Oliver, or so Tom Duckering says (I'm staying well out of this) is less consistent. His meals may be of a higher standard, but they're likely to be slightly different each time. Let's just say that Jamie Oliver's dishes are more "unique".
Anyway, back to the Continuous Integration stuff! In Big CI, you can be tempted by mass standardisation, but all you'll achieve is mediocrity. With less flexibility you're preventing project teams from achieving their potential, by stifling their creativity and individuality. So, as Tom says, when we look at our C.I. system we have to ask ourselves "Are we making burgers?"
Are we making burgers?
– T. Duckering, 2011
Anti-Pattern: The Team Who Knew Too Much
There is a phenomenon in the natural world known as "Build Monkey Affinity", in which build engineers tend to congregate and work together, rather than integrate with the rest of society. Fascinating stuff. The trouble is, this usually leads the build monkeys to assume all the responsibilities of the CI system, because their lack of integration with the rest of the known world makes them isolated, cold and bitter (Ok, I'm going overboard here). Anyway, the point is this, if the build team don't work with the project team, and every build task has to go through the build team, there will be a disconnect, possibly bottlenecks and a general lack of agility. Responsibility for build related activities should be devolved to the project teams as much as possible, so that bottlenecks and disconnects don't arise. It also stops all the build knowledge from staying in one place.
Anti-Pattern: Big Ball of CI Mud
This is where you have a load of complexity in your build system, loads of obscure build scripts, multitudes of properties files, and all sorts of nonsense, all just to get a build working. It tends to happen when you over engineer your build solution because you're compensating for a project that's not in a fit state. I've worked in places where there are projects that have no regard for configuration management, project structures in source control that don't match what they need to look like to do a build, and projects where the team have no idea what the deployed artifact should look like – so they just check all their individual work into source control and leave it for the build system to sort the whole mess out. Obviously, in these situations, you're going to end up with some sort of crazy Heath Robinson build system which is bordering on artificial intelligence in its complexity. This is a big ball of CI mud.
Anti-Pattern: "They" Broke MY Build…
This is a situation that often arises when you have upstream and downstream dependencies. Let's say your build depends on library X. Someone in another team makes a change to library X and now your build fails. This seriously blows. It happens most often when you are forced to accept the latest changes from an upstream build. This is called a "push" method. An alternative is to use the "pull" method, which is where you choose whether or not you want to accept a new release from an upstream build – i.e. you can choose to stick with the existing version that you know works with your project.
The truth is, neither system is perfect, but what would be nice is if the build system had the flexibility to be either push or pull.
The Solutions!
Fear not, for Tom has come up with some thoroughly decent solutions to some of these anti-patterns!
Project Teams Should Own Their Builds
Don't have a separated build team – devolve the build responsibilities to the project team, share the knowledge and share the problems! Basically just buy into the whole agile idea of getting the expertise within the project team.
Project teams should involve the infrastructure team as early as possible in the project, and again, infrastructure responsibilities should be devolved to the project team as much as possible.
Have CI Experts
Have a small number of CI experts, then use them wisely! Have a program of pairing or secondment. Pair the experts with the developers, or have a rotational system of secondment where a developer or two are seconded into the build team for a couple of months. Meanwhile, the CI experts should be encouraged to go out and get a thoroughly rounded idea of current CI practices by getting involved in the wider CI community and attending meetups… like this one!
Personal Best Metrics
The trouble with targets, metrics and goals is that they can create an environment where it's hard to take risks, for fear of missing your target. And without risks there's less reward. Innovations come from taking the odd risk and not being afraid to try something different.
It's also almost impossible to come up with "proper" metrics for CI. There are no standard rules, builds can't all be under 10 minutes, projects are simply too diverse and different. So if you need to have metrics and targets, make them pertinent, or personal for each project.
Treat Your Build Environments Like They Are Production
Don't hand crank your build environments. Sorry, I should have started with "You wouldn't hand crank your production environments would you??" but of course, I know the answer to that isn't always "no". But let's just take it as read that if you have a large production estate, to do anything other than automate the provision of new infrastructure would be very bad indeed. Tom suggests using the likes of Puppet and Chef, and here at Caplin we're using VMWare which works pretty well for us. The point is, extend this same degree of infrastructure automation to your build and CI environments as well, make it easy to create new CI servers. And automate the configuration management while you're at it!
Provide a Toolbox, Not a Rigid Framework
Flexibility is the name of the game here. The project teams have far more flexibility if you, as a build team, are able to offer a selection of technologies, processes and tricks, to help them create their own build system, rather than force a rigid framework on them which may not be ideal for each project. Wouldn't it be nice, from a build team perspective, if you could allow the project teams to pick and choose whichever build language they wanted, without worrying that it'll cause a nightmare for you? It would be great if you could support builds written in Maven, Ant, Gradle and MSBuild without any problems. This is where a toolkit comes in. If you can provide a certain level of flexibility and make your system build-language agnostic, and devolve the ownership of the scripts to the project team, then things will get done much quicker.
Consumer-Driven Contracts
It would be nice if we could somehow give upstream builds a "contract", like a test API layer or something. Something that they must conform to, or make sure they don't break, before they expose their build to your project. This is a sort of push/pull compromise.
And that pretty much covers it for the content of Tom's talk. It was really well delivered, with good audience participation and the content was thought-provoking. I may have paraphrased him on the whole Jamie Oliver thing, but never mind that.
It was really interesting to hear someone so experienced in build management actually promote flexibility rather than standardisation. It's been my experience that until recently the general mantra has been "standardise and conform!". But the truth is that standardisation can very easily lead to inflexibility, and the cost is that projects take longer to get out of the door because we spend so much time compromising and conforming to a rigid process or framework.
Chatting to Christian Blunden a couple of months back about developer anarchy was about the first time I really thought that such a high degree of flexibility could actually be a good thing. It's really not easy to get to a place where you can support such flexibility, it requires a LOT of collaboration with the rest of the dev team, and I really believe that secondment and pairing is a great way to achieve that. Fortunately, that's something we do quite well at Caplin, which is pretty lucky because we're up to 6 build languages and 4 different C.I. systems already!
8 Principles of Continuous Delivery
Dave Farley co-authored "Continuous Delivery", an excellent book in the Martin Fowler signature series, which goes into great detail about the evolution of Continuous Integration, and how to achieve continuous delivery (or continuous deployment) using "build pipelines".
I went along to hear Dave Farley give a talk on Continuous Delivery and how they're doing it where he works, at LMAX. It was a really great session and he managed to cover, in quite a short session, a great deal of content from in the book. I've put together a highlight of what he covered in the talk, mixed with my own take on things
Here's what I learned…
Continuous delivery is basically the logical extension of Continuous Integration – it's a more holistic solution than C.I. though, as it encapsulates a lot more than just the development of software.
For instance, continuous delivery focuses a lot more on requirements than C.I. ever did, and involves a great deal more people on the delivery chain than traditional C.I. as well. It also has a greater customer focus than C.I.
Now, here's something I didn't know about continuous delivery…
There are 12 principles behind the agile manifesto. the first of which is:
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software
Well who'd have thought it? Continuous delivery was mentioned waaaaay back in the days of the agile manifesto, some 2500 years BC* and yet for most of us it seems like a pretty new idea.
Continuous delivery is based on the use of smart automation. This is all about creating a repeatable and reliable process for delivering software. You have to automate pretty much everything in order to be able to achieve continuous delivery. manual steps will get in the way or become a bottleneck. This goes for everything from requirements authoring to deploying to production.
The focus is on the finished article – again, this is described as being:
Working software in the hands of the user
Because the focus is on the software in the hands of the user, there's less tendency from a developers perspective, to simply chuck software over the wall to the QA team, and similarly to the Netops/production team.
Continuous delivery is all about getting that product out there, and getting the feedback from the users. This might mean delivering "unfinished" demo software during your development iterations, and getting your users to give valuable early feedback, or it might mean deploying experimental software to a website cluster and tracking how successful this new site is as compared to the existing system. Either way, it's all about feedback loops. Essentially you want to have as rapid a feedback loop with your users as possible.
Feedback loops are familiar to everyone who has worked on a Continuous Integration system. In C.I. feedback loops are generally about getting test feedback (unit test, acceptance test, performance test etc) as quickly as possible – "Fail Fast" – as you've probably heard.
Continuous Delivery, as described, takes this idea to it's logical conclusion, and gets the users involved in the feedback loop. This is a good example of how Continuous Delivery is more holistic than its C.I. predecessor. In Continuous Delivery, the feedback loop provides feedback not only on the quality of your code, but on the quality of your requirements, and the quality of your processes for delivering software.
8 Principles of Continuous Delivery
- The process for releasing/deploying software MUST be repeatable and reliable. This leads onto the 2nd principle…
- Automate everything! A manual deployment can never be described as repeatable and reliable (not if I'm doing it anyway!). You have to invest seriously in automating all the tasks you do repeatedly, and this tends to lead to reliability.
- If somethings difficult or painful, do it more often. On the surface this sounds silly, but basically what this means is that doing something painful, more often, will lead you to improve it, probably automate it, and this will eventually make it painless and easy. Take for example doing a deployment of a database schema. If this is tricky, you tend to not do it very often, you put it off, maybe you'll do 1 a month. Really what you should do is improve the process of doing the schema deployments, get good at it, and do it more often, like 1 a day if needed. If you're doing something every day, you're going to be a lot better at it than if you only do it once a month.
- Keep everything in source control – this sounds like a bit of a weird one in this day and age, I mean seriously, who doesn't keep everything in source control? Apparently quite a few people. Who knew?
- Done means "released". This implies ownership of a project right up until it's in the hands of the user, and working properly. There's none of this "I've checked in my code so it's done as far as I'm concerned". I have been fortunate enough to work with some software teams who eagerly made sure their code changes were working right up to the point when their changes were in production, and then monitored the live system to make sure their changes were successful. On the other hand I've worked with teams who though their responsibility ended when they checked their code in to the VCS.
- Build quality in! Take the time to invest in your quality metrics. A project with good, targeted quality metrics (we could be talking about unit test coverage, code styling, rules violations, complexity measurements – or preferably, all of the above) will invariably be better than one without, and easier to maintain in the long run.
- Everybody has responsibility for the release process. A program running on a developers laptop isn't going to make any money for the company. Similarly, a project with no plan for deployment will never get released, and again make no money. Companies make money by getting their products released to customers, therefore, this process should be in the interest of everybody. Developers should develop projects with a mind for how to deploy them. Project managers should plan projects with attention to deployment. Testers should test for deployment defects with as much care and attention as they do for code defects (and this should be automated and built into the deployment task itself).
- Improve continuously. Don't sit back and wait for your system to become out of date or impossible to maintain. Continuous improvement means your system will always be evolving and therefore easier to change when needs be.
To go with these principles there are also:
4 Practices of Continuous Delivery
- Build binaries only once. You'd be staggered by the number of times I've seen people recompile code between one environment and the next. Binaries should be compiled once and once only. The binary should then be stored someplace which is accessible only to your deployment mechanism, and your deployment mechanism should deploy this same binary to each successive environment…
- Use precisely the same mechanism to deploy to every environment. It sounds obvious, but I've genuinely seen cases where deployments to QA were automated, only for the production deployments to be manual. I've also seen cases where deployments to QA and production were both automated, but in 2 entirely different languages. This is obviously the work of mad people.
- Smoke test your deployment. Don't leave it to chance that your deployment was a roaring success, write a smoke test and include that in the deployment process. I also like to include a simple diagnostics test, all it does it check that everything is where it's meant to be – it compares a file list of what you expect to see in your deployment against what actually ends up on the server. I call it a diagnostics test because it's a good first port of call when there's a problem.
- If anything fails, stop the line! Throw it away and start the process again, don't patch, don't hack. If a problem arises, no matter where, discard the deployment (i.e. rollback), fix the issue properly, check it in to source control and repeat the deployment process. A lot of people comment that this is impossible, especially if you've got a tiny outage window to deploy things to your live system, or if you do your production changes are done in the middle of the night while nobody else is around to fix the issue properly. I would say that these arguments rather miss the point. Firstly, if you have only a tiny outage window, hacking your live system should be the last thing you want to do, because this will invalidate all your other environments unless you similarly hack all of them as well. Secondly, the next time you do a deployment, you may reintroduce the original issue. Thirdly, if you're doing your deployments in the middle of the night with nobody around to fix issues, then you're not paying enough attention to the 7th principle of Continuous Delivery – Everybody has responsibility for the release process. Unless you can't avoid it, I wouldn't recommend doing releases when there's the least amount of support available, it simply goes against common sense.
* Approximate date.
Continuous Delivery in Practice
A couple of months ago I was fortunate enough to be invited to the Thoughtworks live 2011 event in London. The main topics of this event were agile (as you'd expect from Thoughtworks) and Continuous Delivery. The event was run over 2 days, the first day included talks from Jez Humble and Martin Fowler, as well as a number of other speakers (Dave West did a session on agile vs the rest, Bjarte Bogsnes presented a very interesting session called "Beyond Budgeting", and Scott Durchslag was guest speaking about how agile has worked at Expedia, and his interest in Continuous Delivery). My colleague Steve Morgan was also in attendance on the first day and has produced this excellent post on the contents of day 1. I won't say much more about the first day because Steve's post covers it better than I ever could (he cheated by taking notes/paying attention). However, I will just add that it was great to see so many people who genuinely seemed passionate about the evolution of agile, and the coffee breaks were a great opportunity to pick the brains of Messrs Fowler and Humble. I particularly enjoyed the "Beyond Budgeting" talk from Bjarte, and am happy to say he's written a book about it, which you can buy on the internets!
The only other thing I would say about day 1 was that there was a reasonably good choice of biscuits. Biscuit variety is important and mustn't be underestimated.
Day 2
Day 2 was somewhat more hands on than the first day, and also more tools-oriented. It started off with another session from Jez Humble and Martin Fowler, where they discussed continuous delivery, covering some of the contents of Jez's book, and also going into detail about branching (feature branching vs Continuous Integration). The sessions then moved on to cover some of the products being produced at Thoughtworks Studios. There were talks from Andy Kemp and Suzie Prince about their products Mingle and Twist. Mingle is marketed as an agile project management tool. I've not used it myself but it seems to be based around shared workspaces and good requirements tracking. As you'd expect, it seems to integrate well with the other Thoughtworks products, which is probably one of its main advantages. Twist is a functional testing platform, which is very agile-centric in that it allows you to run requirements specs as tests in the "As a user, I want to…" style. Again it's the integration with the other tools and products that made it look good to me. I've used a lot of agile tools recently, which all seem to be great on their own, but putting them all together so that I can track a requirement through to a test and a change through to a build can be a lot of hard work. The final product that they presented was Go, which is their Continuous Integration system, and they even showed us how they used Go to deliver builds of Mingle, Twist and even Go itself (talk about eating your own dogfood).
Go ci implementation at caplin systems
The centrepiece of day 2 though was of course my very own talk on how we at Caplin Systems are using Go in our Continuous Delivery system🙂 I was invited to present a session on how Go is helping us achieve our Continuous Integration goals, and how we are using it to implement our own brand of Continuous Delivery. I've embedded my presentation above – sorry it's not a video. I won't go into too much detail (I'll save you the lecture), but basically we're using build pipelines along with a high degree of automated tests to deliver builds which are suitable for deployment. The other feature I have to mention is how Go manages multiple agents. We have about 60 active agents which Go manages. A build can be farmed out to any of these 60 agents, and if a particular resource is needed (let's say I need to run a test on a particular OS, like Centos) then Go can be configured to send the builds to the right agents. Particular builds can be configured to be excluded from certain agents, meaning the system is highly configurable.
In the afternoon we had an open-space session, during which I spent most of my time with a group discussing database deployments and how to bring db changes under Continuous Integration. It seems there are still a lot of people around who feel that this aspect is largely overlooked with respect to Continuous Integration, and this was reflected in the way people were talking about using manual db compare tools as a way of deploying db changes to production. My own feelings on this are that database changes should be treated more like code changes, with each change scripted and deployed as a code change would be. There needn't be destructive changes, and a good set of test data should help catch the issues that are often only found in production.
Greasemonkey script for CI system
Here in Caplin Towers (it's not really called that) we've got a couple of projectors displaying the Continuous Integration builds up on the walls. It's pretty useful until you get to the point where you've got more projects than space on the wall. We got to that point a while ago, and have had to resort to only displaying the "most important" builds on the wall. Clearly this is not very cool, because all the builds are important.
So, I decided to try selenium. In theory this should have worked fine, and indeed it would have if I could be bothered to spend a bit more time on it. My plan was to record a journey which loaded up each view, one after another, and then play back this journey using selenium RC so that I could put it into a scheduled cron job and have it run over and over again. Like I said, in theory it works fine, but in practice it wasn't such a great idea afterall. Firstly, there's always that delay as selenium initializes and loads the browser, then there's the presence of the selenium window, and then there's the problem of having to update the script every time a new build group is added. I know most of these issues can be overcome fairly easy, especially if you're selenium savvy or if you have a java framework for laoding and running selenium tests in place. I was just about to go down the route of writing my journey in java (mainly so that I can manipulate the window sizes more easily), when my colleague Edmund Dipple, said "I saw you struggling, so I've knocked this up" and showed me a greasemonkey script which does exactly what I was looking for.🙂
Basically the script runs through each pipeline group, one after the other, and pauses for 5 seconds on each one before moving on. Perfect. He used the chrome developer tools (or you could use Firebug on Firefox) to find out the name of the pipeline group container (which turned out to be "pipeline_groups_container") and then iterate through each of the child elements (the child elements represent each pipeline group). The full script is here:
var timeout = 5000;
var counter = 0;
var groups = document.getElementById("pipeline_groups_container").children;
var groupsLength = groups.length;
function scroll()
groups[i].style.display = "none";
groups[counter].style.display = "block";
if(counter == groupsLength)
counter = 0;
And now we see each build group on screen, one at a time:
What is in a name? Usually a version number, actually.
Another fascinating topic for you – build versioning! Ok, fun it might not be, but it is important and mostly unavoidable. In an earlier blog I outlined a build versioning strategy I was proposing to use with our Java builds. Since then, the requirements have changed, as they tend to, and so I've had to change the versioning convention.
Essentially, what I'm after is a way of using artifact version numbers to tell me some useful at-a-glance information about the artifact I have created. Also, customers want the version number to meet their expectations – that is, when they get a new build, they want to see an easily identifiable difference in the version number between the new build and their old one. What they don't want is a long complicated list of numbers which are hard to distinguish. For instance, it's much easy to identify which of the following 2 versions is the latest:
- 5.0.1
- 5.0.4
but it's not so easy to work out which of these is the latest:
- 22.214.171.12473
- 184.108.40.20653
As we're practicing continuous delivery, any given check-in can feasibly produce a release build. So, I would like some way of identifying exactly which check in produced my builds, or at least have a way of working out which bits of source code went into my released package. There are a couple of ways we can do this:
Tag the source code – We could make the builds tag the source code in our SCM system (Perforce) with every build. This is relatively easy to do using Ant and Maven. With Ant there are numerous different ways of doing it depending on your SCM system, for instance, with subversion you need to use the SvnAnt tasks from subclipse (http://subclipse.tigris.org/svnant/svn.html) and basically perform a copy of your source url:
<copy srcUrl="${src.url}" destUrl="${dest.url}" message="${version.num}"/>
(this is because tags in svn are just cheap copies with a label).
With Maven you just need to use the release plugin – this automatically handles tagging for you.
Tagging the source code is great – it keeps the version numbers as simple as I'd like, and it's nicely traceable. However, it's time consuming, and can result in a lot of tags. The other problem is, I can't tell which check-in caused the build just by looking at the version number of an artifact.
Use the commit number in the build version – We use a build version of Major.Release.Patch-Build in our artifacts. The build number used to be an auto-incrementing number – this worked fine but it didn't give us a link back to which commit had caused the build to be made. So, I decided to use the perforce changelist id (i.e. the commit version) as the build number in the version, so that builds would end up looking something like this: 1.0.0-11531.
The problem here is that the version number is not customer friendly – so I remove the build number as a final step, before the builds get released to customers. To track what version the customers have got, I still keep a record of the full build number (including the commit number) in the release notes, and I could also easily inject it into an assembly info or properties/config file if I so wished, so that customers could very easily read out the full version number just by looking in a menu somewhere.
There were several obstacles I had to overcome to get this working. The first obstacle, and really this was the one that stopped me from tagging the source code, was that the maven release plugin is abysmal when it comes to continuous delivery. I needed to use the release plugin to tag the source code, but one of the other things that the maven release plugin does is to remove the word SNAPSHOT, increment the version number, and check the pom back into source control. This would cause another build to trigger in the CI system, which in turn would increment the build number etc and cause another build to trigger – so on and so on. Basically it would create a continually building project.
So I have decided not to use the maven release plugin at all – it doesn't seem to fit in with Continuous Delivery. In order to create potential release candidates with every successful build, I've removed the word SNAPSHOT from all the poms, so we aren't making any snapshot builds anymore either (except when you build locally – more on that later). The version in the poms now takes the P4 commit number, which is injected via the Continuous Integration system, which in my case is Go. Jenkins also supports this, using the subversion plugin (if you use subversion), which sets an environment variable with the svn revision number (more details here). The Jenkins Perforce plugin does the same thing, setting the P4_CHANGELIST environment variable – so it can easily be consumed (more details here).
Go takes the P4 changelist number and puts it in an environment variable called "GO_PIPELINE_LABEL". I read this variable in, and assign it to a property called p4.revision. I do this in the command that kicks off the build, so that it overwrites a default value which I can keep in my pom – this is useful because it means my colleagues and I don't have to make any changes to the pom if we want to run a build locally (bear in mind if we run it locally this environment variable won't exist on our PCs, so the build would otherwise fail). Here's a basic run down of a sample pom, with more details to follow:
<description>Description about this application</description>
The value for p4.revision is "SNAP" by default, meaning that if I make a local build, I'll get an artifact with the version 5.0.2-SNAP. I know that these builds should never be promoted to production or handed to customers because the word SNAP gives it away. However, when a build is created by the CI system, the following command is passed:
clean deploy sonar:sonar -Dp4.revision=${env.GO_PIPELINE_LABEL}
This overwrites the value for p4.revision, passing in the Perforce commit number, and the build will create something like 5.0.2-1234 (where 1234 is my imaginary p4 commit number).
I've added a property called main.version, which is the same as the full version but without the build number. I've done this so that I can package up my customer builds (ina zip) and label them with the version 5.0.2. After all, customers don't care about the build number.
An important policy to follow is once a build is released to a customer, one of the other version numbers MUST be increased, meaning all further builds will be at least 5.0.3. The decision of which version number to increase depends on various business factors – I like to increase the 3rd number if I'm releasing a patch to a previously released build. If I'm releasing new functionality I increase the second number. The first number gets increased for major releases. The whole issue of version numbers becomes a lot less complicated if you're in the business of releasing software to web servers and you don't actually have to hand software over to customers. In this instance, I just keep the full version number with the build number at the end, as it's usually someone like me who has to look after the production system anyway!


text:
Temptalia Asks You
July 13, 2012
Weekly Feature
I'll open up the packaging by unscrewing caps, but I don't usually cut tube packaging myself. You can cut it across so you can see inside and get all the last bits out, though. With some bottles, depending on what it is, you can add a little water to get the rest off the sides (I do this with shower gels).

subset: commoncrawl


text:
Showing 881 - 900 of 1132
Spin-a-thon fundraiser to benefit professor with cancer
Two of biology professor Darren Proppe's Calvin colleagues have organized a fundraiser to help him with medical expenses and to support his family.
February 28, 2014 | Connor Sterchi
Spoelhof Fieldhouse Complex opens doors
On January 7, 2009, another milestone in Calvin athletics was marked as the men's and women's basketball teams returned to campus for their first games in the new Van Noord Arena.
January 13, 2009 | Phil de Haan
Spreading the word on Honest Tea from Washington, D.C.
Austin Graff '07 is the social media manager for Honest Tea in Washington, D.C.
July 20, 2016 | Jacquelyn Hubbard
Spring Arts Festival
The Visual Arts Guild hosts a party with music, art and food for the whole campus.
May 09, 2011 | Theo Voss
Spring Break Service Trips Set to Depart
From March 17 through 25, while students nationwide join the annual spring break exodus to the beach, 75 Calvin students with less leisurely agendas will be trekking to seven locales on the college's spring break service-learning trips.
March 10, 2006 | Myrna Anderson
Spring break trips build community
Calvin students are serving and learning everywhere from Kermit, West Virginia to Biloxi, Mississippi.
March 08, 2012 | Myrna Anderson
Stage managing from Calvin to NYC
Sharon Suh '16 is spending this summer interning for The Delecorte Theater in New York City.
August 24, 2016
Statement regarding Michael Peroutka's visit to Calvin College
The purpose of this statement is to correct some misinformation that's been floating around in regards to a speaker that talked to a small group of students at Calvin College in late April.
May 14, 2014 | Matt Kucinski
Stehouwer wins Professor of the Year
Scott Stehouwer, a 34-year veteran of the psychology department, is the students' choice.
April 30, 2012 | Myrna Anderson
Stock market game creates online traders
The student organization Calvin Business Forum has kicked off the stock market game for spring semester. The organization hosts the game for interested students each semester.
April 25, 2009 | Matt Decker
Story map project aims for ecological justice in Plaster Creek watershed
Students Annaka Scheeres and Charlotte Reynolds '16 teamed up with Professor Jason Van Horn to create a digital story of the watershed and its populations.
March 23, 2016 | Rachel Watson
Storytelling Guild: no reading allowed
Students gather to enjoy listening to, and telling stories
February 25, 2011 | Theo Voss
Streetfest leaders re-vitalize Camp Waltman Lake
Streetfest leaders were scrubbing, remodeling and staying a step ahead of the beaver at Camp Waltman Lake: Calvin's 319-acre retreat.
September 01, 2011 | Myrna Anderson
Streetfest photo album
Incoming students stock shelves, spread mulch and paint a playground during Streetfest.
September 06, 2011 | Calvin Staff
StreetFest Set for Two Wheels
The 2005 edition of StreetFest will see some 1,000 incoming Calvin College students spend a part of their orientation combining service with learning in a variety of Grand Rapids neighborhoods.
August 22, 2005 | Myrna Anderson
StreetFest's 20th anniversary
"It's an abstract idea until you get into the community. This is consistent with Calvin's mission of being a transformative presence in students' lives."
September 12, 2012 | Matt Kucinski
String Summit
Every year, Calvin hosts six days of rigorous instruction and performance for string performers of all ages.
September 05, 2012 | Amanda Greenhoe
String Summit kicks off
From August 19–24, high school and college students will participate in the fifth-annual Calvin String Summit, a six-day intensive camp for advanced string musicians. Calvin invites members of the community to enjoy two free events during the summit.
August 06, 2012 | Amanda Greenhoe
Student athletes give back
See how Calvin College student athletes go beyond the court to build stronger relationships with each other and with the Grand Rapids community.
June 26, 2009 | Matt Kucinscki
Student collaboration wins Best Student Film
An animated film created by Calvin media production, music and graphic art students was honored at the Chicago Film Festival.
November 09, 2012 | Myrna Anderson


text:
Price paid: $ 1259.04
Purchased from: ebay
Features: The guitar is confused as a dreadnought shape, however, in actual fact it is a "grand auditoriom" shape. It has 20 frets, neck width is 1 3/4 inches at the nut, and scale length is 25 1/2". The guitar has a solid spruce top. The tuners are high quality and chrome plated. The guitar came with a top of the range Taylor flight case which I'm guessing is worth atleast 100. The 414's back and sides are made of ovangkol, an exotic African hardwood said to produce a warm and deep sound. The neck is made of mahogany, and the headstock overly is made of rosewood. The top is solid spruce, and the fingerboard is ebony. // 10
Sound: This is one nice sounding guitar, especially at this price. It was warmer than most Taylors I've played, but not at the expense of the usual lively Taylor treble. The 414 is a very "liquid" sounding guitar, and it is also surprisingly loud. Although less punchy than the Taylor 310, I think it was a louder guitar overall. Sustain is also excellent in the Taylor 414. The only negative would be that the warm and enveloping bass could sometimes overwhelm the midrange and treble notes on delicate fingerpicking tunes. however sound is superb. // 9
Action, Fit & Finish: I purchased the guitar second hand so I'm not 100% sure if the action was altered at any stage. The action is perfect which makes playability effortless. Only flaw is the solid spruce top may be slightly more vulnerable to be marked although it is very attractive. The guitar has never seen a plectrum in it's life as myself, and the previous owner, play a finger picking style so this hasn't been a problem. // 10
Reliability & Durability: This guitar is very reliable and would easily withstand gigs. The brand, Taylor, make high quality guitars and this one is no different. Guitar has been gigged with before with the previous owner and appears to be in almost perfect condition with minimal damage (small nick on the body, barely noticeable). // 10
Overall Impression: This guitar is great for a range of styles although I prodominantly play bluegrass/country. The guitar sounds great amplified through my Marshall MG100DFX which isnt even an acoustic amplifier. I also purchased a Fishman GII EQ which fits the style of the guitar perfectly and increases the quality of the already excellent tone. If this guitar was stolen or lost I would most certainly go out and purchase a new one as it's simply the best guitar I have ever played. I have played many top of the range guitars but this one edges it. I would compare it to the Martin D-16 or the Larrivee D-04 simply because they are similar in price and great guitars. I beleive every guitarist should experince the warmth and depth this guitar produces, cheers. // 9
Price paid: $ 1999
Purchased from: GC
Features: Sitka spruce top, solid Ovangkol sides and back, ebony fretboard and bridge and pins, rosewood headstock overlay, mahogany neck, mother of pearl inlays, Taylor tuners. Comes with hardside case. Taylor Expression pickup system (volume, treble, bass). Fit and finish are really nice. I have found nowhere where I thought this axe could stand to have had a bit more attention to detail. I would like to have had a bit more latitude with the pickup system but it sounds pretty good run through a preamp. Set-up was exceptional right out of the box. // 9
Sound: I play bluegrass, country and gospel/Christian. I run mine through a tube preamp to add a bit of sparkle and then straight into the soundboard. Where this axe shines is acoustic sound though. While recording, I just mic it. Nice clear highs and lows with some nice mid undertones thrown into the mix. Volume is perfect across the neck. The top is incredibly responsive. I was playing mine under a ceiling fan and was getting a vibrato from the fan breeze. Not had that one happen before. The sound quality doesn't compare to high end acoustics like Naugler or Collings or Olsens, etc. But for $2K it is really nice. // 9
Action, Fit & Finish: Set-up was PERFECT out of the box. I wouldn't change a thing. With .012 gauge E string (Elixir Nanoweb) installed from the factory, it plays much like an acoustic with .010's. The inlays are perfect. Taylor uses CNC machines to make their inlays perfect. The top is a gloss finish which is nicely applied. The neck, back and sides are a satin finish which will be nice for warmer weather playing. Nicely chosen bearclaw sitka spruce top and well matched ovangkol back and sides. The mahogany neck is totally from the same piece of wood. // 9
Reliability & Durability: I've only owned mine for a few months - no issues. However, I know of two others in my area that are road tested and have stood the test of time well. They look a bit worn and slightly bruised but I am told that they have had no issue whatsoever with theirs. It appears to me that there is no place that will need attention in the near future. I will be taking the 414 on the road with me and feel fairly secure in not having to bring along a backup. The included hardside case is fairly snug and won't let the guitar do any flopping around inside of it. I guess the only down side I can think of would be the case MAYBE. I would like to see a little heavier latch system on the case but they seem fairly acceptable for now. // 10
Overall Impression: I've been playing professionally for 20+ years and this is my new BUDDY. I can leave the really expensive guitars at home and use the 414 in their place without sacrificing much in the way of sound quality. The 414 gives bluegrass a bit of a different type sound but it is a nice match in my opinion. I would use it for any type of music. I love how playable it is and the sound is nice. The vibration from the back can be intoxicating. I would most definitely buy another if something catastrophic happened to this one. I might look for one with the Expression2 system but the first type is acceptable. // 10
Price paid: $ 2
Purchased from: A local shop
Features: Year: 2014. Made in: Mexico. Has 22 frets. Sitka Spruce top. Ovangkol which is a beautiful Africa wood and it really makes a good high end and beautiful low in. Body style is Gradautitorum Cutaway. The home town store I bought it from sold it to me with there custom gig bag and the case that the guitar itself came with also came with there custom Strap and a Monster cable. They give you option 2 bags of any picks they have and 1 pair of extra elixir string also a warranty for 6 months where they will fix anything on it and replace it if they have too. // 10
Sound: The sound is quick and last at the same time really amazing on high end and pretty good on low end. It is a beautiful sound but being a cut away it's just not as full body as it could be. I play 214CE DLX and the which is also a cut away and I dare say It has a more vibrant tone then this 414CE. Never the less it is still very amazing guitar. It really shows every note while playing fingerstyle. Very vibrant matches perfect with a band and a singer. The guitar can make any sound you want some better then other though on the high end you can crank them out with ease but on the low end they a little bit harder to hammer out some of the notes you want. // 8
Action, Fit & Finish: The guitar is set up very well I had the action lower'd to fit my playing style. The fit isn't the best it kind of sits awkwardly. The only flaw I could see is a little dent in the finish but I'm sure that came after the fact in the store. Other wise the finish seemed to be hand rub and very thin which personally the finish doesn't change much for me as long as it plays well. The bride was a little loose also from apparently a small accident at the store but they fixed it no problem. The bracing is a little loose but still pretty sturdy from the bridge so they left that. // 6
Reliability & Durability: Now this is where this guitar takes the cake for me. I play in a church band and for the last 3 to 4 weeks I've played it as hard as I can and it has held up it comes with great elixir strings that are easy on the hands and just amazing tone. The electric side of this guitar is okay nothing crazy but very stout seems like it will survive a long time. There are 3 knobs on the top they seem a little flimsy but haven't given me any trouble they control high tone low tone and volume. You can diffidently trust this Guitar to play all night long gigs. Leave the backup at home. The finish seems thin but pretty strong and clean there is a dent in the finish from the store but the finish is still strong there just dented the finish not the wood so from that I would say it'll last quite a while. // 9
Overall Impression: I play country and southern gospel in which this is perfect fit for both. I have been playing for 4 years I own a Taylor 214ce a Segualhand crafted from Quebec. A Olympia Handmade no model #. Also a Fender Stratocaster and a Fender Telecaster. A LXK2-Mini from Martin, a Yamaha F335. Fisher acoustic amp and a Orange 20X amp. No the store I bought from really set me up good. If this was stolen I would go back to my 214CE because it was cheaper and plays just as well. I love the looks and sound I hate that I could have gotten same looks for about 1, 000 dollars less. I give it 6 overall since you could get the same thing for much much less. // 6


text:
"I think Bill Richardson has the life experience and work experience to be a truly great president," Mr. McIntosh said. "America could use a great president about now."
He said he will start working for Mr. Richardson on March 15.
Mr. McIntosh has been the state party's director for much of the past two years.
His departure comes as South Carolina's Democrats prepare for what is billed as the first Southern presidential debate April 26 and, a couple of days later, elect a new chairman at their state convention.
Mr. McIntosh was hired to lead the party's day-to-day operations in March 2005. He took an extended leave last June to run Robert Barber's unsuccessful campaign for lieutenant governor.
Mr. Richardson has been tied up getting his education agenda through his Legislature's short session this year, campaigning in Nevada - the nation's second caucus - and raising money in New York, California and New Mexico, lobbyist Crawford Cook said.
Mr. McIntosh will be replaced at the state party helm by Morton Brilliant, who helped run Democratic Gov. Jim Hodges' 1998 campaign.


text:
Veterans Mountain -- Not the Landmark You've Been Looking For
View Larger Map
A Sun City resident who wanted to honor the country's veterans persuaded the Arizona State Board on Geographic and Historic Names to slap a handle on a nameless Phoenix peak and (voila!) Veterans Mountain is born.
It's a sweet and sentimental idea, and Air Force veteran Lanny Brent deserves a certain amount of appreciation for thinking of it.
But this peak literally falls short of the kind of natural monument our veterans deserve. As far as the height of other hills in the Phoenix Mountains Preserve, Veterans Mountain isn't just second fiddle -- it's at least seventh.
Here's the summit elevation data for some of the highest peaks in the preserve:
Upcoming Events
Arizona Coyotes vs. San Jose Sharks
TicketsTue., Nov. 1, 7:00pm
Phoenix Suns vs. Portland Trail Blazers
TicketsWed., Nov. 2, 7:00pm
Arizona Coyotes vs. Nashville Predators
TicketsThu., Nov. 3, 7:00pm
Arizona State University Sun Devils Hockey vs. University of Michigan
TicketsFri., Nov. 4, 7:05pm
Camelback -- 2,704 Piestewa -- 2,608 Mummy -- 2,260 Shaw Butte - 2,149 North Mountain -- 2,104 Lookout Mtn. - 2,054
Veterans Mountain -- 1,920
Part of frustration over Governor Janet Napolitano's push to rename Squaw Peak as Piestewa Peak back in 2003 came from veterans who felt it wasn't right to name one of the Valley's most prominent landmarks after just one fallen soldier. Now veterans have their mountain, but the relative obscurity of the newly named peak makes the victory, well, stunted.
In the Google Maps image above, Veterans Mountain is just left of the marker labeled "D", just west of the Piestewa Peak Freeway (SR 51). We love desert mountains, yet we'd never thought of this one as anything more than part of the collection of hills in the preserve. One of its foothills even seems gravely wounded -- scarred by the earth removal necessary to build the freeway.
We're not sure if a well-defined trail goes to the top -- let us know if you've hiked it and how to best access the route, if there is one. Maybe this crag makes for a good workout, at least. -- Ray Stern
Get the This Week's Top Stories Newsletter
Every week we collect the latest news, music and arts stories — along with film and food reviews and the best things to do this week — so that you'll never miss Phoenix New Times' biggest stories.
In Case You Missed It
Upcoming Events
Phoenix Suns vs. Sacramento Kings
TicketsWed., Oct. 26, 7:00pm
Arizona State University Sun Devils Hockey vs. Harvard University Crimson Mens Hockey
TicketsFri., Oct. 28, 7:05pm
Arizona Coyotes vs. Colorado Avalanche
TicketsSat., Oct. 29, 6:00pm
Phoenix Suns vs. Golden State Warriors
TicketsSun., Oct. 30, 3:00pm
Popular Stories
Sponsor Content


text:
This TV Show was marked as 'Canceled/Ended', which means that is no longer available on TV. But don't worry - we have thousands of TV Shows ready for you here.
When will be Chef At Home next episode air date? Is Chef At Home renewed or cancelled? Where to countdown Chef At Home air dates? Is Chef At Home worth watching?
- TV Show Canceled/Ended -
There is no Next Episode of Chef At Home planned.


text:
"Your innovation is nothing until you find a way to monetize it," says Oleks Osiyevskyy, an assistant professor of entrepreneurship and innovation. "Unless you commercialize your invention you won't earn money nor will you change the world."
Bright IDEA: Student-run venture accelerator celebrates fifth anniversary
Members of Northeastern's thriving entrepreneurial ecosystem came together Friday night to commemorate the fifth anniversary of IDEA and celebrate the announcement of a gift endowing the student-run venture accelerator's CEO position.
Northeastern to make unprecedented $239 million investment in financial aid
Northeastern University will provide a record $239 million in financial aid—not loans—for undergraduate students in 2015–16. The unprecedented investment represents a 7.9 percent increase over last year, and continues a […]
Research, entrepreneurship at Northeastern celebrated at RISE:2015
Students and faculty across many disciplines presented their scholarly research, innovative thinking, and entrepreneurial endeavors at the annual expo, where many young researchers were honored with prestigious awards.
Entrepreneurial opportunities in eldercare
A panel of experts on Wednesday discussed how the growing population of adults over 65 and advancements in technology present opportunities for entrepreneurs to bring much-needed innovations in eldercare. Northeastern's […]
Phone charging station pilot program underway across campus
Five mobile phone charging stations developed by Ryan Wright, E'09, are now at key locations around campus for the Northeastern community to try out and provide feedback on.
President Aoun and inventor Robert Langer talk innovation
President Joseph E. Aoun on Wednesday afternoon sat down with Langer — inventor, entrepreneur, and engineer extraordinaire — to discuss Langer's awe-inspiring ability to harness the power of science and engineering to develop drug delivery systems and life-saving technologies.
Business student embraces the entrepreneurial spirit
Kate Murdock, DMSB'18, has quickly become one of the key members of Northeastern's thriving entrepreneurial ecosystem.
The startup matchmaker
Members of the university's entrepreneurship community convened Wednesday to formally launch the the Daniel J. McCarthy(s) Venture Mentoring Network, which is designed to match student, faculty, and alumni entrepreneurs with experienced alumni and other industry professionals.
Ask the entrepreneur: 3Qs with Chirag Kulkarni
Kulkarni, DMSB'18, accrued a wealth of entrepreneurial knowledge at a young age. Now he shares it with other entrepreneurs through his company C&M Group.
Ask the Global Officer: Matt Bilotti
Matt Bilotti, DMSB'15, leaves today for London, England—the site of his first stop on his six-month journey as President Aoun's Global Officer. Here, he shares what he's looking forward to, what he's anxious for, and the priceless advice he's already received.
Demo Day celebrates student innovation
After seven weeks of intense development and preparation, the young entrepreneurs behind 16 startups presented their ventures to the Northeastern community Monday night at the Husky Startup Challenge's Demo Day.


text:
Shiloh PA Businesses For Sale
Searched the MergerNetwork database for Shiloh PA Businesses For Sale and found no matches. Expanded the search to nearby areas and found the following matches.
Profitable Business Services With Loyal Clients #456127 posted Oct 19, 2016
- Forest Hill, Maryland, United States
- $42,000
Highly Confidential Sale - #1 Nationally recognized brand with as little as 20% down. Join the largest full-service company in this billion dollar recession proof industry. This is a goldmine in the industry with strong brand name recognition; a loyal, repeat customer base; and a dominant market position...
Harrisburg Dental Practice for Sale #455878 posted Oct 14, 2016
- Harrisburg, Pennsylvania, United States
- Available On Request
Beautiful state of the art general dentist practice for sale located a short distance from Harrisburg. 6 operatories in 3,000 sq ft with annual collections over $700K. Dr. wants to stay and continue working after the sale. Call or email for more information and exact location of practice.
Cash Flow Strong Civil Engineering Company #455873 posted Oct 14, 2016
- Baltimore, Maryland, United States
- $2,428,000
This well-known, full-service civil engineering company is located in the Baltimore, Maryland, metropolitan area. The firm enjoys significant operating margins and documented profitability in its regional client base. Its service suite includes high-value civil engineering, surveying, permitting, and...
Gift & Jewelry Store for Sale-$10,000 + Inventory #455464 posted Oct 7, 2016
- Ellicott City, Maryland, United States
- $10,000
Turnkey Opportunity-Business is Ready to Open. Located in the Historic District of Ellicott City. Some of the inventory includes collectibles and gift items such as paintings, unique jewelry, Greek antique replicas, onyx pieces, brass items, game boards and more!
Established GYN practice for 39 Years- Hanover PA #454612 posted Sep 25, 2016
- Hanover, Pennsylvania, United States
- $150,000
Established GYN practice for 39 years in Hanover Pennsylvania, 50 minutes from Baltimore. This practice has room for growth by adding obstetrics. This is state of the art practice with all the equipment in place. The seller will stay and help with transition period. Priced to sell at $150,000.
Baltimore, MD, Practice for Sale #451369 posted Aug 9, 2016
- Baltimore, Maryland, United States
- Available On Request
6-year acupuncture practice collecting $95,664.00. Owner take home pay before taxes $79,147.00. Open T/W/TH. Closed Friday through Monday for extended weekends. Located in heart of historic Mt Washington Village with a water view. Ideal location – across from light rail. Easy access to highway. High...
Almost Brand New Toxicology Lab Assets #450418 posted Jul 26, 2016
- Baltimore, Maryland, United States
- $177,000
Almost Brand New Toxicology Lab Assets at Steeply Discounted Price. Get Full Manufacturer's Warranty while Saving Money. Almost Brand New Assets of a Toxicology Lab are available at a steeply discounted price. AB SCIEX LC/MS/MS system was originally bought by a Baltimore pain management practice...
Solo Family Practice in Northern Baltimore #449894 posted Jul 19, 2016
- Baltimore, Maryland, United States
- $350,000
Well established solo family practice in beautiful semi-rural equestrian area of Northern Baltimore with average collections of $500,000 on a 3 ½ days/week work. This is an exceptional opportunity for a full time physician to take this practice to the next level by adding additional services or a second...
Quality HVAC Contracting and Service Business #449840 posted Jul 19, 2016
- Baltimore, Maryland, United States
- $2,000,000
The Company is a full-services heating, air conditioning (HVAC), mechanical contracting, and service company that has been serving residential and commercial customers in the Baltimore-Annapolis area for many years. This business offers a variety of service contract options to fit customers' needs...
Pharmacy for Sale #448878 posted Jul 6, 2016
- Baltimore, Maryland, United States
- Between $100,000 and $500,000
Retail Pharmacy. Average 24,000+ scripts per year. Gross Sales $1,900,000
Arnold & Bimbo Bread Route, Mifflin and Juniata #410319 posted Jun 24, 2016
- Harrisburg, Pennsylvania, United States
- $165,000
Arnold & Bimbo Bread Route distributorship for sale $165,000. Company financing with an estimated $41k down! Currently grossing an estimated $779,636 per year in sales and nets $90,552! Distributorship and "protected territory" are purchased through independent distributor and contracted through Bimbo...
Staffing Agency for Sale #409423 posted Jun 14, 2016
- Baltimore, Maryland, United States
- $425,000
Turn-key staffing company for sale with a very diverse customer base. IT, finance, office admin, education, healthcare and hospitality. Company has no loans, debt or other non-business operating expenses. Excellent gross margins, growth, expansion and payroll funding available. No additional upfront...
Mechanical HVAC, Plumbing Company for Sale #408158 posted May 28, 2016
- Baltimore, Maryland, United States
- Between $1,000,000 and $2,500,000
Mechanical Contractor provides mechanical services to government and commercial clients in the DC/MD/VA area. This Contractor is non-union. Type of business: Commercial (80% of sales), Government (20% of sales) Typical project: Light Commercial Buildings, Office fit outs and Retail sites.
Pizza Restaurant For Sale, Camp Hill #402663 posted Mar 15, 2016
- Harrisburg, Pennsylvania, United States
- $100,000
Pizza Restaurant for sale in Camp Hill, Pennsylvania for $100,000! Some seller financing available. Established business opportunity, currently grossing an estimated $402,700 and nets $112,530. Income based on an owner/operator role. Restaurant is operating with 9 employees that are willing to transfer...
Profitable Locksmith & Security Co.- Baltimore #388249 posted Aug 20, 2015
- Baltimore, Maryland, United States
- $189,000
This Highly successful B2B business has multiple revenue streams that create strong profit margins. The business operates in a small 350 sq. ft space which keeps overhead low and profits high. We take the business on the road to service all types of locksmith work from full insulation to lock-out services....
Chinese Food Restaurant with Real Estate #386779 posted Jul 30, 2015
- Baltimore, Maryland, United States
- $129,000
This is a profitable and underutilized business making it a prime opportunity for a owner/operator who wishes to expand the hours or change the food offerings. Current owner has been in business since 1990 and is open Monday-Thursday, 5:30pm-1am, Friday and Saturday, 5:30pm-3am, and closed Sunday. This...
Assisted Living Facility #243901 posted Jun 11, 2011
- Baltimore, Maryland, United States
- Between $100,000 and $500,000
Assisted Living Facility pre-approved 8-12 beds only $288,000. Near I95, 895, close to hospitals and nursing centers. Selling below market/for residential value only in order to pursue other business interest. If you are looking for an entry level profitable opportunity in Assisted Living that is...
Did you know that social networking is a great way to buy a business? Sign up now and we'll show you how.


text:
Tigers Homer Past Tribe
CLEVELAND (AP) – Pitching to J.D. Martinez didn't work out very well for the Cleveland Indians on Friday night.
Facing Victor Martinez wasn't a very good idea either.
J.D. Martinez hit a three-run homer, Victor Martinez added a two-run shot and the Detroit Tigers held on for a 6-4 victory.
Rick Porcello (9-4) pitched six scoreless innings for the Tigers and Joe Nathan worked the ninth for his 15th save.
But it was a big Martinez night for Detroit.
Victor Martinez homered off Corey Kluber (6-5) in the fourth inning. J.D. Martinez's home run, following an intentional walk to Victor Martinez, came in the eighth off John Axford.
J.D. Martinez was 3 for 4, extending his hitting streak to 10 games during which he's batting .447 (17 for 38) with four homers and raised his average to .317.
"You see occasionally players get on a hot streak like this and I hope he's hot tomorrow," Detroit manager Brad Ausmus said.
Victor Martinez, providing a big threat behind Miguel Cabrera, has 18 home runs and 47 RBI with a .328 average.
NEW YORK (AP) – Carlos Beltran hit a three-run homer with two outs in the ninth inning, lifting the New York Yankees over Baltimore for their fourth straight win.
Beltran connected off Zach Britton (3-1) right after Brian McCann hit an RBI single. The Yankees struggled with runners in scoring position until the ninth.
ST. PETERSBURG, Fla. (AP) – Jarred Cosart pitched eight shutout innings to beat Tampa Bay again, and rookie George Springer's two-run homer helped Houston to the win.
Cosart (7-5) gave up six hits in winning his third straight start. He walked one and struck out two, retiring 12 straight at one point.
Cosart topped the Rays last Saturday in Houston, the Astros' last victory before losing four in a row.
CINCINNATI (AP) – Edwin Encarnacion hit two three-run homers, and Toronto pulled off the second-biggest comeback in franchise history, rallying from an early eight-run deficit to beat Cincinnati.
Toronto hit four homers during its comeback, including the two by the major leagues' leader. Encarnacion started the comeback by connecting in the third inning off Mat Latos and finished it with his 23rd homer off Sam LeCure during a five-run ninth inning.
KANSAS CITY, Mo. (AP) – Brad Miller homered off Royals closer Greg Holland to break a ninth-inning tie, and the Seattle Mariners beat Kansas City despite blowing an early five-run lead.
Charlie Furbush (1-4) inherited a bases-loaded, one-out jam in the eighth, but the Mariners reliever calmly retired Mike Moustakas on a lazy popup to shallow right field. Furbush then struck out Alcides Escobar to silence a sellout crowd of 38,475 and keep the game tied 5-all.
MINNEAPOLIS (AP) – Brian Dozier's RBI-single with two outs in the bottom of the ninth inning lifted the Minnesota Twins to a victory.
Kurt Suzuki had three hits and an RBI and the Twins picked up closer Glen Perkins (3-0), who gave up two runs in the top of the ninth to blow his third save in 21 chances.
Daniel Webb (4-1) walked Eduardo Escobar and Sam Fuld with one out in the ninth and Escobar beat the throw home with a beautiful slide to win it. The play was confirmed upon review.
Gordon Beckham and Jose Abreu homered for the White Sox.


text:
The atmosphere will be neutral; so people of any belief system, or not at all, will feel welcome.
Why is it important to be able to identify logical fallacies? Portland State University Prof. Peter Boghossian had this to say:
"It matters so that you can differentiate truth from falsity. If you can't differentiate what's true from what's false, then you won't be able to create external conditions in your life that will allow you to flourish. It will also help you to identify what's in your own interest. By doing so, you'll be able to live the Good life."
This course is taught by Bernie Dehler, a Humanist Minister. Bernie participates in local Christian/atheist debates, and videos of past events can be seen here:


text:
Downtown Albany alterations shop wins award
In our Good News of the Day, a locally owned alterations shop is the Downtown Merchants Association Business of the Quarter.
Co-owner Gail Morrill says Gabe's holiday-themed "Winter Wonderland" window display and personalized customer service helped the store win the Business of the Quarter award. Morrill says Gabe's also tries to collaborate with other Downtown Albany businesses to help each other out.
"Well I was very surprised. I didn't even realize anything like that existed, and I was leaving our last meeting and they said 'Oh Gail we need you to stay for one more minute of business' and they presented us with that certificate. I was pleased like a little girl got an A on her report card," says Morrill.
She says her Broad Avenue store works a lot with brides and pageant participants but she says they will do alterations for anyone.
Morrill and Gwen Davis opened Gabe's in March of last year.


text:
Text Size: A | A
Lucas Curtin and Dominic Caputo kept Schalmont unbeaten in the Colonial Council with 14 and 13 points, respectively, as the Sabres beat Mechanicville, 49-45, on Tuesday night. Schalmont (3-0 Colonial Council, 4-1 overall) trailed last year's Class C state semifinalist, 26-13, at the half, but the Sabres outscored the Red Raiders, 21-8, in the third to pull even entering the final eight minutes. Nate Hatalsky poured in 20 points for Mechanicville (0-3, 3-3). Cohoes rallied ...
You Must Log-In or Subscribe to Continue
Individual stories can be found and purchased from our Archives for $2.00
Enjoy this story? Share it!


text:
‹‹ V.25 No.42 | October 20 - 26, 2016
See Alibi covers from
NEWSLETTERS Great Alibi stories, events and deals delivered to your inbox each week. No fooling!
Today's Events
La Festividad de los Muertos/Lanii Xtee Tugul at National Hispanic Cultural Center
Learn about the Zapotec community of Teotitlán del Valle, Oaxaca, Mexico, and their traditional celebration of Día de los Muertos.
Cerrillos Station Farmers Market at Cerrillos Station
I'm Broken • Pantera tribute, rock • Prince Of Darkness • Ozzy Osbourne tribute • Sons of Icarus • Iron Maiden tribute, rock • Full Blast! • Metallica tribute at LaunchpadMore Recommended Events ››
Salsa Dance Party!!! • 10.22.2016


text:
Paul & Shark SALE SALE SALE
As Official Paul & Shark stockists we have one of the largest selections of this brand on the North Wales coast and welcome customers to browse our shop. We stock most styles in sizes Medium, Large, XL and XXL. If you cannot find the style or size that you are looking for we will order it directly from Paul and Shark. Below are some the most popular styles from this seasons Paul & Shark collection. If you wish to order please phone us on 01745 853983 and we will help you with your order.
£110.00£115.00details Add to Cart
£275.00details Add to Cart
£245.00£255.00details Add to Cart
£245.00£255.00details Add to Cart
£285.00£285.00details Add to Cart
£195.00£210.00details Add to Cart
£90.00£140.00details Add to Cart
£195.00£215.00details Add to Cart
£185.00£195.00details Add to Cart
£185.00£195.00details Add to Cart
£70.00£125.00details Add to Cart
£70.00£115.00details Add to Cart


text:
Chess Merida
Sponsored Links
Chess Merida Ranking & Summary
RankingClick at the star to rank
Ranking Level
User Review: 8 (5 times)
File size: 27 KB
Platform: Windows All
License: Freeware
Price: FREE
Downloads: 772
Date added: 2008-10-07
Publisher: Armando Hernandez Marroquin
Chess Merida description
Chess Merida. Home & Shell & Desktop
Chess Merida Screenshot
Chess Merida Keywords
Bookmark Chess Merida
Chess Merida Copyright
WareSeeker.com do not provide cracks, serial numbers etc for Chess Merida. Any sharing links from rapidshare.com, yousendit.com or megaupload.com are also prohibited.
Featured Software
Want to place your software product here?
Please contact us for consideration.
Contact WareSeeker.com
Related Software
An accurate truetype font for your digital documents Free Download
This is a truetype font with a unique design Free Download
An original truetype font you can use on your documents Free Download
A truetype font with a special, pierced look to it Free Download
LCD Font Maker is able to make pixel font data for LCD Free Download
A truly original truetype font Free Download
An original truetype font with a strong appearance Free Download
FontRenamer renames truetype font files (*.ttf) to the real name of the typefaces Free Download
Latest Software
Popular Software
Favourite Software


text:
Subscribe to Our FREE Daily news alerts and get the latest updates on most happening events in the legal, business, and celebrity world. You also get your daily dose of humor and entertaiment!!
Enter your email address and click Subscribe for your free updates:
You can unsubscribe any time.
Newsletter Subscription
Enter your email address and start getting breaking law firm and legal news right now!
Famed litigator Beth Wilkinson left Paul, Weiss, Rifkind, Wharton & Garrison LLP to launch her own boutique with two other attorneys. The trio...
Summary: Law firm work is not meant for everyone no matter what kind of credentials you have, so figure out now if you should be pursuing another career. Read Three Simple Ways to Tell If You Should Work (or Continue […]
Directory of Patent Prosecution Law Firms in Indianapolis
More Legal Rankings and Charts