text:
How to Prioritize New Features vs Bug Fixes
The most lively debates that I regularly encounter leading an Engineering team revolves around the allocation of resources between bug fixing and the development of new features: "Why doesn't Engineering fix all the bugs?" exclaims a customer support person – "Why don't we allocate all Engineering resources to New_Shiny_Feature_X?" wonders the salesperson whose major deal depends on this feature.
These are both absolutely legitimate questions! … It does not mean that their answer is easy.
The main challenge in satisfying these two rightful requests is that they compete for the same resources, and that different people within the company have strongly-held different perspectives. The same person can even switch camps in a matter of days. It all depends on the last sales call. Do we have a customer threatening not to renew until we fix "their bugs", or do have a big deal pending on the delivery of a new feature?
As a consequence, it is imperative to create a business and technical framework that leads to decision making, where every stakeholder can not only express their perspective but also be satisfied about the decision process and thus about the decisions that come out of this process.
Framework for Decision Making
What's more important? Or more precisely, what's more important to implement in this release cycle?
- New features driven by product roadmap and corporate strategy
- Customer-driven enhancement requests
- Bug fixes requested by existing customers
- Paying down technical debt: upgrade architecture, refactor ugly code, optimize operational infrastructure, etc
The process to reach a decision is basically the same as for any business decision: we weigh how much income each item will generate and how much investment it will require.
Implementing a new feature, fixing a bug, enhancing a released feature or paying down technical debt demand the same activities: define requirements, design, code, test, deploy. They also all draw from the same pool of product managers, developers, QA and DevOps engineers. As a consequence, it is relatively easy to define the "investment" side of the equation.
Estimating the income side is a bit more complex, because it comes in multiple flavors. However, the process is the same as prioritizing the backlog of new features: we need to articulate the business case:
- Expected revenue stream (new features & enhancements)
- Reduction in subscription churn (enhancements & bug fixes, as well as new features)
- Cost reduction (technical debt / architecture) through increased future development velocity
- Customer satisfaction (bug & enhancements) which translates in better advocacy for the brand and churn reduction
- Strategic objectives (market positioning, competitive move, commitment to win a major deal)
Each of these categories is important in its own right. Since they cannot all be translated into a common unit of measure (e.g. dollars), I recommend quantifying each of these elements relative to one another (e.g. using T-shirt sizes: S, M, L, XL, …) for each item on the list.
Practically, I create a matrix with rows listing each feature, bug, enhancement request, technical debt, and the following columns:
- Short Description
- Link to longer description (Jira, Wiki, …)
- Summary business case
- Estimated engineering effort
- Estimated calendar duration
- Expected increase in revenue (if any)
- Expected cost reduction (if any)
- Customer satisfaction impact
- Strategic value
While this is not perfect – ideally we'd want to assign a single score for each item – this allows to (a) resolve the no-brainers (high-benefits at a low-cost or high-cost and low-benefits) (b) frame the discussion for the remainder against the business context of the company:
- Are we in a tight competitive race where we need to show momentum in our innovation?
- Do we have one, or more, major deals dependent on a given set of features?
- Are our customers grumbling about our product quality, or worse threatening to leave?
- Is our scalability at risk because of legacy code?
- Are we being hampered in our ability to deliver new features by too much legacy code?
While this will not eliminate passionate debates at Product Council, it will hopefully bound them, particularly if we can first agree on high-level priorities for the business.
Why Not Have a Dedicated Sustaining Engineering Team?
There are two primary reasons why a Sustaining Engineering team is a bad idea: first, it "does not answer the question" of prioritization, and secondly, it is a bad practice as it creates a class of "second-class citizens" engineers.
Say you want to have a Sustaining Engineering team. How large should it be? 5%, 10%, 20%, 50% of all engineering? Why? Should its size remain constant? Or are we allowed to shift resources in and out depending on business priorities? Answering these questions requires the same analysis and decision making as I propose above, but is burdened by the inflexibility of a split organizations
Regardless of whom you assign to Sustaining Engineering, these engineers will be considered second-class by the self-proclaimed hotshots who get to work on new features. Worse, it promotes laziness with respect to quality from the "new feature team": they know that Sustaining will clean whatever mess they leave. It is pervasive, and over time can even lead to cherry-picking of work, which means that Sustaining ends up completing the "new feature" work. For example, the "new feature" team releases a new product on Chrome (so that they can meet "their date"), but Sustaining gets to make it work on Internet Explorer.
A Useful Best Practice
Any bug older than 12 months, should be removed from the bug backlog. They should either be marked as "Won't Fix", or assigned to a secondary backlog list (which, I predict, will never be reviewed). The justification is simple: if a given bug has lived through a year's worth of bug triages without rising to the top and being fixed, then it is almost certain that it will never be prioritized for resolution. Better to put it out of its misery. Furthermore, this will keep the bug backlog to a reasonable size and bug triage a manageable task. Finally, if for some reason, the visibility of this bug raises anew, it can be returned to the active backlog.
The adage "Software always has bugs" remains true, not because it is impossible to write perfect software (I argue that this IS possible), but rather because in a business context, quality is not an end in-and-of-itself. Don't get me wrong high-quality is critical, but fixing ALL the bugs is not a requirement for business success.
As a consequence, only 1 criterion matters: "what moves the business forward the most effectively?"
Typically this means making customers happy. There are times when customers are happier if we fix bugs, at other times they prefer to see a new feature brought to market earlier. The answer depends on what drives their business. Do they prefer that we fix a bug that costs them an extra hour of work per day or that we launch a new feature that will allow them to grow their business by 10% in 6 months?
Notes from SF Data Mining Meetup: Recommendation Engines
Excellent talks on each of the presenting companies approach the design of their recommendation engines based on the specifics of their markets and users
Here are my notes on their respective technology stacks. Hadoop, Hive, Memcached, Java are used by all 3.
1. Trulia: Todd Holloway on Trulia Suggest.
- Hadoop
- Hive
- R on each Hadoop Server
- Memcached
- Java
2. Rich Relevance: John Jensen and Mike Sherman
- Hadoop
- Hive
- Pig
- Crunch
Starting to deploy
- Kafka
- Storm
3. Pandora: Eric Bieschke
- Python. Hadoop. Hive for Offline processing
- Memcached. Reddis: for near line & online
- Java & PostgreSQL for online
Memcached: Used as key-value store in the sky as long as you don't care about losing data
Reddis: "Persistent Memcached"
Scalable Software Architecture for a Startup
Say we are the founders of a startup and we just got a big fat check for our A-round funding. The VCs love our idea, and we all know that our app will attract millions of users in no time. This means that from day one we architect for millions of page-views per day…
But wait … do we really need to deploy Hadoop now? Do we need to design for geographical redundancy now? OR should we just build something that's going to take us through the next 3 months, so that we can focus our energy on customer development and fine-tuning our product features? …
This is a dilemma that most startups face.
Architecting for Scale
The main argument for architecting for scale from the get-go is akin to: "do it right the first time": we know that lots of users will be using our app, so we want to be ready when they come, and we certainly don't want the site going down just as our product catches fire.
In addition, for those of us who have been through the pain of a complete rewrite, a rewrite is something we want to avoid at all costs: it is a complex task that is fun under the right circumstances, but very painful under time pressure, e.g. when the current version of the product is breaking under load, and we risk turning away customers, potentially for ever.
On a more modest level, working on big complex problems keeps the engineering team motivated, and working on bleeding or leading edge technology makes it easier to attract talent.
Keeping It Simple
On the other hand, keeping the technology as simple as possible allows the engineering team to be responsive to the product team during the customer development phase. If you believe, as I do, one of Steve Blank's principles of customer development: "No Business Plan Survives First Contact with Customers", then you need to prepare for its corollary namely: "no initial product roadmap survives first contact with customers". Said differently, attempting to optimize the product for scale until the company has reached clear validation of its business assumptions, and product roadmap, is premature.
On the contrary, the most important qualities that are needed from the Engineering team in the early stages of the company are velocity and adaptability. Velocity, in order to reduce time-to-market, and adaptability, so that the team can rapidly adapt to feedback from "outside the building".
Spending time designing and implementing a scalable architecture is time that is Not spent responding to customer needs. Similarly, having built a complex system makes it more difficult to adapt to changes.
Worst of all, the investment in early optimization may be all for naught: as the product evolves with customer feedback, so do the scalability constraints.
Case Study: Cloudtalk
I lived through such an example at Cloudtalk. Cloudtalk is designed as a social communication platform with emphasis on voice. The first 2 products "Cloudtalk" and "Let's Talk" are mobile apps that implement various flavors of group messaging with voice (as well as text and other media). Predicint rapid success, Cloudtalk was designed around the highly scalable noSQL database Cassandra.
I came on board to launch "Just Sayin", another mobile app that runs on the same backend (very astute design). Just Sayin is targeted to celebrities and allows them to cross-post voice messages to Twitter and Facebook. One of my initial tasks coming on board was to scale the app, and it was suggested that we needed it to move it to Amazon Web Services so that we can scale rapidly as more celebrities (such as Ricky Gervais) adopt our product. However, a quick analysis revealed that unlike the first two products (Let's Talk and Cloudtalk), Just Sayin' impact on the database was relatively light, because communications were 1-to-many (e.g. Lady Gaga to her 10M fans). Rather, in order to scale, we first needed a Content Delivery Network (CDN) so that we could feed the millions of fans the messages from their celebrities with low response time.
Furthermore, while Cassandra is a great product, it was somewhat immature at the time (stability, management tools) and consequently slowed down our development. It also took us a long time to train new engineers.
While Cassandra will have been a good choice in the long run, we would have been better served in the formative stages of the company to use more established technology like mySQL. Our velocity in developing new features, and our ability to respond to changes in product strategy would have been significantly faster.
Architecting for Scale is a Process, not an Event
A startup needs to earn the right to design for scale, by first proving that it has found a legitimate market. During this first phase adaptability and velocity are its most important attributes.
This being said, we also need to anticipate that we will need to scale the system at some point. Here is how I like to approach the problem:
- First of all, scaling is an on-going process. Even if traffic increases dramatically over a short period of time, not all parts of the system need to be scaled at the same time. Yet, as usage increases, it is likely that any point in time, some part of the system will need to be scaled.
- In order to avoid complete rewrites of the system, we need to break it into independent components. This allows us to redesign each component independently, and have different teams work on different problems concurrently. As a consequence, good modularization of the system is much more important early on, than designing for scale
- Every release cycle needs to budget time and resources for redesign – including both modularization and scalability. This is just like maintenance on the Golden Gate bridge: the painters are always working; when they finish at one end, they start all over at the other end.
- We need to treat our software architecture the same way, and budget maintenance work every release cycle: dollars, time, people. CEOs have to be trained to not only think about the "shiny features" – those that are customer-facing – but also about the "continuous improvements" of the architecture that has to be factored in every release cycle.
- We also need to instrument the code to tell us were it is under strain. Unlike the Golden Gate bridge, we can't always see where it's breaking, or even rationalize it. Scaling sometimes works in mysterious ways that are not always obvious to predict.
In summary, designing for scale is a high-class problem, on which we only get to work once we have demonstrated true demand for our product. During this first phase, velocity and adaptability are critical, and are better served with well-understood technologies, and a well modularized design. Once our product reaches an adoption phase, then designing for scale is a continuous process that hopefully can be focused on individual modules in turn – guided by proper instrumentation of the code
QA does not stop in QA
Quality Assurance does not stop after the software receives the "thumbs up" from the QA team. QA must continue while the product is Live! … because QA is not perfect, and real users only exist on a Production system. We need to be humble and accept that our design, development and quality processes will not catch all the issues. Consequently, we must equip ourselves with tools that will allow us to catch these problems in Production as early as possible … rather than "wait for the phone to ring"
When the product exits QA, it simply means that we have we've run out of ideas on how to make the system fail. Unfortunately, this does not imply that the system, once in Production, will not fail. If we are successful and get a high volume of traffic, the simple law of large numbers guarantees that our users will find yet-never-thought-of ways to – unintentionally – make the system fail. These are part of the "unknown unknowns" as Mr. Donald Rumsfeld would say. Deploying the product on the production servers, and handing-off (abdicating?) the responsibility to keeping it up to the Ops team shows wishful thinking or naïveté, or both.
Why QA must continue in Production
There are a few categories of issues that one needs to anticipate in Production:
- Functional defects: in essence, bugs that neither developers, nor QA caught – while this is the obvious category that comes to mind, it is far from being the only source of issues
- User experience (UX) defects: Product works "as spec'd", but users either can't figure how to make the product work, or don't like it. A typical example is a high abandon rate in a purchasing experience, or any kind of work flow, or a feature that's never used, a button that's never clicked.
This is not reserved to new products, by improving the layout of a given page, we may have broken another feature on that same page
- Performance issues: while we may have run performance, and load tests, in our QA environments, the real world always offers surprises. Furthermore, if we are lucky enough to have the kind of traffic that Google or Facebook have, there is no other way but to test and fine-tune performance in production
Running tests on non-production systems requires to not only simulate the load of the system, but also to simulate the "weight" of existing data (e.g. in database, file system) as well as longevity to ensure that there is no resource leak (memory, threads, etc)
- Operational issues: while all cloud applications are typically clustered for high-availability, there are other sources of failure than equipment failure:
- External resources, such as partners, data feeds, can fail, or have bugs of their own, or simply not keep up their response time. Sometimes, the partner updates the API without notification.
- User-provided data can be mal-formed, or in an unexpected format, or a new data format can be introduced after the launch of the product
- System resources can be consumed at an unexpected rate. Databases are notorious for having non-linear response times based on load: as long as the load is under a given threshold response time is high, but once the load exceeds this threshold response time can deteriorate very rapidly.
A couple of examples:
- At my previous company, weeks after the product had been launched, we started receiving occasional complaints that some of the user-created videos were not showing up in their timeline. After (reluctantly) poking around in our log files, we did find out that about 10% of the videos that had been uploaded to our site for the past 2 weeks (but not earlier) were not processed properly. Our transcoder simply failed. Worse, it failed silently. The root cause was a minor modification to the video format introduced by Apple after our product was released. Since this failure was occurring for a small fraction of our users, and we had no "operational instrumentation" in our code, it took us a long time to even become aware of it.
- Recently, we launched a product that exchanges data with our partner. Their API is well documented, and we tested our product in their sandbox environment, as well as their production environment. However, after launch, we had reports of occasional failures. It turns out that users on our partner's site were modifying the data in ways that we did not expect, and causing the API to return error codes that we had never seen. Our code duly logged this problem each time it occurred in our log files … among the thousands of other log events generated every minute
Performing QA on Production Systems
As I mentioned, the Google and Facebook of the world, do a lot (if not most) of their QA on Production systems. Because they run hundreds of thousands of servers, they can use a small subset to run tests will live user data. This is clearly a fantastic option.
Similarly, "A/B comparisons" techniques are typically used in Marketing to compare 2 different user experiences, where the outcome (e.g. a purchase) can be measured. The same technique can be applied in testing, e.g. to validate that a fix of an intermittent bug difficult to reproduce does work.
More generally, Production code needs to be instrumented:
- To detect failures, or QoS (Quality of Service) degradations, with internal causes (e.g. database is slowing down)
- To detect failures, or QoS degradations, with external causes (e.g. partner API times out a lot)
- To monitor resource utilization for each service or application – at a finer grain than provided by Operations monitoring tools which are typically at the server level.
The point is that if a user can't buy a book on our website because our servers crash under load – this is a bug. While the crash is not due to code written incorrectly, it is due to the absence of code warning us that the system was running out of steam … this is still a bug.
In order to monitor quality in Production, we need to:
- Clean up the code that writes to log files: eliminate all logging used for code testing, or statements such as "the code should never reach here". Instead, write messages that will be meaningful to the poor soul who, a few weeks later, will be poring over megabytes of log files on a Sunday night trying to figure out why the system crashed
- Ensure that log messages have consistent severity levels (e.g. as recommended by RFC 5424 – Wikipedia has a nice table), so that meaningful alerts can be triggered
- Use a log aggregation system, like GrayLog2 (open source), so log files from multiple nodes in the same cluster, as well as nodes from different services can (a) be searched from a console and (b) viewed, time-aligned, on a single page (critical for troubleshooting). GrayLog2 can handle hundreds of millions of log events and terabytes of data.
- MEASURE: establish a base line for response time, resources consumption, errors – and trigger alerts when the metrics deviate from the baseline beyond a predetermined threshold
- Track that core functions – from a user perspective – complete, and log when, and ideally, why, they fail along with key parameters. E.g.: are users able to upload files to our system, are failures related to file size, time of day, location of user, etc?
- Log UX and operationally meaningful events to track how users actually use the system, what features are most used and track them over time. These metrics are critical for the Product Management team
- Monitor resource utilization and correlate with usage patterns. Quantify key usage parameters in order to scale the right resources in advance of the demand. For example, as traffic grows, the media server and the database servers may grow at the different rates.
- Integrate alarms from application errors into the Ops monitoring tools: e.g. too many "can't connect" errors should trigger an Ops alert that our partner is down – slow response time on a single server in a cluster may indicate the disk is failing
Quality is not a one-time event, it is an everyday activity, because users change their behaviors, partners change their APIs, systems get full and slow down. What used to work yesterday, may not work today, or no longer be good enough for our customers. As a consequence, the concept the "test driven" development must be extended to the Production systems, and our code must be instrumented to provide metrics that confirm that everything works as desired, and alerts when they don't. But that's not sufficient, developers and QA engineers must also take the time to look at the data, not just when a fire drill has been called, but also on a regular basis to understand how the system is being used, and how resources are consumed as the system scales, and apply this knowledge to subsequent releases.
Migrating a Self-hosted Architecture to the Cloud
While it may possible to migrate a self-hosted architecture to the cloud with servers in identical configuration, it almost certainly will lead to a sub-optimal architecture in terms of performance, and higher costs, in some cases prohibitively so.
The common objectives for moving to the cloud are:
- Ability to scale transparently as the business grows
- Reduce costs
- Benefit from a word-class IT infrastructure without having to hire the talent
We'll focus on the first two objectives as the third one is achieved – by nature – the moment you flip the switch to the cloud.
Memory Drives Pricing in the Cloud – not CPU
in the cloud, whether with Amazon EC2 or other vendors, the primary dimension driving pricing is the amount of memory (RAM) available in the server. In addition, CPU allocated is roughly proportional to the amount of RAM.
For example, as of this writing, per the Amazon EC2 pricing and the Amazon EC2 Instance Types definitions:
- A Small instance has (only) 1.7 GB of RAM – and 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit) and costs: $0.08 per hour On-Demand
- An Extra Large instance is 8 times bigger than a small instance and costs 8 times as much: 15 GB of RAM – and 8 EC2 Compute Unit and costs: $0.64 per hour On-Demand
- In order to get 32 GB of RAM, one needs to move to the High-Memory Double Extra Large Instance aka m2.2xlarge: $0.90 per hour On-Demand
Note that the prices quoted here are for US East (N. Virginia) zone. Prices for US West (Northern California) are more expensive (about 12% based on a few data points I correlated)
Database Servers
Database servers have unique requirements:
- They require fast I/O to disk. While Amazon recommends using networked storage, this is typically not practical from a performance perspective.
- So database servers also require large local disks: to hold the data
- Most databases require a fair amount of memory at least 16 GB. We use Cassandra, and they recommend 32 GB per server.
- They hate noisy neighbors (see previous blos). While virtualization technology does a fairly good job at partitioning CPU and RAM, it does a much poorer job at sharing I/O bandwidth. Having another virtual machine running on your database server can kill its performance, even if the neighbor does not do much, it can kill the I/O efficiency. All the tricks that databases use to optimize I/O performance assume that the database is in control of all I/O busses.
As a consequence, one should first of all use a reserved instance – simply because the cost of getting data in and out of the local disks makes it impossible to set-up / tear-down database servers "at will".
Secondly, one should buy a large enough instance (e.g. m2.4xlarge) so that we are the only tenant on the server. This will cost $7,203 per year – based on Heavy Utilization Reserved Instances pricing, and get us 68.4 GB memory, 4 cores (8 virtual – with Intel Hyper-Threading) and 1.69 TB of local storage.
As Adrian Cockcroft from Netflix illustrates in his detailed post: Benchmarking High Performance I/O with SSD for Cassandra on AWS, moving to SSD instances for I/O and compute intensive systems can bring significant cost reductions. In his example, he compares a traditional system with 36 x m2.xlarge + 48 x m2.4xlarge instances at a cost of $772,806 (Total 3 Year Heavy Use Cost) – with a 15 x hi1.4xlarge system at a cost of $354,405 – a 54% savings.
As the article illustrates, selecting one versus the other requires careful understanding of the computational profile of the application, and some changes in the application's architecture
Do I want to use Proprietary Amazon Solutions?
Following the logic that motivates us to move to the Cloud forces to consider using Amazon proprietary solutions: reduce need for sys admin talent, leverage out-of-the-box a scalable high-availability solution, etc
- Should I replace mySQL with RDS? Or Casssandra, HBase with DynamoDB?
- Should I replace my ActiveMQ (e.g.) message queue with SQS?
- … and similarly for AWS many products
These are excellent products, battle tested by Amazon. However, there are 2 very important considerations to examine:
- First, these products are obviously proprietary – making a move to another cloud provider like Rackspace or Joyent, will be take an extensive code rewrite. This may turn out to be impractical.
- Secondly, cost can be a (bad) surprise once the application is deployed live. For both RDS or SQS, pricing is driven by data bandwidth AND the number of operations performed using the service – which requires careful analysis to estimate ahead of time. For example, polling every 10 seconds to check whether new data is present in SQS generates 250K operations per month (assuming each check requires only 1 operation). This is fine if this function is performed by a few servers, but would break the bank if it's performed by 100,000 end-user clients. This adds up to $25,000 ($0.000001 per Request).
Algorithm Tuning and Server Selection
More generally, Amazon offers seven families of servers: Standard, Micro, High-Memory, High-CPU, Cluster Compute, Cluster GPU, and High I/O (SSD). Porting an existing application will thus require an iterative process evaluating the following questions:
- How do I best match each of my system's components with an Amazon instance types>
- Can I fine-tune, or even re-write, my algorithms to maximize RAM & CPU utilization? In particular, would I make the same memory vs computation trade-offs? Do I need this hash-table, or can I re-compute the query?
- How does my architecture evolve as I scale out? For example, do I need to replicate shared resources – like caches – or will sharding (e.g.) avoid this duplication of data – which will directly impact my cost since pricing is memory driven. An algorithm may work best using a approach favoring memory (and minimizing CPU) when running on a single server but it may be more cost-effective when optimized for memory when scaled out over many servers.
- How do new technologies like SSD impact my architecture? As the Netflix article illustrates, the cost impact can be radical, but it required substantial architecture redesign, not just a simple server replacement
In conclusion, moving from a hosted environment (where each server can be configured at will) to the cloud where servers come in pre-determined configurations requires not only an architecture review, but a sophisticated excel spreadsheet to compare the costs of various architectures. This upfront financial modeling is absolutely necessary in order to avoid unpleasant surprises as the business scales up.
Want to Predict your Cost in the Cloud? Roll Up Your Sleeves!
The selection of a cloud service provider is a critical decision for any a software service provider. Cost is, naturally, a key driver in this selection. However, predicting the cost of running servers in the cloud is a project in, and of, itself, because the only way to build a reliable model of costs, is to go ahead and deploy our systems with the service providers.
Why is not possible to forecast costs with pen and paper?
The main reason that pricing is so hard to forecast is that our system architecture in the cloud will likely be different from the one currently running in our own datacenter: the server configurations are different, the networking is different, and most likely we want to take advantage of the new features that come "for free" with a deployment in the cloud: higher availability, geographical redundancy, larger scale, etc. We'll cover this in details in an upcoming post.
Another reason why it is hard to predict costs is that we don't really know what we are getting:
When one considers the primary attributes of a server: RAM, CPU, storage, I/O (network bandwidth) – only RAM and storage capacity are guaranteed by cloud vendors. Vendors provide varying degrees of specificity about CPU and other key characteristics. Amazon defines EC2 Compute Units: "One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor". Rackspace's price sheet categorizes servers by available RAM and disk space (the more RAM, the more disk space). Their FAQ mentions the number of virtual cores each server receives, based on the amount of RAM allocated, but I could not find their definition of a virtual core. GoGrid, or Joyent provide similarly limited information.
As a side note, one needs to be aware that vendors typically refer to "virtual cores" – as opposed to real (physical) cores. A virtual core corresponds to one of the two hyperthreads that run on modern Intel processors since 2002. Conversely, a server with a quad-core Intel Xeon processor runs 8 virtual cores. You can read this 2009 post, plus the comments thread, for more specifics. While the data is dated, the observations are still relevant.
So, there is a lot that we don't know about the servers on which we will run our system: CPU clock, size of LI, L2 RAM, I/O bus speed, disk spindle rotation speed, network card bandwidth, etc.
Furthermore, performance will vary across servers (since cloud vendors have a diverse park of servers of different age) and thus, each time a new image is deployed, it will land on a random server, with the same nominal specs (RAM, storage), but unknown other physical characteristics (CPU clock, I/O bandwidth, etc).
Another well-documented problem is that of noisy neighbors. While the hypervisors do a fairly good job at controlling allocation of CPU and memory, they are not as effective at controlling the multitude of other factors that affect performance. I/O in particular is very sensitive to contention. While VMware affirms that vSphere solves this problem, most (all ?) cloud vendors use open source hypervisors.
In any event, this problem is systemic and cannot be solved by the hypervisor. For example, we did a lot of research on the best configuration for our Cassandra servers (database for big data). One of the main performance optimizations driving Cassandra's design is to maximize "append" (rather than update) operations, thus minimizing random movement of the read/write head of the disk, and thus maximizing disk I/O. Unfortunately, all this clever optimization goes out the window if we share the server – and thus the disks – with a noisy neighbor who is performing random read-write operations. I had the chance to discuss this a couple of months ago with members of the Cassandra team at Netflix (one of the largest users of Cassandra and almost 100% deployed on Amazon): they solve the issue by only using m2.4xlarge instances on AWS, which (today) ensures that they are the only tenant on the physical server – and don't have any noisy neighbor.
Adding all this together makes it pretty clear that vendor comparison on paper is practically fruitless.
Let's Try It Out
The only practical way to create a realistic budget forecast is to actually deploy systems on the selected cloud vendor(s) and "play" with them. Here are some areas to investigate and characterize, beyond simply validating functionality:
- Optimal server configuration for each server role (web, database, search, middle tier, cache, etc). We need to make sure that each server role is adequately served by one of the configurations offered by the vendor. For example, very few offer servers with more than 64 GB of RAM
- Performance at scale (since we only pay for the servers we rent, we can run full-scale performance tests for a few hours or days at relatively low cost – e.g. a few hundred dollars) – Netflix tested Cassandra performance, "Over a million writes per second", on AWS for less than $600 and clusters as large as 288 nodes
- End-to-end latency (measured from an end-user perspective) – since latency will be impacted by the physical distribution of the servers
- Pricing model
For these tests to be meaningful, one needs to ensure that deployments are realistic: for example, across service zones and regions, if we plan on leveraging these capabilities – as they impact not only performance (due to increased network latency) but also pricing (data transfer charges).
In addition, each test must be run several (10 – 20) times – with fresh deployments – at different times of day – in order to have a representative sample of servers and neighbors.
As important as the technical performance validation, the pricing model must be validated as vendors charge for a variety of services in addition to the lease of the servers: most notably bandwidth for data transfers (e.g. across regions), but also optional services (e.g. AWS Monitoring or Auto-Scaling), as well as per operation fees (e.g. Elastic Block Store). The "per operation" fees can add up to very large amounts, if one is not careful. For example, see the Amazon SimpleDB price calculator – we have to run SimpleDB under real load in order to figure out what numbers to plug in. Overlooking this step can be costly.
Once the technical tests have been completed, and the system configuration validated,
I recommend at least a full billing cycle of simulated operations, in order to obtain an actual bill from the vendor from which we can build our pricing model.
Deploying to the Cloud? Hang on to your Trousers!
My team and I have spent the past months investigating a deployment to the Cloud with vendors such as Amazon, Rackspace, GoGrid … to name a few who provide Infrastructure As A Service (IaaS).
A few conclusions have surfaced:
- One needs to clear about one's motivations to migrate to the Cloud- different motivations will lead to different outcomes, for a given product
- It is almost impossible to predict the cost of a cloud-hosted system – without deploying a test system with the selected vendor. As a corollary, precise comparison shopping is almost impossible.
- It is almost impossible to design, let alone deploy, your system architecture – without prior hands-on experimentation with your selected vendor. Also, the optimal architecture once deployed in the Cloud is likely to be radically different than one deployed on your own servers.
- Some Cloud vendors are moving aggressively up the value chain by offering innovative software technologies on top of their infrastructure. They are thus becoming PaaS (Platform As A Service) vendors. For example, as we commented in a previous post "Is Amazon After Oracle and Microsoft?" Amazon is deploying an array of software technologies – combined with services – that are tailored specifically for the Cloud, and are technically very advanced
We expand each of these points in upcoming posts, starting with the first one today.
The main arguments advanced in favor of a cloud infrastructure are:
- Offload the system management responsibilities to the Cloud services provider:
This is more than an economic trade-off: managing systems for high-volume Internet applications is a complex task requiring a broad set of technical skills – where said skills are in permanent evolution. Acquiring all these skills typically requires multiple engineers with varied backgrounds: computer hardware, operating systems, storage, networking, scripting, security, etc. These system administrators have been in high-demand for the past couple of years, demand high compensation, and usually want to work for companies which offer challenging work … namely those with a very large number of systems. As a result, some companies are simply unable to hire the necessary system administration talent in-house, and are forced to move to the Cloud for this single reason.
- Leverage best practices established by Cloud vendors.
Cloud services providers have optimized every aspect of running a datacenter. For example, Facebook released the Open Compute project in 2011 for Server and Data Center Technology. RackSpace launched the OpenStack initiative in late 2010, to standardize and share software for Compute (systems management, Storage, Media, Security, as well as Identity and Dashboard. Even managing systems at a hosting provider requires constant tuning of system management tools – whereas a Cloud service provider will take on this burden
- Benefit from the economies of scale that the Cloud vendors have created for themselves
Building data centers, finding cheap sources of power, buying and racking computers, creating high-bandwidth links to the Internet, etc. are all activities whose cost drops with volume. However, to me, the impact of price is much smaller than that of pure skills. The aforementioned tasks are becoming more and more complex, to the point where only the largest companies are capable of investing enough to keep up with the state-of-the-art.
In particular, Cloud vendors offer high-availability and recoverability "for free" – namely: free from a technical perspective, but not from a financial one.
- Ability to rapidly scale systems up or down according to load
This is one of the main theoretical benefits of the cloud. However, it requires a few architectural components to be in place:
(a) the software architecture has to be truly scalable and free of bottlenecks. For example, traditional N-tier architectures were advertised to be scalable because web servers could be added easily. Unfortunately, the database rapidly becomes the throttling component as the load rises. Scaling up traditional database sub-systems, while maintaining high-availability , is both difficult and expensive.
(b) Tools and algorithms are required to detect variations in load, and to provision/decommission the appropriate servers. This requires a good understanding of how each component of the system contributes to the performance of the whole system. The complexity increases when the performance of components does not behave linearly with load.
(c) Data repositories are slow and expensive to migrate. For example, doubling the size of a Cassandra (noSQL database) cluster is time consuming, uses a lot of bandwidth (for which the vendor may charge) and creates load on the nodes in the cluster.
- Ability to create/delete complete system instances (most useful to development and testing)
The Cloud definitely meets this promise for the front-end and business logic layers, but if an instance requires a large amount of data to be populated, you must either pay the time & cost at each deployment or keep the data tier up at all times. This being said, deploying complete instances in the Cloud is still a lot cheaper and faster than doing it in one's data center, assuming it can be done at all.
- The Cloud is cheaper:
This is a simple proposition, with a complex answer. As we'll examine in the next blog: figuring out pricing in the cloud is a lot more complex than adding the cost of servers.
Appreciating the business and technical drivers that motivate a migration to the Cloud will drive how we approach the next steps in the process: system architecture design, vendor selection, and pricing analysis. As always, different goals will lead to different outcomes.
Is Amazon After Oracle and Microsoft?
Amazon is quietly, slowly, but surely becoming a software vendor (in addition to being the largest etailer), with product offerings that compete directly, and in some cases, are broader than the "traditional" software vendors such as Oracle and Microsoft.
For example, a simple review of Amazon Products shows no less than 3 database options Amazon Relational Database Service (RDS), SimpleDB and DynamoDB (launched earlier this year), which offers almost infinite scale and reliability.
Amazon also offers an in-memory cache – ElastiCache. You can also use their SIMPLE services: Workflow Service (SWF) – e.g for business processes, Queue Service (SQS) – for asynchronous inter-process communications, a Notification Service (SNS) – for push notifications, as well as email (SES). Amazon calls them all "simple", yet a number of startups have been built and gone public or been acquired in the past couple decades on the basis of a single of these products: PointCast, Tibco, IronPort, just to name a few.
This is not all … Amazon offers additional services in other product categories: storage, of course, with S3 and EBS (Elastic Block Store), Web traffic monitoring, Identification management, load balancing, application containers, payment services (FPS), billing software (DevPay), backup software, content delivery network, MapReduce … my head spins trying to name all the companies whose business is to provide just a single one of these products.
Furthermore, Amazon is not just packaging mature technologies and slapping a "cloud" label on them. Some of them, like DynamoDB, are truly leading edge. Yet, what is most impressive, and where Amazon's offering is arguably superior to that of Oracle, Microsoft or the product category competitors, is that Amazon commits to supporting and deploying these products at "Internet scale" – namely as large as they are. This is not only a software "tour-de-force" but also an operational one – as anyone who has tried to run high-availability and high-throughput Oracle or SQL Server clusters can testify.
Given its breadth of products, its ability to operate them at Internet-scale with high-availability, Amazon could become the default software stack: a foundation on which to architect products, displacing the traditional stacks such as: .Net, LAMP, or {mySQL,Oracle}-Java-Apache-JavaScript
The costs of deploying software on the Amazon stack is another story … and the topic of a future post
Day-by-Day Model of an Iteration
This post presents a practical guide of what happens during a typical Agile iteration – a sort of play-by-play for each role in the team, day by day. Please open the attached spreadsheet which models the day-by-day activities of a 2-week Agile development iteration, and describes the main activities for each role during this 10-day cycle of work. In addition, we will highlight how to successfully string iterations together, without any dead time; as the success of any given iteration is driven by preparation that has to take place in earlier iterations.
This is intended as a guide, rather than a prescription. While each iteration will have its own pace – a successful release will follow a sequence not too different from the one presented here.
Golden Rules
Each company is different, each project is different, each team is different, each release is different, and each interpretation of Agile is different. The following states the immutable principles to which I personally adhere.
- Once Engineering and Product Owner agree on the deliverables of an iteration, they are frozen for this iteration
- Engineering must deliver on time
- Features cannot be changed, added, or re-prioritized
- Only exception is a "customer down" escalation of a day or more
- Engineering delivers "almost shippable" quality code at the end of the iteration
- Each release is self-contained: all the activities pertaining to a given user story must be completed within the iteration, or explicitly slated for another iteration at the start of the iteration
- E.g.: QA, unit tests, code reviews, design documentation, update to build & deployment tools, etc
- Dev & QA engineers scope their individual tasks at the beginning of each iteration. The scope and deliverables of the iteration are based on these estimates.
- Engineers are accountable to meet their own estimates
The above implies that Engineers must plan realistically by
(a) accounting for all activities that will need to take place for this iteration, and
(b) accounting for typical levels of interruptions and activities not specifically related to the project (scheduled meetings, questions from support, beer bashes, vacations, etc).
Estimates must be made with the expectation that we are all accountable to meeting them. This sounds like a truism, except that it is rarely applied in practice.
Day by Day
Before the Start of an Iteration
Preparation and planning prior to an iteration are critical to its success. As the spreadsheet highlights, the Product Manager spends the majority of his/her time during a given iteration planning the next iteration, by
(a) Prioritizing the tasks to be delivered in the next iteration
(b) Documenting the user stories to the level of detail required by developers
(c) Reviewing scope with Project Manager and Tech Lead
Pre-requisites at the Start of a Release
The following must be delivered to Engineering at the start of a release. The Product Owner, Project Lead and Tech lead are responsible for providing
- "A" list of user stories to be implemented during the release
- Detailed specs of the "A" list user stories
- Design of the "A" list features sufficient to derive the coding and QA tasks necessary to implement the features
- Estimated scope for each feature – rolling up to a target completion date for the iteration
These estimates are "budgetary". Final estimates are given by the individual engineers.
Day 1 – Kick-Off
The whole team gets together and kicks-off the iteration: the PM presents the "A" list features to Eng, and the Tech Lead presents the critical design elements. Tasks are assigned tentatively.
During the rest of the day, engineers review the specs of their individual tasks, with the assistance of PM and Tech Lead. This results in tasks entered into Jira, with associated scope and individual plans for the iteration.
The Project Lead combines all tasks into a project plan (using artifacts of his/her choice) to ensure that the sum of all activities adds up to a timely delivery of the iteration. The Project Lead also identifies any critical dependency, internal and external, that may impact the project.
A delivery date is computed from the individual estimates, and the team (lead by Product Owner, assisted by Project Manager and Tech Lead) iterates to adjust tasks and date
Day 2 – Deliverables are Finalized
Day 1 activities continue if necessary – resulting into a committed list of deliverables and a committed delivery date
The team, lead by Project Manager, also agrees on how the various tasks will be sequenced to optimize use of resources, and to front-load deliverables to QA as much as possible.
Developers start coding, QA engineers start writing test cases and/or writing automation tests
Day 6 – V1 Spec of the Next Iteration
By Day 6, the Product Manager provides the V1 Spec of the next iteration.
V1 Spec is a complete spec of all the user stories that the Product Owner estimates can be delivered in the next iteration. Typically, V1 will contain more than can be delivered, in order to provide flexibility in case some user stories are more complex than originally thought to implement.
During the remainder of the release, the Tech Lead (primarily) will work with the Product Owner to flesh out the details of the next release, to design the key components of the next release to a degree sufficient to be able to (a) list out the tasks required to implement the user stories, (b) estimate their scope, and (c) ensure that enough details has been provided for developers and QA engineers.
During the discussions of the next release, the Project Lead will identify any additional resources that will need to be procured, whether human or physical.
Day 7 – Release to QA
Release to QA means more than "feature complete". It means feature complete and tested to the best of the developers' knowledge and ability (see below).
Day 9 – Code Freeze
By Day 9, all bugs must have been fixed, so that the QA team can spend the last day of the iteration running full regression tests (ideally automated) and ensuring that all new features still work properly in the final build
By that time, the content and scope of the next release has been firmed up by Product Owner, Tech lead, and Project Manager, and task are tentative assigned to individual engineers.
Day 10 – Show & Tell
At the end of the last day of the iteration, Eng demos all the new features to the PM, the CEO and everyone in the company we can enroll.
We then celebrate.
Tools and Tips
Sequencing Iterations
- Depending on the complexity of the user stories, the Tech Lead (and other developers) may need to spend all of their time doing design, and may not be able to contribute any code.
- It is sometimes more productive to write automation tests once a given feature is stable. As a consequence, the QA team may adopt a cycle where they test manually during the current iteration and then automate the tests during the next iteration (once the code is stable)
- Exceptions to "almost shippable" are things like performance and stress testing, full browser compatibility testing, etc.
- These tasks are then planned in the context of the overall release, and allocated to specific iterations
Release Duration
The duration of a given iteration is at the discretion of the team. It is strongly recommended that iterations last between 2 and 4 weeks. It is also recommended that the duration of iteration be driven by its contents, in order to meet the Golden Rules. There is nothing wrong with a 12- or a 17-day iteration.
Start on Wednesday
Similarly, the starting day of the iteration is up to the team. Starting on a Wednesday offers several advantages:
- The iteration does not start on a Monday -). Mondays are often taken up by company & team meetings.
- Iteration finishes on a Tuesday rather than a Friday. Should the iteration slip by a day or two, it can be completed on Wednesday, or Thursday if need be. This means that the QA team is not always "stuck" having to work weekends in order to meet the deadline, nor do they have to scramble to make sure that developers are available during the weekend to fix their bugs, as would be the case if the iteration started on Monday
- By the second weekend of the iteration, the team will have good enough visibility into its progress, and determine whether work during the weekend will be required in order to meet the schedule.
The artifacts, format and level of details through which specs are delivered to Engineering is a matter of mutual agreement between Product Owner and Engineering, recognizing that Engineering is the consumer of the specs. As such, it is Engineering who determines the adequacy of the information provided (since Engineering cannot create a good product from incomplete specs).
Specs must be targeted for QA as well as Dev. In particular, they must be prescriptive enough so that validation tests can be derived from them. For example they may include UI mockups, flow charts, information flow diagrams, error handling behavior, platforms supported, performance and scaling requirements, as necessary.
Release to QA
While the QA team has the primary responsibility of executing the tests that will validate quality, developers own the quality of the software (since they are the ones writing the software). As a consequence, when developers release to QA, they must have tested their code to ensure that no bugs of Severity 1 or 2 will be found by QA (or customers) – unless they explicitly agree in advance with the QA team that certain categories of tests will be run by QA.
Regardless of who runs the tests, the "release to QA" milestone is only reached when enough code introspection and testing has been performed to warrant confidence that no Severity 1 or 2 bugs will be found.
Releasing to QA
Developers and QA can agree on how code will be released to QA. While the spreadsheet shows one Releate to QA milestone, this was done for clarity of presentation. In practice, it is recommended that developers release to QA as often as possible. Again, this should be driven by mutual agreement.
Furthermore, each developer must demonstrate to his/her QA colleague that the code works properly before the code is considered to be released. This demo is accompanied by a knowledge transfer session, where the developer highlights any known limitations in the code, areas that should be tested with particular scrutiny, etc.
Estimating Scope Accurately
One of the typical debates is whether time estimates should be measured in "ideal time" (no interruptions, distractions, meetings), or "actual time" (in order to account for the typical non-project-related activities). This is a matter of personal preference – what counts is that everyone in the team uses the same system.
I prefer to use "Ideal time": each engineer keeps 2 "books" within an iteration: the actual iteration work – scoped in "ideal time", and a "Other Activities" book, where all non-project-related activities are accounted for. This presents the advantages of (a) using a non-varying unit to measure the scope of tasks so that you can compare across people, project, time, and (b) having a means to track "non-productive time" on your project – and thus have data on which to drive decisions (e.g. pleading management for less meetings)
Software Specification is a Process Not a Document (2 of 2)
Timing is Everything
Product Management delivering complete specifications in a timely fashion greatly improves the productivity of the Engineering team (Complete being relative the type of specifications – as we discussed in the previous blog). The more precise the information provided at the start of each phase (scoping, release or iteration), the more efficient and accurate will the resulting development work be.
This sounds boringly obvious, but I have seen the contrary scenario over and over again, where business leaders grumble that the Engineering team is not productive, while failing to provide more than a PowerPoint level specification at the start of releases. As a consequence, developers spend the first third to half of the release working with the Product Managers to define the specs, instead of writing code – or even worse, developers start writing code without spec, and then having to do it over once the specs have been thought through.
Scoping is a 2-way Commitment
Another pitfall to avoid is "scope-creep". While the name itself would imply that it should be avoided at all costs (who wants to be creepy?), scope creep is an all-too-common occurrence
Scope creep, on the surface, appears to stem from good intentions (we want to meet every customer request – even last minute ones), yet it is one of the most demoralizing behaviors for the Engineering team – akin to continuously pushing back the finish line, after the start of a race.
In order to avoid scope creep, we (Engineering) need to remind the business team that based on the information provided during the scoping phase, Engineering reserved a set of resources for the duration of the release, and committed to deliver the feature set in the allotted time. This in turn creates an implicit contract that the scope of the release – will be bound by the amount of resources allocated to the release. While changes are expected as we get closer to the release start, and even once the release has started, the business team can't forget that there are only 24 hours in a day, and that no matter how cool it would be to add another 25% functionality, asking the Engineering team for such an increase in scope flies in the face of the process: If we could really do 25% more, we'd have said so the first time during the scoping phase.
In summary, once Engineering allocates resources for a release and commits to deliverables and schedules, the business team, in turn, must commit to keep the scope of the release commensurate to the resources allocated.
Use the Right Artifacts for the Job
As we replaced Waterfall development process with Agile Software development, we also replaced Market/Product Requirement Documents with User Stories. I have to admit that I don't get that part, or rather that I find that sometimes user stories are the best vehicle to express customer requirements, and other times, straight requirements do a better job.
For example, when a workflow needs to be implemented, nothing beats a flow chart or a state diagram to define it – we can dispense with the user story on the 3×5 card.
Write Things Down
There is no dispute that face to face discussions are the fastest way to nail down a user story. Often the expected behavior is self-evident from the software implementation itself. However, we must remember that multiple constituencies need to reach common understanding on the software's behavior: not only the Product Champion and developers, but also, QA, support, services, etc.
Again, there is no way that more than 2 people can reach the same understanding of how a workflow should perform, or what a report is meant to compute unless it is written down, preferably in pictorial form
Technical Risk Must Be Eliminated Prior to Scoping
The business team expects estimates that are fairly accurate – say within 10%. You can see eyes roll when you present your estimates and then add that the estimate is accurate within 30% … and it's a fair reaction. As a consequence, time must be invested in research, design and/or prototyping, in order to reach the desired level of accuracy. Sometimes, we need to invest the time to build a prototype in order to validate a design or an architecture. While this initially may appear to be a prohibitive price to pay, a much much higher price would be paid if one embarks on a release, only to miss the deadline by a month or more, because we found out that the original design was inadequate.
Managing Perceptions
Which scenario is best?:
(A) Promise to deliver 12 features and end up delivering 10 – OR –
(B) Promise to deliver 9 features and end up delivering 9
In my experience, Scenario (A) is a perceived failure, while (B) will be perceived as a success.
If you agree with me, then you will want to think hard about your iteration plan, and about what features you implement in which iteration. Naturally, the later the iteration within the release, the more likely it is that its features will not be implemented (either because of schedule slips, or changes in priorities). Consequently, plan low-impact features for the last release(s); this way you'll have to option of jettisoning them if necessary while still nailing the committed schedule. Conversely, if you high-impact features for the end, your only choices will be to disappoint — by taking them out in order to meet the schedule, or to disappoint — by forcing a schedule slip.
In conclusion, software development is a team activity – not only within the Engineering team but also with the business team: Engineering depends on the business team to create actionable specifications early enough before a release, to control the scope to a level commensurate to resources and time available, and to use artifacts that are relevant to the information to be conveyed.


text:
Franklin Graham: Obama/Warren Criticisms Ludicrous
Evangelist Franklin Graham knows a thing or two about getting flack for praying at an inauguration. He took heat after praying in Jesus's name at President Bush's inauguration in 2001. I just spoke with Franklin Graham, who gave his take on Obama's decision to ask megachurch pastor Rick Warren to give the invocation at his inauguration. We have also compiled our coverage of Warren over the years in a special section.
Here are some highlights from the interview:
[Obama] is including evangelicals at his inauguration, but I don't know if he'll include them in his administration. Time will tell. But Rick Warren will have Obama's ear on important issues.
Does Warren's acceptance of the invitation give an implicit nod to Obama's administration?
For anybody to be upset at Rick Warren for offering a prayer to almighty God, asking God to give wisdom and guidance to the Obama administration, is ludicrous.
Should Rick Warren pray in the name of Jesus at the inaugural?
I would hope he does because he's a minister of the gospel. There's no other way to pray. A Muslim should not be offended. [Warren] has no other way to pray than in the name of Christ. No one should be offended, because Rick Warren should be who Rick Warren is, and that's a minister of Jesus Christ.
I know you said a month ago that your father would not be serving as a spiritual adviser to Barack Obama.
He's 90 years old. He's just happy to get up in the morning.
Do you have any advice for Rick Warren?
My advice to Rick is to stay true to your convictions, and don't back up one step. I don't think he will. When you have the far left and the gay advocates mad at you, you must be doing something right.

subset: commoncrawl


text:
350 search results for Jim Carrey
Jim Carrey, Keanu Reeves teaming up with 'Girl Walks Home Alone at Night' director
Type: Post | Date: Friday, Mar 20, 2015Jason Momoa also starring in 'The Bad Batch'
Recap: 'Saturday Night Live' – Jim Carrey and Iggy Azalea
Type: Post | Date: Saturday, Oct 25, 2014The "Dumb And Dumber To" star returns for the third time
Jim Carrey tries to get spooky in his Halloween 'SNL' promos
Type: Post | Date: Wednesday, Oct 22, 2014Bill Murray tells Ellen his Christmas special still doesn't have a network
Jim Carrey and Jeff Daniels say only they can play their 'Dumb & Dumber' roles
Type: Post | Date: Monday, Nov 10, 2014And it sounds like they had fun torturing Kathleen Turner
Jim Carrey and Jeff Daniels say they're the only two who can play Harry and Lloyd
Type: Video | Date: Monday, Nov 3, 2014HitFix sat down with the legends themselves to chat about returning to the big screen after 20 years in 'Dumb and Dumber To'.
Jim Carrey, Kate McKinnon, Iggy Azalea dance-off to Sia's 'Chandelier' on 'SNL'
Type: Article | Date: Sunday, Oct 26, 2014Wigging out
Jim Carrey on finished 'Dumb And Dumber To': 'Very funny'
Type: Article | Date: Friday, Feb 28, 2014Why the actor's eating lead paint, and who his favorite musicians are right now
Jim Carrey spoofs Scarlett Johansson's 'Lucy' in new 'Dumb and Dumber To' posters
Type: Article | Date: Saturday, Aug 16, 2014Maybe Jeff Daniels should play the Joker next
'SNL' took aim at Matthew McConaughey's Lincoln car commercials
Type: Article | Date: Sunday, Oct 26, 2014Who do those kids belong to?
'Dumb and Dumber To' poster unites Jim Carrey, Jeff Daniels and the Shaggin' Wagon
Type: Article | Date: Tuesday, Jun 10, 2014Farrelly Brothers sequel


text:
16 year old runner killed in the Valley by hit and run driver.
[Update: I originally got this story wrong and wrote that Lynch was a cyclist. He was running across the street when this happened. As a runner, I know it's very dangerous out there for runners just as it is for cyclists. I learned the defensive riding skills that keep me alive riding in LA first […]Read more


text:
Design gurus seem to love radiator covers, usually wooden boxes with grilles at the front. To me they seem absurd, interfering with both radiative and convective heat transport. Can anybody tell me the effect these things will have on my heating bills, before my boyfriend gets any fitted?
• How do radiator covers affect heating bills? The answer is at best not at all and at worst it could actually increase them.
Your correspondent correctly notes that radiators work by both convection and radiation heat transfer into the room. If the existing radiators are boxed in then the heat transfer rates will be reduced.
Manufacturers of radiators provide heat output reduction factors if radiator shelves or boxed covers are fitted to their products.
When a heating installation is designed with radiator covers, the size of radiator selected to meet the heating requirement will necessarily be larger than a radiator that ...
To continue reading this article, subscribe to receive access to all of newscientist.com, including 20 years of archive content.


text:
By PATRICK CASEY
The Paxton Bobcats boys basketball team won three games in three days at Walton High School to claim the eight-team tournament championship. Paxton defeated Tuscaloosa 55-29 in the opening round and Pensacola Tate 82-66 in the second round to set up a showdown with the Walton Braves for the tourney title.
The showdown didn't last long as Paxton won the game 73-50. Paxton was on fire from the floor and hit 31-of-57 shots from the court while Walton could only connect on 21-of-67 shots during the game.
Paxton had four players in double figures as Demetrius Moore scored 19 points and Colby Forehand added 17. Shaq Jackson scored 14 points and Zack Vickers had 10. Vickers scored 30 points in the Bobcats semi-final win over Pensacola Tate as the Aggies sent Paxton to the free throw line 39 times in trying to rally from a 38-24 halftime deficit.
James Towery had 19 points against Paxton and put 38 points on the board in the season opener against Dothan Northside as the Braves won going away 83-48. Walton struggled in the second round as they fell behind Robertsdale in the first half, only to rally for a 64-54 second round victory as Tarrell Bramlet scored 31 points to lead Walton to the win.
Paxton (3-0) and Walton (2-1) will meet in the regular season on Friday, February 6, in Paxton and could see each other in the 16-team Freeport tournament in late December.
The meeting between Walton and Paxton's boys teams was only the third regular season match-up in over a decade. Paxton won last year 60-49 after Walton had defeated the Bobcats during the 2005-06 season 58-30. The winning team both of those seasons went on to play in the Region final, a good sign for the Bobcats, who lost at the buzzer to FAMU in last year's Class A Region final.


text:
Golf: West Briton round-up March 27
Posted: March 27, 2014|
CORNWALL'S men delivered a resounding victory to new county captain Richard Simmons in his first match in charge, thumping the Royal Navy 13-2 in the annual friendly fixture at Tehidy Park on Saturday.
Simmons, of Tehidy Park, was delighted with the team's performance.
He said: "The result was never in doubt, but a good run-out all the same.
"There was some great early-season golf on a course that has suffered over the last few months, but turned out in terrific condition – and will be quite stunning come County Week in May."
Tom Fox (China Fleet) was in electric form, leading out the side in both the foursomes and singles. The quality of his ball striking was acknowledged by the players he beat in the morning and afternoon.
His singles victim, a solid two-handicap player, was unable to live with the power and shot-making of his plus-two opponent.
After being demolished 7&6 by the former county champion, who rattled in a flurry of birdies that left him five under par when the match came to an early end after 12 holes, his shell-shocked opponent said: "It's just a different game altogether."
But Fox's singles victory was not the biggest of the day as Launceston's Richard Jasper registered an even more one-sided 8&7 win. There were other comfortable victories right through the order, including wins for former six times county champion Ian Veale and Rob MacGregor, both of Tehidy Park, and Perranporth pair Conor Wilson and Chris Fleming.
The only Cornish player to miss out was Steve Davis (Trevose), who battled hard from five down with six to play to take his match up the 18th, where he jammed the back of the hole with his birdie putt only to see the ball pop out.
Earlier, the hosts had won three and halved the other two morning foursomes with Fox and Adam Curtis (Bude and North Cornwall) setting the tone for the rest of the match with a fine 3&1 win as Cornwall led 4-1 at lunch.
Simmons' men will be looking for a repeat of this performance when they take on Gwent in their opening Channel League match at Perranporth on Sunday, April 20.
Stalwarts Joe Cruse (St Enodoc) and four-time Winter Golf Series champion Josh Greenaway (Bowood Park) will be available to return to the side for this all-important opening fixture.
Cornwall results. Foursomes – Tom Fox & Adam Curtis won 3&1; Ian Veale & Steve Davis halved; Harry Hall & Matt Richards won 6&5; Rob MacGregor & Richard Jasper won 6&5; Conor Wilson & Chris Fleming halved. Singles – Fox won 7&6; Veale won 4&3; Hall won 4&3; Davis lost 1 down; MacGregor won 3&2; Jasper won 8&7; Curtis won 3&2; Richards won 4&3; Wilson won 3&2; Fleming won 3&2. Overall match score: Cornwall 13, Royal Navy 2.
Truro's reigning English ladies strokeplay champion Sarah-Jane Boyd faces a crucial few weeks in her bid to be selected for this year's Curtis Cup team, which will be announced in May.
Her first engagement is the prestigious 36-hole Leveret Trophy at Formby in Lancashire on Saturday, April 5.
Depending on her result there, she may also play in another top one-day tournament over 36 holes, the Hampshire Rose, at North Hants on April 13.
She then moves north of the border to play in the Scottish Ladies Open Championship at Troon over 54 holes from April 25-27 before going on to Prestatyn for the Welsh Ladies Open on May 3/4.
The PGA South West season kicks off in earnest with the traditional and ever popular seasonal opener, the Cornish Festival, which is played over three of the county's top courses at Trevose, St Enodoc and Newquay from next Tuesday, April 1 to Thursday, April 3.
Entries for the festival have been received from four former Ryder Cup players - last year's winner Peter Baker, joint winner in 2012 Paul Broadhurst, Steven Richardson and Weshman Phillip Price.
Guy Norris, of Trevose, and Falmouth's Tristan Gurd overcame strong winds and rain to card 39 Stableford points and win the pairs prize at the latest meeting of the Cornwall County Professional Golfers'Alliance at Mullion last week.
They finished just a point ahead of Newquay professional Chris Gill and China Fleet amateur John Sanders on 38.
Gill also shared the individual professional's prize with Killiow's Richard Sadler after both shot three over par rounds for 33 points. The next Alliance meeting is at Falmouth on Thursday, April 10.
Full results. Pairs – 1, Guy Norris (Trevose) & Tristan Gurd (Falmouth) 39 pts; Chris Gill (Newquay) & John Sanders (China Fleet) 38; 3, Matt Richards (West Cornwall) & Marek Churcher (Falmouth) 36; 4, Paul Martin & Will Sumner (Carlyon Bay) 36; 5, Richard Sadler (Killiow) & Max Pocklington (Newquay). Individual professionals – 1=, Chris Gill, Richard Sadler 33 pts; 3, Grant Slater (Trevose) 31; 4, Paul Turpin (Carlyon Bay) 30; 5=, Paul Martin, Antony Nash (St Enodoc), Richard O'Hanlon (The Point at Polzeath), Dan Wood (Ivyleaf) 29.
Mother and son Debbie Sedgeman and Martin Harding, from West Cornwall, shot a fine round of 38 Stableford points to win the Cornwall Ladies' start-of-the-season Mixed Greensomes at a bright and breezy Newquay on Sunday.
They were closely followed by home members Sue and Charlie Phillips on 37, with Newquay ladies' captain Janet Cox and her husband Frank third on 35.
Results: 1, Debbie Sedgeman & Martin Harding (West Cornwall) 38 pts; 2, Sue & Charlie Phillips (Newquay) 37; 3, Janet & Frank Cox (Newquay) 35. First 9 holes: Helen Carson (Mullion) & Scott Willett (Praa Sands) 19. Second 9 holes: Liz & Robert Gillman (Perranporth) 19.


text:
Accra — President John Agyekum Kufuor, in his New Year message, has urged the two leading parties, the NPP and NDC, to yield to the authority of the EC to ensure the transitional process is kept on track. President Kufuor is expected to hand over to a new president on January 7.
He said any outstanding issues that remain unresolved may be settled later by due process. President Kufuor also cautions the media to exercise circumspection in its reportage of the election process.
He also expressed confidence that the security forces will remain neutral throughout the entire process. President Kufour concludes by calling on Ghanaians to remain calm until the results are declare by the EC.
Below is the entire statement from the Castle:
The year 2009 has dawned on us and it is my greatest pleasure to wish the entire nation a very happy New Year.
For many Ghanaians the joy of the New Year has been tampered to an extent by the intense anxiety and high tension arising out of the ongoing Presidential elections which is yet to throw up a clear winner, despite last week's run-off.
The two contesting parties have raised pertinent concerns which are being addressed by the Independent Electoral Commission, among others.
It is important however that we meet the Constitutional time table for handing over on the January 7th. I therefore urge all stakeholders to yield to the authority of the Electoral Commissioner when he declares the results. Any outstanding issues may be settled later by due process.
I should also appeal to the media also to exercise the greatest sense of responsibility and circumspection in discharging their functions.
They must not inflame passions or incite people to undertake negative actions. Rather, their work should be guided by the public good at all times.
On the part of the security forces, I know that the nation can count on them to remain neutral and to display the highest sense of professionalism in the conduct of their work at this very critical time in the country's history.
Lastly, I entreat the general public to remain calm until the result of the election is declared by the Electoral Commission to facilitate the transfer of power from this government to the next on 7th January as mandated by the nation all Constitution.
May God bless us all.
Signed: John Agyekum Kufuor
President of Republic of Ghana
2nd January, 2009


text:
10% Off $99+ / 15% Off $169+ / 20% Off + Free Shipping on $249+ | Promo Code DAILYDEAL | Expires 7/2/15
Shipping offer applies to U.S. customers only
Wedding Decorations and Wedding Toasting Flutes
Wedding reception decorations should be as beautiful and special as the rest of your big day, as well as affordable. Our selection of wedding reception decorations is filled with discount wedding accessories ranging from wedding glasses and wedding guest books to wedding reception favors, wedding napkins and more. Why pay extra for wedding reception decorations when you've got Ann's Bridal Bargains by your side? Choose us for your wedding and see for yourself how perfect affordable can be!


text:
Similarly, clandestine foreign operations have sullied the civilian courts.
Much has already been written about the recklessness inherent in any director of the CIA entering into a clandestine relationship.
The government sought to prove the existence of a plot through a campaign of undercover surveillance and clandestine recordings.
1560s, from Latin clandestinus "secret, hidden," from clam "secretly," from adverbial derivative of base of celare "to hide" (see cell), perhaps on model of intestinus "internal." Related: Clandestinely. As a noun form, there is awkward clandestinity (clandestineness apparently being a dictionary word).


text:
A skilled lawyer has the ability to consider the big picture while maintaining intense focus on the smallest of details. A practical, reasoned and cost-effective approach to legal services lies at the core our firm's continued success. At Thomas Law Firm PC we never take this for granted and we always work hard for you.
In litigation, the firm has handled and continues to handle a wide variety of legal disputes in many areas of law. We have extensive experience in taking on complicated and difficult disputes and do not hesitate to stand and fight to the end to strive for the very best outcome for the clients.
Today's legal consumers want not just results, but value as well. Thomas Law Firm PC applies human and business resources in a way that enables the client to solve the problems at hand in the most cost-effective manner. The firm's cost and fee structure was constructed with this objective in mind. We know that a divorce, injury or legal dispute can often prove emotionally and financially devastating. That is why the firm will advance costs in appropriate cases in order to ensure that its clients are not denied their day in court.
Thomas Law Firm, P.C. is pleased to announce that after years of providing services to Allen County residents, we have established an office near Downtown Fort Wayne. Our Fort Wayne office provides legal assistance in the same areas of law as our Auburn practice, however, all scheduling and billing will be run through our Auburn location.Contact our Auburn office at 260-925-1703 if you wish to schedule appointments at our Fort Wayne location.


text:
Investigators: Hospital 'deficiencies' overlooked
Medicare reviewers missed two-thirds of safety issues, GAO says
Tuesday, July 20, 2004 Posted: 1:05 PM EDT (1705 GMT)
WASHINGTON (AP) -- The private organization that clears hospitals to receive Medicare payments missed most problems later identified by state inspectors, potentially compromising patient safety, congressional investigators said Tuesday.
The Joint Commission on Accreditation of Healthcare Organizations, made up mainly of health professionals, failed to find 167 of 241 "serious deficiencies" in a survey of 500 hospitals that were reviewed between 2000 and 2002, the Government Accountability Office said. The agency, Congress' investigative arm, was formerly called the General Accounting Office.
Article: http://www.cnn.com/2004/HEALTH/07/20/hospital.inspections.ap/index.html


text:
5th Annual Atlanta Hip Hop Day Festival
- Music
- 91 Peachtree St. Atlanta, GA 30303, US
Join MPUG today and attend any in-person or web-based meeting at no charge. In addition, you'll have access to Microsoft Project resources, including recorded training sessions, the project management knowledge library, a dedicated project manager career center and discussion forums. Your registration in this event ensures you'll receive email notification about MPUG and its activities.
Atlanta: Annual Microsoft Project Trivia Night
Tuesday, November 12, 2013 from 6:00 PM to 8:00 PM (EST)
Event Description:
Ready to kick off your Thanksgiving season with a cornucopia of fun? Then join PM Resource Group, ATG Learning, and Atlanta MPUG for our annual Microsoft Project Trivia event! Network with your peers during this fun and entertaining learning experience where we all share and expand our knowledge of Microsoft Office Project and Project Server. Four evenly matched teams will compete in 3 rounds of questions on topics related to Microsoft Project and Project Server. Minimum experience with Project is required but you must have a sense of humor, team spirit and willingness to have FUN! In the spirit of the giving season, members of the lucky team with the most points will receive prizes. Food and drinks will be provided!
Learning Objectives:
Increase your knowledge of Microsoft Project and Project Server by participating in a "Jeopardy "style trivia event where evenly matched teams will compete in 3 rounds of questions on topics related to Microsoft Project and Project Server.
Tracy Jacob
Tracy Jacob is the Founder and CEO of PM Resource Group, LLC. Through her leadership, PM Resource Group is one of the most widely recognized organizations for the implementation of successful Microsoft project and portfolio management (PPM) solutions. Tracy also serves as the Manager of Consulting Solutions at PM Resource Group. She has worked with Microsoft enterprise project management products for over 11 years as a solution design and implementation consultant, trainer, and instructional designer.
With her extensive knowledge of technology, project management and portfolio
management, Tracy has led more than 100 PPM implementations within a broad range of industries and including several Fortune 100 customers.
Tracy earned a B.B.A. in Computer Information Systems from Georgia State University. She is certified as a Project Management Professional with the Project Management Institute, a Microsoft Certified Trainer, a Microsoft Certified Technology Specialist in Microsoft Project and Project Server, and a Microsoft Certified Information Technology Professional in Microsoft Project Server.
Tracy is a frequent speaker at MPUG and PMI events throughout the Southeast. Her expertise extends to Project and Project Server from versions 2002 through 2013.
PM Resource Group is a PMI Registered Education Provider, a Microsoft Certified Partner in Project and Portfolio Management, and a strong advocate of the PMI/MPUG community.
Where Project Managers and Microsoft® Meet. MPUG delivers our users relevant and independent Microsoft Project Management content, resources, opportunities and community networking world wide. MPUG connects thousands of project management practitioners -- to help them better leverage their investment in Microsoft tools such as Microsoft Project, Excel and Sharepoint for Project Management and ultimately advance their businesses and careers.
In order to purchase these tickets in installments, you'll need an Eventbrite account. Log in or sign up for a free account to continue.


text:
In the News | Recent Press | Blog
About Lynx | Development Plan | Payload Capabilities | The Lynx Experience | Lynx Videos | Wet Leases
A spaceflight participant will sit to the right and just aft of the pilot. Lynx will have a pressurized cabin; however, pilot and participant will wear full pressure suits during flight for safety in case of an emergency. The pressure suit will be custom designed for XCOR by one of the leading manufacturers of pressure suits such as David Clark Company or Orbital Outfitters.
Prior to a Lynx flight, potential spaceflight participants will go through a screening process, which will take place over several days and includes familiarizing the participant with all aspects of the suborbital flight experience. To make their flight safer and more enjoyable, spaceflight participants will engage in medical screenings, seminars, and g-force training. This is the basis of our Explorers Wanted. suborbital flight training program.
During the flight, participants will experience an exhilarating rocket ride to space, out-of-this-world views and the feeling of weightlessness. The sky is black during the day and the curvature of the earth is visible, along with the 'thin blue line". After re-entry, participants continue to enjoy this view until descending to lower altitudes and then landing from the same runway that the spaceplane intitally took off from. Below is a graphic overview of the flight profile.
<<< Payload Capabilities | Lynx Videos>>>
Copyright © 2015 XCOR Aerospace, Inc.
Privacy Policy | Terms of Use


text:
Camberley Cricket Club
The home of cricket in Camberley for both players and social members
Featured in Cricket Clubs in Camberley
Ideal Scaffolding (Southern) Ltd
Operating since 1999 and based in Frimley Green near Camberley, Ideal Scaffolding (Southern) Ltd has grown to be recognised as the premier scaffolding company in the area.
Featured in Scaffolding in Camberley
Disposables Direct
Disposables Direct are leading suppliers of cleaning and janitorial supplies and equipment....
Featured in Cleaning Material Suppliers in Camberley
Rolf Evans Photography
Thoughtful, creative, timeless images....
Featured in Weddings and Portrait Photographers in Camberley
Yateley MOT Centre Ltd
For professional, reliable and affordable MOTs, car repairs and servicing, look no further than Yateley MOT - one of the leading MOT test centres in the Camberley area.
Featured in Car Repairs and Servicing in Camberley
Blush Hair, Nails & Beauty Salon
Blush Hair, Nails & Beauty Salon is a stylish hair and beauty salon just minutes away from Camberley town centre, where a friendly team of hairdressers, beauticians and nail technicians offer treatments such as Wella hair colouring, hair extensions, Gelish and NSI nails, Crystal Clear facials, lash extensions, microdermabrasion, waxing and much more.
Featured in Nail Technicians in Camberley
Lightwater Cleaning Services
Lightwater Cleaning Services - with a team of reliable, punctual domestic cleaners and a wide range of cleaning services available, are the ideal choice for quality domestic cleaning in Camberley and the surrounding areas.
Featured in Domestic Cleaners in Camberley
A-Plan insurance brokers
Established in 1963, A-Plan insurance pride themselves on offering traditional, honest, face to face advice on all aspects of insurance....
Featured in Insurance Brokers in Camberley
Direct Cleaning Services
Direct Cleaning Services are commercial cleaners offering high quality and reliable cleaning services to a diverse range of businesses, throughout Camberley and the surrounding area.
Featured in Commercial Cleaners in Camberley
Evolved Pilates
Evolved Pilates is a modern, well equipped Pilates studio in Windlesham, near Camberley, Surrey and offers clients a multi-faceted approach to total wellbeing....
Featured in Chiropractors in Camberley


text:
Syrian men sift through the rubble of houses following an air raid by regime forces in Al-Bab, 35 kms northeast of Syria's commercial capital Aleppo, September 15, 2012. (AFP PHOTO/MARCO LONGARI)
Aleppo, Syria: The army pounded the rebel-held Shaar district of Aleppo on Sunday, killing 11 people, a rights group said, while 11 others including seven rebels were killed elsewhere in Syria's northern city.
"Eleven civilians were killed, three of whose names have been documented, when the army shelled the district of Shaar," the Syrian Observatory for Human Rights said in a statement.
Amateur video posted on YouTube showed what appeared to be the streets of Shaar in ruins, with rubble everywhere, electricity cables hanging from buildings, and black smoke rising.
"God curse you, O army," said an unidentified cameraman recording footage in Shaar, his voice trembling.
Another video showed bloodied corpses, including at least one child.
The violence came a day after UN and Arab League envoy Lakhdar Brahimi warned issued a grim warning after meeting President Bashar al-Assad.
"The crisis is dangerous and getting worse, and it is a threat to the Syrian people, the region and the world," he said.
Seven rebels were also killed elsewhere in Aleppo, the Observatory said.
It said a child was killed in overnight shelling of the southwest Fardoss neighbourhood, adding that a media activist with a rebel group was killed elsewhere in the city, and two civilians including a child were killed in the Masaken Shaabiyeh district.
Fresh fighting erupted in the central Midan district, a battleground for more than a week, an AFP correspondent said, while the Observatory reported clashes in the nearby Arkoub district.
Overnight, the army shelled Bustan al-Basha just north of Midan and pushed into neighbouring Arkoub after seizing a mosque between the two areas following fierce fighting on Friday, a military source told AFP.
Violence has raged in Syria's commercial capital since July 20 when regime forces launched an offensive aimed at driving rebels out of the city.
In Azimiyeh district, 17 unidentified bodies were found, said the Observatory, which has warned that deaths in Syria are becoming more difficult to document.
Elsewhere in the northern province, the watchdog reported one rebel killed in the town of Al-Bab, while waves of residents fled Al-Safira, fearing a security forces operation.
In Damascus, fighting erupted at dawn in the northeast suburb of Harasta, while the army also shelled the southern suburb of Al-Hajar al-Aswad from several directions.
Explosions also rocked the northeast suburb of Douma, while the rebel Assali and Qadam districts in south Damascus were also bombarded.
The northwestern province of Idlib, where much of the countryside is rebel-held, was also pounded. Shelling of the rebel-held town of Kafr Aweid killed at least 10 people, among them women and children, said the Observatory.
At least 66 people were killed nationwide on Sunday, it said.
Meanwhile, at least seven people were killed in an explosion targeting a bus on the road near Khirbet Ghazaleh in the southern province of Daraa, the Observatory said.
A similar attack on a bus carrying civilians and troops killed at least four people a week ago in the central province of Homs.
In Homs province, shelling killed two civilians in the rebel-held town of Rastan, and a man was killed in shelling in Tal Kalakh bordering Lebanon.
The army also pounded the region around the Crac des Chevaliers crusader castle, the watchdog said.
Another man was killed by sniper fire in the eastern city of Deir Ezzor while fighting in the town of Albu Kamal on the Iraqi border killed a woman and a man.
At least 115 people, most of them civilians, were killed on Saturday, according to the Observatory, which gathers its information from activists, medical workers and other sources.
The Observatory estimates that more than 27,000 people have been killed since the uprising against Assad's rule erupted in March last year. The United Nations puts the toll at 20,000.
Your feedback is important to us!
We invite all our readers to share with us their views and comments about this article.
Disclaimer: Comments submitted by third parties on this site are the sole responsibility of the individual(s) whose content is submitted. The Daily Star accepts no responsibility for the content of comment(s), including, without limitation, any error, omission or inaccuracy therein. Please note that your email address will NOT appear on the site.
Alert: If you are facing problems with posting comments, please note that you must verify your email with Disqus prior to posting a comment. follow this link to make sure your account meets the requirements. (http://bit.ly/vDisqus)


text:
End Of Semester Clean Up
Thank you all for a wonderful semester. It was a treat to work with you all.
As you may have noticed from the signs around Regis, you have until May 15 to get all of your stuff out of the building; on that date, all that is left will be recycled/trashed.
Please make sure to come by Regis and
1) clean up the shelves the metal shop
2) take apart and clean up the final installation in the sky way
3) set aside all motors, all solenoids, and all wires that are not yours. Most of that material is my personal belongings, and the rest will be recycled to be used again in future Kinetics courses.
Please leave all the motors/solenoids/wires on the shelves in the metal shop.
Thanks and enjoy your summer,