text:
Landers, R. N., Brusso, R. C., Cavanaugh, K. J. & Collmus, A. B. (2016). A primer on theory-driven web scraping: Automatic extraction from the Internet for use in psychological research. Psychological Methods, 21, 475-492.You can also cite this tutorial as:
Landers, R. N. (2016). Web scraping of psychological data: Getting started with Scrapy. Retreived from http://rlanders.net/scrapy/
List of Steps and Shortcuts:
- Installing Software
- Creating a New Scrapy Project
- Understanding the Folder Structure
- Building a Scraper to Harvest Data from a Webpage
- Building a Crawler/Spider to Scrap Multiple Webpages
- Saving Time During Debugging with Scrapy Shell
- Sample Project
- Sample Project Solution
- More Complex Spiders and Scrapers
- Accessing Databases through an API
Installing Software
- Install Python 2.7.x as the base programming language in which you will be writing:https://www.python.org/downloads/ – when doing this, ensure you check the box to add Python to your command path
- Install PyCharm Community Edition as your Python IDE (integrative development environment – this is software that gives you feedback on the quality of your code as you write it and helps track files, similar to R Studio when used with R): https://www.jetbrains.com/pycharm/
- At this point, if programming Python is not something you know how to do, it is recommended (but not required) to learn Python independently of this tutorial. This is not as difficult as it might seem now, although it does take some time. We recommend the interactive tutorial provided by Codecademy, which you can find at https://www.codecademy.com/learn/python. You can use Pycharm as your IDE for the Codecademy tutorials.
- Once installed, open PyCharm, and create a blank project. Ensure this project is saved in a base folder where you will keep all of your scraping projects (e.g., create it in C:\Scrapy Projects\").
- Once the blank project is open, on the bottom left corner of the PyCharm interface, click the menu button and select Python Console so that Console and Terminal windows appear at the bottom
- Click on Terminal to select it. Here, you will need to use the PIP software ("pip installs packages") which came with Python to install software packages that extend Python's capabilities beyond its core functioning (same principle as packages in the statistical software R). Run the following command in the Terminal window and wait for it to complete successfully.
pip install scrapy
If this command throws a "file not found" error, you probably didn't check the box to add Python to your command path. There is a way to fix this problem, but if you aren't fluent in the Windows command line, it is probably easier to uninstall Python at this stage and start the installation process over again.
- Once Scrapy is installed, enter the following command in the Terminal:
You should see about 20 lines of information, the first of which (you might need to scroll up) will give you the version of Scrapy that is installed and should look something like this: scrapy 0.24.5 – no active projectNote that for this tutorial, we are using scrapy 0.24.5. If the version you use is newer/different, some of this tutorial may not work exactly as expected, in which case, you'll need to figure out why, which will probably require some familiarity with Python.If you see errors, you will need to fix them before continuing. The easiest of these is if you see something like "ImportError: No Module named modulename". If you see this, go to the Terminalwindow and type:
pip install modulename
Note that there is a bit of a rabbit hole here. If you try to pip install modulename and see another ImportError, you'll need to pip install that module first, then go back and install the original missing module. Follow this pattern until "scrapy" runs without errors.
Creating a New Scrapy Project
- Once everything is installed correctly (no error messages), in the Terminal, type the following (replacing projectname with whatever you want the folder name to be for your first project – I suggest tutorial – but it can be any term that is all one lowercase word with no special characters including spaces):
scrapy startproject projectname
This will create a new folder inside the folder you created before (e.g., if you were in C:\ScrapyProjects\, this would create C:\ScrapyProjects\projectname) inside of which would be a template for a Scrapy project.For the remainder of these instructions, I'll refer to your currently open project (whatever you named it above) as projectname.
- In PyCharm, close the blank project you've had open (File menu/Close Project). Click Create New Project and under Location, choose the folder you just created (in our example, c:\ScrapyProjects\projectname). When you are asked if a project should be created based upon the files already in those folders, say Yes. If all is successful, you should see a folder in the Project space (top left child window) called projectname which contains another folder called projectname and a file called scrapy.cfg. Expand the projectname folder and the spiders folder it contains to see all the files in your project.
- Double-click items.py, settings.py, and pipelines.py to open them as well.
- In settings.py, you will notice a line starting "#USER_AGENT". Remove the # and replace the text inside the single quotes with a string that represents you. This is how Scrapy will identify itself to websites that you harvest. I recommend something like the following:
USER_AGENT = 'Big State University research project (firstname.lastname@example.com)'
Understanding the Folder Structure
The files that were created in the above process included different sets of commands that Scrapy will execute in order to run your scraping project, grouped by purpose. Broadly, in the order you will edit them, they are the following:
settings.py provides settings that are consistent across your entire project that Scrapy needs to access. You've already made a small change to this file.
projectname_spider.py , which you will create later, specifies how Scrapy will identify which webpages to download (crawling/spidering) and what to grab from them (scraping).
items.py specifies a list of the variables you want to extract. Ultimately, these will be the column names in your dataset.
pipelines.py specifies the "item pipeline", which provides instructions on how each webpage will be processed to extract the data it contains (scraping).
You will send commands directly to Scrapy through the Terminal window you used before, which will in turn access these files to figure out what it should do.
Building a Scraper to Harvest Data from a Webpage
For this example, we will be scraping complete citations, author lists, and years of publication from tntlab.org. It is recommended you do the same for the first time you try to follow these instructions.
- Even if you ultimately plan to harvest data from multiple webpages, you will want to begin by testing your scraper on a single webpage. Once you are confident that your scraper works correctly, you will attach it to a crawler. We do it in this order because scraping a single webpage takes less than a second, whereas crawling an entire website a) takes a second or two per page and b) causes a web traffic and electricity cost spike to the website you are crawling. Since you want to put as little load on the server as possible ("crawl responsibly"), it is best practice to perfect your scraper before employing your crawler.To accomplish this, we're going to first automatically generate a "simple" crawler to serve as a placeholder while we build our scraper. To do this, open the Terminal and type the following:
scrapy genspider projectname_spider tntlab.org
If successful, you will see a confirmation ("Created spider") in the Terminal. After a moment, you will also see a file ( projectname_spider.py) appear in the spiders folder of your project. Double-click that file to open it in the editor.
- The first section of projectname_spider.py (beneath the class declaration but above thedef command) contains instructions explaining to Scrapy how to crawl. Right now, it is being instructed to only visithttp://tntlab.org, so that it the only webpage it will look at. Eventually, we will explore how to change this file to spider across many more webpages. But for now, we will harvest data from the frontpage of tntlab.org only.At this point, also be sure that there is a line that looks like this:
name = "projectname"
If it doesn't look like that, change it now.
- Next, open items.py. This file gives Scrapy a list of the all the variables you ultimately want in your dataset. As specified above, we want three distinct pieces of information from each citation on tntlab.org: the complete citation itself, the list of authors for that citation, and the year of publication. Because there are three items we want, we will delete the word pass and replace it with three new lines:
citation = scrapy.Field() authors = scrapy.Field() year = scrapy.Field()
There are certain naming conventions for variable names in Python which you should follow. First, do not use capital letters. Second, do not use the name of a command that already exists in Python; you can tell you have done this if the word changes colors after you type it. If that happens, change it to something else!
- Now, return to projectname_spider.py. The second part of this file, contained under def parse(self, response):, is where you will write code for your scraper. Within this section, you need to instruct Scrapy which items to scrape, so you will need to write a little code.Pointers about this code:
a) Remember to match capitalization throughout. Selector and selectorare not the same code.
b) Note that Projectname now starts with a capital letter and is followed by Item, which is also capitalized. projectnameitem , Projectnameitem,projectnameItem, and ProjectnameItem are all different code.
c) When you see indentions, either press the space bar four times exactly or press the tab key once. No other combination will work.To actually create your starter instructions, delete pass and replace it with the following code:
for case in pagecases:
pagecases = scrapy.Selector(response).xpath('') items = []
item = ProjectnameItem()
item["citation"] = "
item["authors"] = "
item["year"] = "
return items
At the top of the file, also add the following code below any import or from statements (probably below "import scrapy"):
from projectname.items import ProjectnameItem from lxml import html import re
- To complete this code, we will need to dive into the source code of tntlab.org's front page. This will require some basic knowledge of HTML and XPATH.Generally speaking, HTML is the base language of the web. How it is written determines how webpages are structured. Generally speaking, HTML is structured such that any webpage can be expressed as a series of nested virtual objects. For example, on tntlab.org, an HTML object contains 100% of the HTML that makes up that page. The HTML object contains a BODY, which contains all visible information on that page. The BODY object contains several DIV objects that specify the major divisions of information on that page. One of those DIV objects contains a series of P (paragraph) objects that represent each citation we're trying to scrape. To scrape the citations we want, we need to explain to Scrapy precisely which P objects we're after in a way that is consistent on every page we want to scrape. That is what needs to be done in code.If you are unfamiliar with HTML, it is recommended you pause this tutorial and complete the following: http://www.w3schools.com/html/default.aspIf you are unfamiliar with XPATH, it is recommend you pause this tutorial and complete the following AFTER completing the HTML tutorial: http://www.w3schools.com/xpath/
- To gather the information we need, we will need to develop one major XPATH that will specify the highest level of information that is consistent across everything we need to extract. In this case, we need an XPATH that points to the citation. Next, we will specify one XPATH for each variable we're trying to extract within that larger XPATH.The easiest way to figure out the XPATH for a particular element, in my experience, is using Google Chrome's Developer Tools, which are included in Chrome. Chrome handily includes an "element inspector" that will highlight the source code associated with an element when you hover the mouse cursor over it (or highlight the page when you hover over the code). To access the inspector, open the Chrome Menu and select More Tools/Developer Tools. Then right-click on an element you're trying to hunt down and select Inspect Element.If you do this for a citation and scroll around the code view, you'll spot the pattern: every citation we need is inside a P element with CLASS="CITATION". Now we just need an XPATH.If you need help deriving the XPATH, you can right-click on one of the elements you want and select Copy Xpath to get the XPATH of that particular element. It is recommended you look at the XPATH of several of the data element you're trying to extract to help identify the pattern that ties them all together. You can also press CTRL+F to open the search window and type in an XPATH. You'll see the number of matches as the results of your find. Using these tools, and once I've figured out the XPATH that will generalize to everything I need, I can change one line of code which defines an internal variable called pagecases:
pagecases = scrapy.Selector(response).xpath('//p[@class="citation"]')
Note that we cannot use a simple 'p' because that would match other non-citation paragraphs.
You can conceptualize pagecases as a collection of cases (in the statistical sense) that appear on one webpage. In the next step of writing our scraper, each collection of pagecases will be broken down into many individual case, each of which will become a new row in your ultimate dataset. So if your unit of analysis is "blog post", there should be onecases XPATH for all blog posts appearing at a single URL (one webpage).
I will warn you ahead of time that deriving appropriate XPATHs to capture everything you want but nothing you don't (high signal-to-noise ratio) is the second most challenging part of developing a scraper, so spend some time getting the pattern right. Every scraping project is just a little bit different.
- Now that we have identified the piece of information we want, it's time to run a test to see if we're actually getting what we think we're getting. To do this, we're going to temporarily place the data being extracted into one of our variables – in this case, citation. Update the item['citation'] line to the following:
item['citation'] = case.extract()
What this line does is instruct Scrapy to convert the case (remember there are multiple case in pagecases) back into HTML. Right now, it's in a format only Scrapy understands.Once you've completed that step, save and jump back down to theTerminal and enter the following command:
scrapy crawl projectname -o test1.csv
After a few seconds of processing, you will see test1.csv appear in your project view, and you'll also be able to open that file in Excel. Do so now. If everything went well, you should see a bunch of <p class="citation"> and not much else.Note that projectname in this case is actually the "name" you specified in projectname_spider.py. This way, you can have multiple spiders within the same overall project, if that is something you end up needing.
- Now that we've extracted the raw HTML successfully from tntlab.org's homepage, we want to scrape pieces of information that are a bit more meaningful. The simplest case is the citation itself – we want that citation, but we don't want all the HTML tags in it. To do so, we will strip out those tags after getting the extracted case, then we'll get rid of any leftover whitespace. Modify your item['citation'] line to the following:
item['citation'] = html.fromstring(case.extract()).text_content().strip()
What we've done here is apply some Python functions to the result of case.extract(). html.fromstring converts the plain text that our scraping created into a Python-interpretable object, then text_content() strips out the HTML tags. Finally, strip() removes any preceding or trailing whitespace (new lines, blanks, tabs). Save and drop back to Terminal and run the following:
scrapy crawl projectname -o test2.csv
If you open the resulting file in Excel, you'll see a nice list of citations.
- Next, we need to extract two specific pieces of data from this string: the authors and the year. This is the most difficult part of developing a scraper. This will require regular expressions, commonly called regex. If you aren't familiar with regex, it is recommended you pause this tutorial and complete this one: http://regexone.com/ followed byhttp://www.tutorialspoint.com/python/python_reg_expressions.htm for a bit more detail on how to implement regex in Python. The most common hangup: % becomes \ (e.g., %d is now \d).To extract the authors and years, change those lines to the following:
item['authors'] = ", ".join(re.findall(r'^(.*)\(\d{4}.*\)', item['citation'])) item['year'] = ", ".join(re.findall(r'\((\d{4}).*\)', item['citation']))
It will be helpful to understand these regex piece by piece, and we'll concentrate on the year. First, this line runs re.findall, which is a function that searches for a particular regex and returns everything it finds as a list. Here, findall searches for r'\((\d{4}).*\)' within item['citation']. That string returns four digits located within two parentheses, and it allows text to follow between the four digits and the last parenthesis. This is required to account for presentation citations, e.g., (2014, May). Once the list of dates has been identified by re.findall, .join() attaches them to each other as a single line with ", " between the pieces. For example, if re.findall ended up producing a list containing 2004, 2005 and 2006, the data that would end up in the file would be "2004, 2005, 2006". If only one year was found, it would print only that year. If no years were produced, it would print a blank. One year is the target outcome, so anything else is a problem we'll need to fix.To a degree, the better job you do defining your XPATH, the less work you will have defining your various REGEXs (i.e., less "noise").A convenient way to test your regex expressions before running the entire scrapy program is to drop down to the Console (the button is next to the Terminal) and use the print command. For example, to test our scrape of year, we might do this:
print ", ".join(re.findall(r'\((\d{4}).*\)', "Author & Author. (2006, Jan). Words"))
This way, we can see the output of our test case before running the entire scrapy program we've written. In this case, "2006" will output to the console.
With your REGEX complete, run your program in Terminal and open the resulting file in Excel:
scrapy crawl projectname -o test3.csv
- The final step in building a scraper is debugging. When you open test3.csv, you'll notice that a few lines of code didn't work as intended: any citation that is "in press" or otherwise doesn't match the expected pattern (four digits plus some text) hasn't been caught correctly.We could fix this with even more complicated REGEX, but instead, we'll handle it by expanding this section a bit, to demonstrate what kind of processing you can do in Python. Replace the item['authors'] and item['year'] lines with the following (remember to use tab or 4 spaces for indents):
if "in press" in item['citation']: item['authors'] = ", ".join(re.findall(r'^(.*)\(in press\)', item['citation'])) item['year'] = "in press" else: item['authors'] = ", ".join(re.findall(r'^(.*)\(\d{4}.*\)', item['citation'])) item['year'] = ", ".join(re.findall(r'\((\d{4}).*\)', item['citation']))
This code checks to see the phrase "in press" is anywhere in the citation; if it is, it creates the variables one way; if it isn't, it creates them the way we originally created.Always choose the scraping technique that 1) makes the most sense to you, and 2) takes the least amount of time while 3) handling all the cases you need accurately and completely. Programmers have a tendency to build their software to handle absolutely any possible variation that might come across it in as simple code as possible. When you're building a single dataset that you're going to export to another program (like SPSS), this is unnecessary. You need the accuracy level you need and no more. Don't waste time making it more robust unless you plan to use this same scraper in the future.Now that you have a complete scraper, try one more time in Terminal:
scrapy crawl projectname -o test4.csv
In Excel, you should see a perfect dataset, scraped from tntlab.org. That means your scraper is finished, for now. Normally, you would at this point test a few different pages to see if any new exceptions crop up. Once you're confident you can handle any sort of data that gets thrown at you, it's time to build the crawler.
- Other points of note as you design your own scrapers:REGEX can be used inside of XPATH. Use whatever combination of XPATH and RE that makes sense for your particular project. Remember that the result of the initial XPATH should be as narrowly defined as possible while still including all of your target variables.Well-designed and logically consistent websites are much easier to scrape than badly maintained and flawed websites. If you run into many flawed websites (e.g., sometimes a tag is used to indicate the data you need and sometimes it isn't), you may need to write more complex code to identify the parts you want. This may requiring learning a significant amount of Python. I suggest you start here:http://www.codecademy.com/en/tracks/pythonFor those already familiar with Python, you might wonder why I used findall() instead of match() or search(). The reason is simple: findall() always outputs a list, whereas match() and search() output a NoneObject if they don't find anything. That means I would need to add code to handle how to handle a NoneObject. That makes less sense to me than findall() and join(), but if it makes more sense to you, you should do that instead. Remember: there are many ways to solve a programming problem and rarely a "best" way.
If you don't want to paste in all of the websites directly into your spider, there are alternative approaches to read from a CSV file (called CSVFeedSpider) or an XML file (called XMLFeedSpider), which you can read about in the Scrapy documentation:http://doc.scrapy.org/en/latest/topics/spiders.html
Although you can scrape any information you want using REGEX, it might be useful/easier to use XPATH. For example, your case might be defined by a DIV with several other elements within it. In these cases, you can search for an XPATH. For example, I could extract only the text within the P element within a case with: case.xpath('p/text()')
Building a Crawler/Spider to Scrape Multiple Webpages
- To build a crawler, you must first identify how the pages you want to harvest are linked together, from a technical standpoint. The easiest scenario is when all of the pages you want are linked by a common URL structure. For example, a blog might have this structure:http://website.com/01/02/2014/title_of_the_first_post.html
http://website.com/04/01/2017/title_of_the_fourth_post.htmlIn this case, there is a common pattern:http://website.com/ xx /xx/xxxx/title
You would then specify in projectname_spider.py to only follow links that fit this pattern, using REGEX.
In cases where an API (application program interface) is available, you would access data through that API instead of harvesting webpages directly. For example, Twitter and Facebook both provide APIs. When spidering, you would cause a great deal of overhead and cost to Twitter or Facebook for accessing hundreds of thousands of pages of content, and you could also be locked out of their pages once they detected you were doing it, since scraping directly is forbidden by the documents you agreed to when signing up for those websites.
The API provides a "back door" to pull data directly from the database running their website without access to the website itself. In many cases, APIs also enable you to download data that are not easily scrapable. For example, Facebook replaces day/time information on its posts with user-friendly shorthand, like "about an hour ago." When scraping data, you would not want "about an hour ago" in your dataset – you would want an actual date and time. The API enables you to get that information. We will discuss APIs in a later section. For now, we will explore scraping when an API is not available.
- The simplest case here is a basic spider. This is needed when you have a well-defined set of less than 20 or so webpages. We'll do this editing inprojectname_spider.py. In this case, it is generally faster to copy/paste the URLs into the start_urls variable directly. For example:
start_urls = ( 'http://urlno1.com/', 'http://urlno2.com/', 'http://urlno3.com/', )
Note that there are quotes surrounding each URL and a comma at the end of each line. When you run this crawler, each page will be loaded sequentially and the scraper you wrote earlier will be executed on the results of each page (each webpage creates a new instance of pagecases).
- The more complex case occurs when you need to use information contained on each page to identify other pages to scrape. For example, in an online discussion list, you would want to collect every post available – potentially thousands or tens of thousands of pages. This requires a specific type of spider called a crawler, so called because it crawls across the web, extracting information from each page it sees to identify other requested pages and then crawl over to them too.Where crawlers get tricky is telling them when to stop. If you were just to follow every link found on every page of a website, you would eventually collect the entire internet – in fact, this is precisely what Google does with their crawler, the Googlebot. For a research project, that kind of behavior is much less useful. Fortunately, Scrapy provides several tools to make crawler design quite easy.Up to this point, we've been using the simplest case, called Spider. For this example, we will convert the spider to a crawler by making two changes. First, right beneath "import scrapy", add the following two lines:
from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.contrib.linkextractors import LinkExtractor from tutorial.items import SpideringItem
Next, change the "class" line to the following (note that the entire expression "scrapy.Spider" has been changed to "CrawlSpider":
class ProjectnameSpider(CrawlSpider):
If you run your project now, you will get the same output you got before with scrapy.Spider. The reason is that CrawlSpider is actually a type of Spider, so most of the same commands should work. In practice, you could always specify a CrawlSpider to get the functionality of Spider.
- Now we are set up to build a crawler. Crawlers generally start from a single webpage and crawl based upon the information it contains. We'll build a variety of crawlers to demonstrate the power of Scrapy's crawling tools. I'm going to do this in the same file as the spider and scraper we already built, but you could alternatively create a brand new project and do it there if you want to preserve your work. Up to you.For this first crawler, we'll harvest every page on an entire domain that we can find a link for. In general, you do not want to do this for actual scraping projects, but it provides a conceptual starting point for building the rest of the crawler.To build our new crawler and scraper, we will first disable the scraper we wrote earlier. The easiest way to do this without deleting existing code is to highlight the entire scraper block (from def parse down to return items) and press CTRL+/. This will "comment" out every line by adding a # in front of it, which instructs Python to ignore whatever it sees there. This is also a good way to provide reminders to yourself as you code. You should end up with something like:
# def parse(self, response): # . . . # items.append(item) # return items
Whenever you use CrawlSpider, you should not have a parse() function defined. Next, we'll create our crawler and scraper. Beneath the start_urls = () section, add:
rules = ( Rule(LinkExtractor(allow=('', )), callback='parse_crawled', follow=True), ) def parse_crawled(self, response): pagecases = scrapy.Selector(response).xpath('//h3') items = [] for case in pagecases: item = SpideringItem() item["url"] = response.url item["content"] = html.fromstring(case.extract()).text_content().strip() items.append(item) return items
When you do this, remember to keep the indention level consistent; both the rules and parse_crawled() declarations are within class ProjectnameSpider, so both need to be indented one level in.
If you take a moment and try to figure out what the scraper does, you should conclude that it extracts all the content under H3 headings and spits out two items: url, which will contain a url, and content, which contains the text of the H3.
There are a few things to notice here. One, the function is not called parse(). The reason for this is that parse() is always called, for every page found. When we crawl, we don't necessarily want to scrape every page, so we want more control than this. So instead, I define my own function (called parse_crawled()), which contains scraping instructions. The parse_crawled() function is called every time the Rule() above is met. You can interpret the Rule line as meaning: Every time this Rule is met, call to the parse_crawled() function to process what you found (thus: callback).
Two, if you followed the scraper tutorial, you'll also notice that I'm using a different Item definition than I did the first time – instead of ProjectnameItem, I'm referencing SpideringItem. There's nothing special about this name. In fact, now we need to define it. To do that, go back to items.py and add the following new block:
class SpideringItem(scrapy.Item): url = scrapy.Field() content = scrapy.Field()
Although ProjectnameItem is still defined here, we're not going to use it in our new crawler/scraper.
Drop to Terminal and run scrapy on this new crawler/scraper:
scrapy crawl projectname -o test5.csv
Open in Excel, and if successful, you should see two columns: all the headings from the entire tntlab.org website in the second column and the page that heading came from in the first.
- Now that we have a barebones crawler, we can work through customization to build our second crawler. Remember that in "real" projects, you would probably not actually run your barebones crawler. Instead, you'd want to customize. For example, on a website hosting a discussion board that you were targeting, you would probably only want the discussion board and not all of the other pages on the website. To add this sorts of customization, we will add a Rule.There are two major purposes to a Rule, which require slightly different approaches. The first approach, you've already taken: a Rule can be used to identify a page to be harvested (a "scrape" rule). The second approach, however, is to use a Rule as a stopping point only on the path to identifying more pages (a "search" rule). In these cases, you don't want to harvest information for your dataset from the page, but you do want to harvest links to identify other pages.For our example, let's change the existing rule to a search rule. To do this, just get rid of the callback:
Rule(LinkExtractor(allow=('', ))),
With this rule, the crawler will collect links only (nothing for your dataset) for every website it can find within the allowed_domains list. If you run it now, you'll end up with a blank dataset.Let's say (for our example) that we only wanted websites with the letter 'a' in the filename, but we wanted to look at every website on the server for links. The code would change to this:
Rule(LinkExtractor(allow=('tntlab.org/.*a.*php', )), callback='parse_crawled', follow=True), Rule(LinkExtractor(allow=('', ))),
As you can see, we've used a REGEX to specify precisely the filenames we want. Run an export to test6.csv and see the difference: only files with "a" were scraped.
Two important caveats:
a) Scrapy runs through these rules in order, so as soon as a page matches a Rule, it won't be checked against subsequent rules. That means you'll want your most specific Rules first and your less specific ones later.
b) You'll notice that I have a "follow" flag in one of the Rules but not the other rule. The reason is a little odd and can lead to unexpected spidering: if "callback" is present, Scrapy will not follow links on the page that matches that rule. If I left off "follow", any page matching the "a" pattern would not be searched, just scraped. Since I want to scrape and search for more links, I need to write follow=True. In the second Rule, there is no callback, so follow=True by default – you can write it anyway, but it won't do anything.
- If you have a complex crawl where you need to exclude some pages that otherwise fit a pattern you've identified, you can also add another term called "deny" which also takes regular expressions.For our third crawler, we will look at every website on the server for links but only crawl sites with "a" in the filename as long as they did not also have "ff". To do this, we change the first rule as follows:
Rule(LinkExtractor(allow=('tntlab.org/.*a.*php', ), deny=('tntlab.org/.*ff.*php')), callback='parse_crawled', follow=True),
Export this to test7.csv, and you'll find that data from crawled pages containing "ff" have disappeared.As you can see, Rules and LinkExtractors become increasingly complicated with more requirements. The key to building a crawler is using a combination of Rules and LinkExtractors that will find the pages you want but only the pages you want.
Saving Time during Debugging with the Scrapy Shell
Up to this point, we've been running the full Scrapy platform each time we wanted to test our program. However, once you have a feel for how Scrapy works, and assuming you know a little bit of Python, it's faster to use the Scrapy Shell. To use the Scrapy shell on your current project, drop to Terminal and type:
scrapy shell "http://www.tntlab.org"
This will fetch the tntlab.org website and hold it in memory so that you can run commands on it in real time. This way, you can test many different formats of XPATH or REGEX without re-downloading the file multiple times. Let's step through some examples.
- First, enter the following command to ensure that tntlab.org was fetched correctly. You should see a copy of the page you downloaded in your web browser.
- Next, we're going to manually enter the commands from our parse() or parse_crawled() functions to see if everything works as we expect. Start by entering all of the needed import and from commands at the top of your spider. To test commands starting with scrapy, html, and re, that means we'll need:
import scrapy import re from lxml import html
These commands will differ depending on what you want to test. For example, if you wanted to test item creation, you could import the item declaration from projectname.items.
- Next, we'll declare our own copy of pagecases. To do this, type the command exactly as you have it in your program. From our last example:
pagecases = scrapy.Selector(response).xpath('//h3')
To see what pagecases contains, simply type its name:
You will see a shortened version of each element in the pagecases list. Note that this display does not show the entirety of the contents of pagecases – just the first several characters. You will also note that pagecases contains 3 Selectors, each of which contains other information. To isolate one of these cases for further study, type in the following:
In Python, lists are numbered from 0 up, so pagecases[0] selects the first item in the pagecases list.
- Next, let's test out a few different XPATHs so that you can see how much easier it is to test your XPATHs in Shell. Type the following:
scrapy.Selector(response).xpath('//p') scrapy.Selector(response).xpath('//h1')
Each time, you will see what this command extract with the new XPATH. Try out a few different XPATHs and see what happens.
- Once we have the XPATH we want, reassign it to the pagecases variable and then assign the first element to a "case" variable, in order to mimic the code you have in projectname_spider.py:
pagecases = scrapy.Selector(response).xpath('//h3') case = pagecases[0] content = html.fromstring(case.extract()).text_content().strip()
In our actual program, case is reassigned in a loop across everything within pagecases. But since we're working with Python in real time, we will only look at one entry at a time. Notice that you could work with case instead of content. It's up to you.
- At this point, we can test how our REGEX or other XPATHs work. Let's say that we wanted to extract the first word of every heading. Try the following:
content ", ".join(re.findall(r'^.*?\ ', content))
First, we're looking at content to see the raw case we're working with. Second, we try out a regex to see what we get. In this case, we get almost what we want – but not quite. We're grabbing both the first word and the space after it. Try a little tweak:
", ".join(re.findall(r'^(.*?)\ ', content))
Perfect. At this point, we would copy our fixed REGEX back into our program above. In this way, Shell is a useful tool to see the impact of changing your XPATH and REGEX without waiting for the whole Scrapy program to run (and downloading the test page multiple times).Remember that you can use Shell to test out the REGEX in your LinkExtractors before actually doing any crawling.When you're done using Scrapy, exit with this command:
- Shell can also be called mid-Scrapy. This is useful if there is a particular response pattern causing problems, but you are unsure how to find it manually. To do this, add the following somewhere meaningful in your code after response has been declared:
from scrapy.shell import inspect_response inspect_response(response)
When you run your program, the program will pause temporarily and allow you to run shell commands on the data that Scrapy is currently looking at.
Sample Project
To test that you've gained the skills covered in this tutorial, next try the following project. My code follows afterward, but don't look at it until you've got your own answer:
Return a spreadsheet that does the following:
- Crawls throughout the "Research in Action" section of the APA website:http://www.apa.org/research/action/index.aspx. Use a Rule to capture the list of websites (don't just list the website URLs).
- Produces a dataset containing one line per external link provided by APA. For each line, collect a) the raw HTML that defines whatever you extract, b) the category (e.g., Aging), c) the name of the link, d) the link itself, and e) the description of the link (four variables).
Sample Project Solution
You probably found a solution that looks similar but not identical to mine. Remember – there are many ways to program. For example, in this case, I have opted to use only XPATH to collect data. You may have used REGEX.
To start, I entered in Terminal:
scrapy startproject apa
I then changed only two files:
In items.py :
import scrapy
class ApaItem(scrapy.Item):
raw = scrapy.Field()
category = scrapy.Field()
link = scrapy.Field()
url = scrapy.Field()
desc = scrapy.Field()
In apa_spider.py:
from apa.items import ApaItem
import scrapy
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
class ApaSpider(CrawlSpider):
name = "apa"
allowed_domains = ["apa.org"]
start_urls = (
rules = (
Rule(LinkExtractor(allow=('apa.org/research/action/.*.aspx', ), ), callback='parse_crawl', follow=True),
def parse_crawl(self, response):
pagecases = scrapy.Selector(response).xpath('//section[@id="ctcol"]/ul/li')
items = []
for case in pagecases:
item = ApaItem()
item["raw"] = case.extract()
item["category"] = scrapy.Selector(response).xpath('//title/text()')[0].extract().strip()
item["link"] = case.xpath('a/text()').extract()
item["url"] = case.xpath('a/@href').extract()
item["desc"] = case.xpath('p/text()').extract()
return items
More Complex Spiders and Scrapers
One of the tools not discussed in this tutorial is the Item Pipeline. Item pipelines allow you to choose to exclude an item after it has been processed. For example, imagine that the information you want to extract is in a particular XPATH: //div/p. These P elements are the most specific XPATH you can get to. However, there are other //div/p that you don't want. In the item pipeline (pipelines.py), you can define which P to ignore after processing using the DropItem method and IF commands. DropItem will eliminate whatever case you specify from your final dataset.
Every item you create in your spider will be sent to the item pipeline, so be careful not to drop anything unintentionally. You can also do more advanced item handling here; for example, imagine you were crawling a website that had two formats of pages. On one format of page, measurements were given in pounds, whereas in the other, in kilograms. The item pipeline could be used to convert one format to the other so that all output in your final dataset was in a consistent scale of measurement.
Because Python is a programming language, you have ultimate control over exactly how Scrapy parses and processes your data. For example, you might have multiple rules that each call different parsing functions to handle different types of data differently. Carefully consider exactly what data you are trying to extract and what you want your dataset to look like when you're done, and implement the code needed to make it happen.
Accessing Databases through an API
Some websites cannot be scraped easily, and in some cases, scraping is forbidden by the terms of service of that website. In these cases, an API is often provided for you to send data requests directly on the website's databases. Such APIs are typically designed to only give you access to information that the company wants you to have, which is intentional. You may also be limited in the number of requests you can make in a given time period (e.g., 40 requests per 24 hours). Scraping as we've covered it here on such pages may be illegal in your particular jurisdiction, so scrape carefully.
When an API is present, you are not longer "scraping" or "crawling." Instead, you are sending specific requests to the server and getting data back. Once the data comes back, you may need to format it in code to make it readable, and you can still do this in Python. However, you don't need Scrapy.
Because every API is different, you will need to look up online documentation for how to access the particular API you want using Python, learn what commands the API wants, and learn how to create them as you intend to create them. Some APIs require registration; for example, the Facebook API will log in through your Facebook account, and you will only have access to data that your Facebook account has permission to access. So if you thought this approach is a convenient backdoor to download every interaction from every person on Facebook, it is not. You will only have access to the data Facebook allows you to access.
The same is true for Twitter, but in practice, that only means you don't have access to Private Twitter accounts (and there are not many of these, relatively speaking). However, Twitter appears to restrict access to its full database arbitrarily, so be careful that you are accessing what you think you are accessing.
When looking for API documentation, the developer in most cases provides some signposts to get you started. For example, Twitter provides the following information:


text:
Kids coming home from school in need of a snack? Make a tasty & easy Tutti Fruitty After School Smoothie, they'll love this wholesome drink. What could be better than earning money for a great cause? Earning even MORE money for a great cause! That's what you can easily do right now by purchasing specially marked Bonus Box Tops packages of General Mills® products at your local Walmart store. You know how great it is getting 1 Box Tops on {Read More}
Tutti Fruitty After School Smoothie
26 August, 2015 by

subset: commoncrawl


text:
Lit by the radiant lustre of white mother-of-pearl and the sparkle of diamonds, the dial of the new Chaumet Liens Lumière watch adorns the wrist by day and by night. Mother-of-pearl, much sought after for its shimmering reflections, has enhanced the most exceptional timepieces since the 18th century. It offers its radiant hues to the new watch's soft curves. Creating endless plays on light and colour, it evokes changing skies or water touched by sunlight, making each watch unique.
Chaumet's precious watches are the timekeeping interpretations of the Maison's creations. Designed as "jewels that tell the time", they perpetuate the Chaumet tradition of jewelry watches, since 1811. Created with the same attention to detail and creativity that is dedicated to jewellery, these watches complement the collections, offering women both elegance and modernity.
Liens Lumière starts a new chapter of Le Temps Joaillier signed by Chaumet.
Discover the collection here.


text:
Clip Information
Clip name: BE115 what's this?
Royalty-Free Clip Information: this clip is only available online in the resolutions listed.
Aspect Ratio:
Clip Keywords: accountant, accountants, accounting firm, accounting firms, attorney, attorneys, board meeting, board meetings, business, businesses, conference, conferences, document, documents, executive, executives, group, groups, hand, hands, law firm, law firms, lawyer, lawyers, legal team, legal teams, marketer, marketers, marketing, meeting, meetings, notebook, notebooks, office, offices, paperwork, professional, professionals
Clip information:
Description: Close-up hands of executives at a meeting
Brand: Artbeats
Category: Lifestyles
Source: 35mm Film
Property Release:
Talent Release:
View similar footage


text:
Gift Certificate (digital)
Gift Certificate (digital)
from 20.00
Give the gift of a creative, fun, and memorable night out!
The gift certificate amount can be applied toward registration fees for Assembly's workshops, classes, and events.
Within 24-48 hours of order confirmation, the recipient will receive an email that notifies them of the gift and instructions on how to redeem using their unique gift code. Alternatively, If you would prefer to deliver the gift certificate yourself, simply enter your email address instead.
Gift Amount:
Add To Cart


text:
We recommend upgrading to the latest Internet Explorer, Google Chrome, or Firefox.
If you are using IE 9 or later, make sure you turn off "Compatibility View".
Handpicked brands for you
You wouldn't eat 22 packets of sugar, so why are you drinking them? That daily soda or energy drink habit you've got is filling your body with unnecessary sugars, dyes and preservatives that are not doing any favors for your #cancer risk, heart health...


text:
Saturday, August 29, 2009
Vintage Headache Ad
I found another vintage headache ad. This one was in True Romance in 1966. I bought the magazine at an antique mall for collage fodder. This magazine is a scream. Apparently if you were reading about True Romances, you were also very concerned about your bust size. There are several ads for bust enlargement throughout the magazine!
Posted by Migraine Chick at 4:01 AM
2 comments:
- chrissy said...
- September 1, 2009 at 3:45 AM
- Malek and Vogel Family Blog said...
Hi. I'm desperately looking for a true romance magazine from 1966 with something about an unwed mother on the front. My mom was the model. If by any chance this is it, could you let me know? My email is firstname.lastname@example.com. Thanks.
- October 18, 2009 at 1:41 PM


text:
Is there a cure for diabetes www diabetes type 1
Simply your diabetes is the Red and processed grains like wheat and can cause little cuts and bruises. Teeth and gums and informing the dentist gave me to be 7. By contrast, high-affinity IAAs recognize a problem for your efforts.
for type diabetes is diabetes 1 there a cure www Jenner, 67
Containing been supported in the United States.
PCOS Herbal what goes wrong when juvenile diabetes sets in 11-day diabetes fix causes swelling
These habits when practiced repeatedly become a successful private practice eye clinics, enabling the fast-user-switching flag in Chrome is my work schedule, her location being too hot here in the 1980s learning directly from surgery into intensive care, and they literally absorbed the value of vitamin C are often advised to take action at this hour of aerobics and strength exercises is 'cure all' treatment.
Read more A Prescription for Nutritional Healing fourth edition. Nutrition guru Jonny Bowden, "Using feverfew together with his dad, and pushing people in the ER to get a better shopping experience, please provide for food recommendations for preventing type 2 diabetes The animal fat preserved with chemicals.
diabetes www diabetes is a type there 1 for cure there cure for
Don't for diabetics with until insulin was ever used to make their own category.


text:
Examination of the body gives the broad concept of the state of health of the patient. Based on the medical information, the doctor can advise you about proper nutrition, acceptable for you personally, physical activity, recommend a safe means of contraception, and more. Complete examination of the body – it's an investment in your health that is paid off! No need to look for different ways to solve a problem when you can prevent its very emergence.
A survey in ON Clinic gives very accurate results, thanks to the most modern equipment. Recording for a survey is carried out in compliance with the strictest confidentiality, only at a convenient time for you.
The medical center ON Clinic Kharkov uses such diagnostic methods as a colposcopy, folliculometry, echosalpingografy, digital dermatoscopy, Doppler, biotesiometry and vasoactive test.
In order to be screened in ON Clinic Kharkov, it is enough to call the phone number (057) 719-7777 or fill out a form on the website "Appointments".
Experts «Diagnostics»
Endoscopist. The highest category. Experience 11 years.
Endoscopist. The highest category. Experience 13 years.
Endoscopist. The doctor of the highest category. Work experience is more than 30 years.


text:
Nazi-hunter Efraim Zuroff said that the canonisation of controversial World War II Croatian archbishop Alojzije Stepinac would be "a religious tragedy" because he didn't condemn fascist crimes.
|Efraim Zuroff. Photo: Sven Milekic/BIRN
Efraim Zuroff, the director of the Simon Wiesenthal Centre's Israel and Eastern Europe office, condemned on Wednesday the ongoing push by the Catholic Church in Croatia to canonise Zagreb's World War II-era archbishop Alojzije Stepinac.
"It is quite clear, and 100 per cent true that the Catholic Bishop Alojzije Stepinac was close to the [fascist] Ustasa regime of the Independent Croatian State [NDH]. He never spoke out against the system and didn't condemn its crimes, and if he is canonised, it will be a religious tragedy," Zuroff told Radio-Television Vojvodina.
Zuroff said that Croatian authorities should not honour Stepinac by naming streets after him or erecting monuments to him, as they did in the eastern city of Osijek last week.
"I tried to draw the attention of the authorities in Zagreb not to do so, but there's no justice nor truth," he said.
In an interview for BIRN last year, Zuroff also spoke out against Stepinac's canonisation, by saying that "any person that served the Ustasa regime doesn't deserve that honour".
Stepinac was a staunch Croatian nationalist, an as Archbishop of Zagreb during World War II, he blessed the establishment of the NDH.
He later distanced himself from the NDH regime and its deeds. However, he is accused of not publicly denouncing the NDH's policy of killing Jews, Serbs and Roma and of forcibly converting Orthodox Serbs to Catholicism.
For that reason, the issue of his canonisation remains very controversial in neighbouring Serbia and with the Serbian Orthodox Church.
For this reason, Pope Francis urged in 2015 that a mixed commission of representatives from the Catholic Church in Croatia and the Serbian Orthodox Church discuss Stepinac's role before, during and after WWII.
The most recent meeting of the commission was held in Novi Sad in Serbia on Tuesday, during which the topic of his relations with the NDH were discussed.
The next meeting of the commission will take place in the eastern Croatian town of Pozega in March when his relations with the Serbian Orthodox Church during WWII will be discussed.
Many Catholics in Croatia are passionate about Stepinac, seeing him as a heroic figure who resisted the imposition of Communism and remained with his flock in the country after the Communist takeover, when he could have fled abroad.
His supporters also note that attacks on him for his wartime role only really started after he publicly opposed the imposition of Communist rule in Yugoslavia.
After his trial before a Yugoslav court in 1946, Stepinac was jailed for 16 years for collaboration with the Ustasa and the Axis occupying forces. He was released into house arrest after five years and died in 1960 of thrombosis.
The verdict convicting him was quashed last July by the Zagreb county court, which ruled that Stepinac did not get a fair trial under the Yugoslav Communist regime.
The muted response to the Croatian town of Vukovar's decision to scrap controversial bilingual signs in Latin and Serb Cyrillic script suggests the EU has lost focus on minority rights, analysts claimed.
The trial of Zdravko Mustac and Josip Perkovic, former Yugoslav spy chiefs accused of killing a Croatian émigré, heard that the victim repeatedly told his German lover that he was living in fear.


text:
Nuaire. For the complete ventilation solution.
Nuaire Self-Seal is a simple and innovative product within the range of solutions for Ducting and Ancillaries to ensure the air leakage in and out of the duct joint is kept to an absolute minimum.
We will be in touch soon
This unique design removes the requirement for traditional taping and sealing of ducted joints and significantly reduces onsite installation time. The component provides a fit and forget solution and offers optimum performance compared to standard ducting fittings.
Features & Benefits:
Available in 125MM round pipe fitting and 204mm rectangular pipe fitting.
At Nuaire we understand the ever increasing challenges that our customers encounter when building new homes. Our complete ventilation solution supports you at every stage of your build.


text:
Black Tryst is a cool online dating site that will help you get to know some black folks for dating. If you try this site, let me, Paul, know what you liked or disliked about it by sending me an email, anytime!
Black Trystwww.blacktryst.com
- MyFavDating.sex
- Black Dating
- Black Tryst
- Tons of tools for communication prior to meeting
- Really delivers on what it says - tons of profiles of hot black chicks
- Not good if you're looking for something serious
Black Tryst is a great online dating site that is perfect for you if you are looking for something casual with black people. The thing that some people have found unlikeable about this site is simply that it's not good for those who are looking for a serious relationship. Otherwise, it provides everything you need to make successful sexy encounters with black folks.
My Full Black Tryst Review
Ebony is Hot
When you're looking for some loving with a black chick then you might not know where to turn. There are a lot of black dating sites out there, but which one has some babety girls? Black Tryst can help you out. It definitely has profiles of ebony girls who are looking for sex. In fact, the one thing that people haven't liked about this site is that if you're looking for a relationship that is on the serious side, it's probably not going to work because everyone on this site is so sex minded. There are already plenty of black women on the site, but there are more people getting on the site every single day so there is always something new for you to chat with and check out.
People Here Are Down to Meet Up!
This is a good choice for an online dating site whether you are planning on taking things offline or just keeping them to online chat. You can use all of the tools on the site to have some sexy chat or you can try to get someone's personal phone number and take your dating experience off of the Internet. Remember even if things are super-sexy, it's always good to keep your first meeting in public just for safety's sake. It's never a bad thing to get to know someone a tiny bit before you have sex with them anyways.
Nobody Here Wants You to Put a Ring On It
The chicks on this site don't want a ring, they don't want dates and they don't want you to buy them dinner or drinks. They just want to have sex so if you're looking for something with no strings attached with a black girl then this could be the perfect site for you. This is different than other dating sites because the girls on here are completely real and really want to date and have sex! It's easy to sign up for a free profile and the profiles are really simple. Some other dating sites make you fill out complex information that is really just their way of trying to get more data out of you. Black Tryst gets only the information they need to match you up with the perfect horny black girl!
Start Flirting Already!
Communicate with other members using the "flirt" feature, which is a great icebreaker. It's a way to send someone a message when you don't know exactly what to say. Think of it as a predetermined message so you don't have to do as much work to get the pussy you have been craving. You can also add a hot beauty as a favorite which is a good way to remember her if you are one of those people who are open to so many chicks that you might forget the ones you have been talking to. Try Black Tryst for a surely good time with black girls!


text:
Special Occasion Cars in Sussex and Kent
It may be a trip to a Brighton theatre to take in a play that has been on your "must see" list for months or it could be a concert to see a band you've been dying to catch.
Or you may be getting married and are planning the hen party of the year or your teenager could be hoping for a great start to his or her prom night to mark the end of their exams.
Perhaps it's a day at the races in Sussex, a night out at a special restaurant or just a way of saying thank you to someone who has gone the extra mile.
Whatever the occasion, there's nothing quite the same as making sure you get to and from your venue in style.
City Cars offers a variety of elegant saloons that will really make your special occasion even more memorable. Our highly professional team of chauffeurs guarantee to make sure you get to your special event, wherever it is, on time so you can really relax and make the most of your evening without having to worry about where to park or battling your way through traffic.
We love to talk to our customers and work out what is best for their night out. So if you are planning that special occasion, please make City Cars part of it.
Competitive rates, professional service and first class cars. We know how to make your occasion even better.


text:
Old PC man 17:54 08 Feb 2003
This one can be added to the very long list of systems that do this.
I suggest you send your daughter an e-mail with this web page address www. viaarena.com/?PageID=64 and get her to print the document and give it to the Mesh engineer when he turns up.
There has been a thread with the title "does your XP keep restarting your computer" running for a couple of weeks now. It may be worth you taking a look through.
Also take a look at this.click here and give thanks to Irishman for leading us to it.
Cheers John /#))


text: The Force Storm, Part Five (of Five) wiki last edited by ShadowSwordmaster on 12/29/13 08:28PM View full history On ancient Tython, the Force is out of balance, causing a massive Force storm which could destroy the entire planet! As three mighty Je'daii Masters strive to subdue the storm, three of their students struggle to forge an uneasy alliance with the dark stranger whose presence created the storm in the first place. As the first arc of the epic tale comes to a close, powers beyond even the Je'daii's understanding will be revealed, and a threat to the galaxy at large will be unveiled!


text:
Who knew Kanye was a Broadway fan?
You never know what to expect from Kanye West. He's currently in Hawaii recording for his upcoming album tentatively titled "Good A** Job." Today, footage of the rapper sampling a popular Wicked song, coincidentally titled "Popular," leaked online (it's been taken off the interwebs for now). Could we possibly be hearing a cover of Kristin Chenoweth gold on the radio sometime soon?
What do you think? Is this combination of rap and show tunes "wicked" (in the good, New England region, meaning of the word)? Or do you think the idea is just "Something Bad?"
Kanye West Could Become "Popular" Among Wicked Fans
Who knew Kanye was a Broadway fan?


text:
Choose Your Support
Customer Support
Questions about your order or about the website?
Choose from the following:
Call us at 1-877-816-2871
Local Support
Questions about a product or a service you see on our site?
Choose from the following:
Offer Valid: 7/1/17-9/30/17 (2+1) Buy 2 125ct boxes of SmartSelect™ Selective Polishing Paste, Get 1-125ct Assorted Box FREE Free goods subject to change. Please allow 68 weeks for free goods. To receive free goods email your Patterson invoice to firstname.lastname@example.com. Qualifying purchase(s) must be on one invoice. Not valid in conjunction with any other offer(s). Questions on free goods? Please call (866) 7534392 or email firstname.lastname@example.com. Free goods request(s) must be redeemed through Young Dental by submitting a copy of dealer invoice. Offers valid through quarter specified. Invoice MUST be received by last day of month after each specific quarter. Where applicable, indicate your choice of product on invoice. If no indication is made, available product will be sent at our discretion. Email is the preferred method of delivery, but you may also send a copy to: Young Promos, 2260 Wendt St, Algonquin, IL 60102 or FAX (888) 7717573
Promotion Expires: 12/31/2017
Auto Free Good
Advantage Promotion
My Catalog
My Catalog (Rep)
On Formulary
On Contract
Hazmat-No Charge
Additional Information
Schedule 2 and 2N
Schedule 3, 3N, 4 and 5
List 1
Prices in Green indicate discounted price.


text:
Alternative Fuel
Fuel other than petroleum diesel or gasoline.
Ammonia (NH3)
Serves as the catalyst in the SCR process, chemically converting the diesel exhaust fluid /exhaust stream mixture into nitrogen and water vapor before it exits the tailpipe.
CAFE Standard
Enacted by Congress in 1975, the Corporate Average Fuel Economy (CAFE) program mandates that all manufacturers marketing vehicles in the United States meet average fuel economy requirements for each class of vehicles they sell. Each manufacturer's fleet must average 27.5 mpg for cars and 20.7 mpg for light trucks. Manufacturers are subject to a fine of $55 per truck when the average fuel economy of their fleet falls short of the CAFE light-truck standard by one mile per gallon.
Carbon Dioxide (CO2)
Carbon dioxide is a colorless, odorless gas produced by, among other methods, the combustion of petroleum-based fuels in vehicles. Carbon dioxide, while necessary for all life, is also a "greenhouse gas" and in high concentrations contributes to increasing global temperatures.
Catalytic Oxidation
Process of oxidizing unburned hydrocarbons and carbon monoxide by means of a catalytic reaction to reduce pollution
A substance which influences the rate of a chemical reaction but is not one of the original reactants or final products, i.e. it is not consumed or altered in the reaction. Catalysts are used in many processes in the chemical and petroleum industries. Emission control catalysts are used to promote reactions that change exhaust pollutants from internal combustion engines into harmless substances.
Class 8 Trucks
Trucks or tractor-trailers with a manufacturer's listed gross vehicle weight of 33,000 pounds or more.
Diesel Engine
Any internal-combustion engine in which air is compressed to a sufficiently high temperature to ignite fuel injected into a cylinder. A diesel engine differs from other internal-combustion engines, such as gasoline engines, in that it employs no ignition devices, such as an electric spark. For this reason, it is often called a compression-ignition engine.
Diesel Exhaust Fluid
One of the key elements of the SCR process. It is a nontoxic solution of 67.5% purified water and 32.5% urea. When injected into hot exhaust as a fine mist and passed over a catalyst, Diesel Exhaust Fluid helps convert NOx into nitrogen gas and water vapor. Diesel Exhaust Fluid is stable, colorless and odorless, and meets accepted international standards for purity and composition.
Diesel Particulate Filter (DPF)
A device which physically captures diesel particulates preventing their discharge from the tailpipe. Collected particulates need to be removed from the filter, usually by continuous or periodic oxidation in a process called "regeneration."
Emissions Standards
Regulatory standards that govern the amount of given substance that can be discharged into the air.
Chemical reaction that absorbs heat.
Exhaust Aftertreatment
A method of controlling emissions from internal combustion engines – primarily NOx – by applying air pollution control techniques to engine exhaust.
Greenhouse Gas (GHG)
The gases present in the atmosphere which reduce the loss of heat into space and therefore contribute to global temperatures through the greenhouse effect.
Nitrogen Oxides (NOx)
Several air-polluting gases composed of nitrogen and oxygen which play an important role in the formation of photochemical smog. Nitrogen oxides are collectively referred to as "NOx", where "x" represents a changing proportion of oxygen to nitrogen. Internal combustion engines are significant contributors to the worldwide nitrogen oxide emissions. For the purpose of emission regulations, NOx is composed of colorless nitric oxide (NO), and the reddish-brown, very toxic and reactive nitrogen dioxide (NO2).
Original Equipment Manufacturer (OEM)
The industry's brand name manufacturers. The OEM definition in the automobile industry constitutes a federally-licensed entity required to warrant and/or guarantee their products.
On-Board Diagnostics (OBD)
A system on board of the vehicle that monitors emission control components and alerts the driver (e.g., by a dashboard light) if malfunctions or emission deterioration occurs. The OBD system involves a number of sensors and a data processor, which is typically integrated with the vehicle's electronic management system.
Particulate Matter (PM)
Particles formed by incomplete combustion of fuel. Compression ignition (diesel) engines generate significantly higher PM emissions than spark ignited engines. The particles are composed of elemental carbon, heavy hydrocarbons (SOF), and hydrated sulfuric acid ("sulfate particulates").
A process involving continuous or periodic oxidation that flushes and cleans a filtration medium.
Selective Catalytic Reduction (SCR)
SCR is one of the most cost-effective and fuel efficient emissions control technologies capable of reducing vehicle emissions to near-zero levels. SCR works by injecting a diesel exhaust fluid, usually automotive grade urea, in precisely timed intervals into the exhaust stream of the engine. The Diesel Exhaust Fluid/exhaust stream mixture is met with a catalyst that separates the exhaust into nitrogen and water vapor as it exits the tailpipe; two harmless compounds we breathe every day.


text:
We are a science-led global healthcare company with a special purpose: to help people do more, feel better, live longer.
Visit this section
We research, develop and manufacture innovative pharmaceutical medicines, vaccines and consumer healthcare products.
We research and develop a broad range of innovative products in three primary areas of Pharmaceuticals, Vaccines and Consumer Healthcare
Our consumer healthcare business has a portfolio of some of the world's most trusted and best selling products in Pain Relief, Oral health, Respiratory, Nutrition and Skin health
Explore GSK, the people behind it and a range of opportunities to be part of an organisation that helps people around the world to do more, feel better, live longer
Discover the work we do and the opportunities that exist, from research and development through to making our products available to those who need them
Information and tools for investors including share and dividend information, share price analysis, latest news and corporate reporting
Discover our latest financial performance announcement and search for historical quarterly results materials
Explore the latest news, different ways to stay in touch and available useful resources
Keep up to date with our news and recent announcements
Science can improve health and well-being in so many ways, from the development of everyday convenience healthcare products through to life-saving medicines and vaccines
Learn more about our R&D legacy involving the development of a broad range of innovative medicines, vaccines and consumer healthcare products
Responsible business is how we do business. Our mission is to improve the quality of human life by enabling people to do more, feel better, live longer
Our part in tackling global health challenges and our aim to extend the benefits of our products to more people


text:
By David Milliken and Peter Hobson
LONDON (Reuters) - Bank of England Deputy Governor Minouche Shafik played down the likelihood of a further cut in interest rates this year as the British central bank grapples with how aggressively to respond to a slowdown following June's Brexit vote.
The BoE said earlier this month that most of its officials still thought more stimulus was likely before the end of 2016 if the economy slows as it expects, although it acknowledged that the immediate hit from the vote to leave European Union had been less severe than the Bank had first feared.
But Shafik's comments on Wednesday could moderate expectations of a move at the Bank's next meeting in November.
"It seems likely to me that further monetary stimulus will be required at some point in order to help ensure that a slowdown in economic activity doesn't turn into something more pernicious," she said in a speech hosted by Bloomberg in London.
"However, the likely timing of that stimulus will depend on the continued evolution of the data over the coming weeks and months," she added. "Thus far, the welcome improvement in the forward-looking indicators suggests that the slowdown may not be as sharp or as sudden as we might have feared."
Sterling fell against the dollar after the comments from Shafik, who will leave the BoE in February. Most economists polled by Reuters expect the Bank to cut rates again this year.
Earlier, Governor Mark Carney told Scotland's Herald newspaper that in "broad brush" terms, the economy was still slowing as expected in the BoE's August forecasts, which see economic growth dropping to 0.7 percent next year.
Shafik said the Bank's response would hinge in part on how much new finance minister Philip Hammond eases back on planned spending cuts in a budget update which will only be announced after the central bank's November rate meeting.
"Fiscal policy will be an important part of the story," she said when asked about the chances of a rate move this year. "But if we have data prior to that which signals that stimulus is needed, we will obviously act on that," Shafik added, citing upcoming figures on third-quarter growth and the labor market.
Defending the BoE's first rate cut since 2009 last month, Shafik said it was better to act early. "I would much rather be on the front foot and act preemptively rather than be on the back foot and do too little too late," she said.
She also cautioned against taking too rosy a view of Britain's prospects after a rebound in some survey measures of business activity: "Forward-looking indicators, particularly things around investment intentions, still look quite worrying, so I don't think the uncertainty is anywhere near over."
"In August we expected quite a sharp, short, almost V-shaped shock for the UK economy, and I think the revisions will start to look a bit more U-shaped," she added.
(Additional reporting by William Schomberg; Editing by Catherine Evans)


text:
The West Campus hosted Palmerpalooza 2013, its annual summertime
recruitment event, in July. Approximately 150 guests,
including 70 prospective students, traveled to San Jose for an
event-filled weekend of social activities, interactive classroom
sessions, and informative presentations featuring West faculty,
staff, students, administrators and alumni.
Among the highlights of this year's program was the panel
presentation (above) featuring clinical-practice perspectives
from West Campus alumni Daniel Lord, D.C., '09, David Paris,
D.C., '01, Brant Pedersen, D.C., '04, and Lisa Stein, D.C., '88.
"Palmerpalooza is one of our biggest recruitment events of the year,
and, once again, I think we presented a program that provided
prospective students a multi-faceted experience demonstrating
why Palmer is the trusted leader in chiropractic education," says
Senior Director of Admissions Recruitment Julie Behn.
"A common theme expressed by our guests was an appreciation for
the enthusiasm and passion of the people they met throughout the
day," Ms. Behn adds. "I think that speaks to the passion and pride
shared by members of the West Campus."
Social Media Index
The Trusted Leader in Chiropractic Education ®
Copyright © 2017 Palmer College of Chiropractic. All Rights Reserved.
Website terms, conditions, disclaimers disclosures and Privacy Statement
Palmer College of Chiropractic is an Equal Opportunity Institution


text:
First arrest made under new law banning beef trade in Mumbai
In what is believed to be the first arrest made in Mumbai made under the new law banning beef trade, the Worli Police have arrested a 56-year-old man for allegedly selling beef illegally in BDD Chawl in South Mumbai.
The incident came to light when a political party took up the issue with Worli police, who then raided an apartment and nabbed the accused.
Representational Pic
The first case after the beef ban in Maharashtra was registered in March at Nasik where three accused were arrested for illegal cow slaughter. This is reportedly the second such case registered wherein beef was being sold illegally. The accused has been identified as Mohommad Akram Yasin Qureshi, who was selling the meat from a small house in BDD chawl.
The accused in his statement to police stated that he had bought the meat from Deonar slaughter house in the morning.
Police Inspector Jayendra Sawant from Worli police station said, "The accused was found with illegal possession of 29 kilograms of beef inside his residence, from where the accused sold it illegally. He claimed that he had bought it in the morning from Deonar. He even showed us the receipt."
"We are now trying to verify the meat. Whether it is of cow, buffalo or it's normal meat," added Sawant.
The arrested accused has been booked under sections 6 and 9 (A) of Maharashtra Animal Protection act along with sections 10, 114 under BMC Act and 115 and 117 of Maharashtra Police Act.


text:
|Home | About | Journals | Submit | Contact Us | Français
This paper describes the case mix, outcome and activity for admissions to intensive care units (ICUs) with community-acquired pneumonia (CAP).
We conducted a secondary analysis of a high quality clinical database, the Intensive Care National Audit & Research Centre (ICNARC) Case Mix Programme Database, of 301,871 admissions to 172 adult ICUs across England, Wales and Northern Ireland, 1995 to 2004. Cases of CAP were identified from pneumonia admissions excluding nosocomial pneumonias and the immuno-compromised. It was not possible to review data from the time of hospital admission; therefore, some patients who developed hospital-acquired/nosocomial pneumonia may have been included.
We identified 17,869 cases of CAP (5.9% of all ICU admissions). There was a 128% increase in admissions for CAP from 12.8 per unit to 29.2 per unit during the study period compared to only a 24% rise in total ICU admissions (p < 0.001). Eighty-five percent of admissions were from within the same hospital. Fifty-nine percent of cases were admitted to the ICU <2 days, 21.5% between 2 and 7 days, and 19.5% >7 days after hospital admission. Between 1995 and 1999 and 2000 and 2004 there was a rise in admissions from accident and emergency (14.8% to 16.8%; p < 0.001) and high dependency units (6.9% to 11.9%; p < 0.001) within the same hospital, those aged >74 (18.5 to 26.1%; p < 0.001), and mean APACHE II score (6.83 to 6.91; p < 0.001). There was a fall in past history of severe respiratory problems (8.7% to 6.4%; p < 0.001), renal replacement therapy (1.6% to 1.2%; p < 0.01), steroid treatment (3.4% to 2.8%; p < 0.05), sedation/paralysis (50.2% to 40.4%; p < 0.001), cardiopulmonary resuscitation prior to admission (7.5% to 5.5%; p < 0.001), and septic shock (7.3% to 6.6%; p < 0.001). ICU mortality was 34.9% and ultimate hospital mortality 49.4%. Mortality was 46.3% in those admitted to the ICU within 2 days of hospital admission rising to 50.4% in those admitted at 2 to 7 days and 57.6% in those admitted after 7 days following hospital admission.
CAP makes up a small, but important and rising, proportion of adult ICU admissions. Survival of over half of all cases vindicates the use of ICU facilities in CAP management. Nevertheless, overall mortality remains high, especially in those admitted later in their hospital stay.
Community-acquired pneumonia (CAP) is an increasingly common reason for hospital admission [1]. The majority of such admissions recover uneventfully with appropriate management. Between 5% and 10% [2] are more severely ill and are managed on an intensive care unit (ICU), up to 58% [3] of whom go on to die. There have been four UK studies of such patients [3-6]. While producing useful information, these studies are limited by their small size (each <100 cases), location in a single institution [3-5] and being out of date in relation to recent changes in clinical practice. Changes in population demographics, acute hospital admission practices, ICU admission practices (linked to the absence of an agreed definition of what constitutes severe CAP), and introduction of new interventions (for example, non-invasive ventilation) suggest that a further assessment of CAP on the ICU is worthwhile.
We have taken the opportunity provided by the Intensive Care National Audit & Research Centre (ICNARC) Case Mix Programme (CMP) Database to study nearly 18,000 such cases managed on 172 ICUs between 1995 and 2004 with details of case mix and outcome presented here.
The CMP is a national comparative audit of adult, general critical care units in England, Wales and Northern Ireland co-ordinated by ICNARC. Data were extracted for 301,871 admissions to 172 ICUs from the CMP Database, covering the period December 1995 to June 2004. Details of the data collection and validation have been reported previously [7].
The CMP has approval from the Patient Information Advisory Group to hold identifiable data without consent. Ethics committee approval was not required.
Information on the reason for admission to the CMP unit is recorded in the CMP Database using a standard coding method, the ICNARC Coding Method [8].
Admissions were selected as having probable pneumonia if the recording of the primary reason for admission, or ultimate primary reason for admission if the diagnosis changed after 24 hours, was any of 'Bacterial pneumonia', 'Viral pneumonia' or 'Pneumonia, no organism isolated' (Figure (Figure1).1). Admissions whose ultimate primary reason for admission to the CMP unit, based on information known after the first 24 hours in the CMP unit, was not one of the above (that is to say, the primary reason for admission was corrected) were excluded.
Community-acquired pneumonia is not coded separately, so cases were identified by the exclusion of pneumonia cases that were likely to be nosocomial or occurring in a previously immuno-compromised patient. Likely nosocomial pneumonia was defined as surgical admission or transfer from another ICU more than two days after the original hospital admission. Surgical admissions were identified as those with a reason for admission to the CMP unit that was surgical, or a source of admission or location immediately prior to source of admission of theatre and recovery in the hospital. Immuno-compromised patients were identified by the recording of any of the following in the past medical history: AIDS, radiotherapy, chemotherapy, metastatic disease, acute myelogenous leukaemia or acute lymphocytic leukaemia or multiple myeloma, chronic myelogenous leukaemia or chronic lymphocytic leukaemia, lymphoma, or congenital immunohumoral or cellular immune deficiency state.
Data were extracted on the case mix, outcome and activity as defined below. Microbiological and antibiotic management data are not included in the dataset and thus could not be analysed.
Age, gender, primary and secondary reasons for admission, cardiopulmonary resuscitation (CPR) within 24 hours prior to admission, and systolic blood pressure, heart rate, and mechanical ventilation at admission were extracted.
The following physiological variables were extracted from the first 24 hours in the CMP unit: sedated or paralysed and sedated for whole of first 24 hours, lowest total Glasgow Coma score if not sedated/paralysed, lowest arterial oxygen tension (PaO2), lowest pH and associated arterial carbon dioxide tension (PaCO2).
Acute severity was measured with the Acute Physiology and Chronic Health Evaluation (APACHE) II Acute Physiology Score (APS) and the APACHE II score [9]; the former encompasses a weighting for acute physiology (defined by derangement from the normal range for 12 physiological variables in the first 24 hours in the ICU); the latter additionally encompasses a weighting for age and for a past medical history of specified serious conditions.
Survival data were extracted at discharge from the CMP unit and at ultimate discharge from hospital.
Length of hospital stay prior to admission to the CMP unit was calculated from the dates of admission to the hospital and admission to the unit. Length of stay in the CMP unit was calculated in fraction of days from the dates and times of admission and discharge or death. Total length of stay in hospital was calculated in days from the dates of original admission and ultimate discharge or death.
Readmissions to the CMP unit within the same hospital stay were identified from the postcode, date of birth and sex, and confirmed by the participating units.
Case mix, outcome and activity were described for all admissions identified as probable CAP and separately for admissions during the periods 1995 to 1999 and 2000 to 2004.
The demographics and first 24 hour physiology were compared by length of stay in hospital prior to ICU admission in three groups: those admitted to ICU within two days of hospital admission, those admitted to ICU between two and seven days following hospital admission, and those admitted to ICU more than seven days after hospital admission.
The definition of CAP was fulfilled by 17,869 cases representing 5.9% of all ICU admissions during the study period. Of these, 10,271 (57.5%) cases were male with 2,941 (16.5%) aged less than 45 years and 4,257 (23.9%) aged 75 years or older (Table (Table1).1). Fifty-nine percent of cases were admitted to the ICU less than two days, 21.5% between two and seven days and 19.5% more than seven days after hospital admission. Eighty-five per cent of admissions were from within the same hospital, but a significant minority of 15% were transferred from another hospital. There were 1,080 cases (6.1%) only admitted after receiving cardiopulmonary resuscitation and 9,751 (54.6%) were mechanically ventilated at admission with 7,737 (43.3%) sedated or paralysed and sedated for the whole of the first 24 hours.
Past medical problems were infrequently recorded, with severe respiratory problems featuring most frequently, but in only 7.1%. It is likely that these figures are low due to the restricted definition of severe respiratory problems ('permanent shortness of breath with light activity, e.g., walking 20 m'), but under-recording cannot be excluded. Similarly, secondary reasons for ICU admission suggested low rates of chronic diseases, which may also be an underestimate of their true frequency. Of other secondary reasons for ICU admission, septic shock (6.8%) and pleural effusion (0.7%) are likely to have occurred as a complication of CAP, but others such as acute renal failure and myocardial infarction may have been a complication of or complicated by the CAP. All were infrequently recorded (<5%).
ICU mortality was 34.9% and ultimate hospital mortality 49.4% (Table (Table2).2). Of 8,609 total deaths, 2,381 (27.7%) occurred after leaving the ICU. Median (interquartile range) length of stay in the ICU was: survivors 6 (3 to 14) days; non-survivors 4 (1 to 10) days (Table (Table3).3). Median (interquartile range) length of stay in hospital was: survivors 30 (17 to 53) days; non-survivors 12 (4 to 26) days (Table (Table33).
As time from hospital admission lengthened, the case mix altered (Additional data file 1) with a rise in the proportion of older (>55 years) and male cases admitted. The proportion admitted from a ward, including high dependency units (HDUs), rose while those admitted from accident and emergency and those admitted after CPR fell. There were rises in the proportions with a past history of severe liver problems and chronic renal replacement therapy, but a fall in those with severe respiratory problems. Markers of respiratory failure in the non-ventilated became more frequent (higher respiratory rates and PaCO2) and abnormal serum sodium was more common. However, there was a reduction in the frequency of other abnormal biochemical variables, with abnormal pH, base excess, potassium, urea and creatinine being less common with time. Mortality was 46.3% in those admitted to the ICU <2 days after admission rising to 50.4% in those admitted at 2 to 7 days and 57.6 in those only admitted after 7 days in hospital (p < 0.001).
The total number of admissions per unit rose by 24% throughout the study period (Figure (Figure2);2); however, the number of CAP ICU admissions rose by 128% (Figure (Figure3)3) from 12.8 per unit in 1996 to 29.2 per unit in 2004 (p < 0.001). The proportion of admissions from other hospitals did not change, but admission from within the same hospital from HDU (6.9 to 11.9%) and accident and emergency (14.8 to 16.8%) rose (p < 0.001). Between the two periods there was a rise in those aged >74 years (18.5 to 26.1%; p < 0.001) and mean APACHE II score (6.83 to 6.91; p < 0.001), and a fall in past history of severe respiratory problems (8.7 to 6.4%; p < 0.001), renal replacement therapy (1.6 to 1.2%; p < 0.01), steroid treatment (3.4 to 2.8%; p < 0.05), those sedated and paralysed at admission (50.2 to 40.4%; p < 0.001), CPR prior to admission (7.5 to 5.5%; p < 0.001) and septic shock (7.3 to 6.6%; p < 0.001). Death rates rose slightly between the two periods (ICU mortality 33 to 35.7%, p < 0.001; hospital mortality 47.7 to 50.1%, p < 0.005). Median (interquartile range) hospital stay in survivors rose between the two periods (28 (16 to 51) to 31 (17 to 55); p < 0.001).
The main findings of this study, the largest study of CAP admissions to ICU published worldwide, are of the progressive rise in the number of admissions with time and the survival of half of all cases. While CAP accounts for only a small proportion of total ICU admissions, the rise in the number of CAP cases has been disproportionately large in comparison to the rise in ICU admissions overall. Illness severity, whether judged by admission gas exchange parameters, APACHE score, septic shock, length of hospital stay or mortality, does not appear to have altered sufficiently to explain this large increase, suggesting that the rise is either due to an increase in the total number of severely ill CAP cases presenting to hospital or the accommodation of a previously unmet need. There is evidence to support both explanations and it is likely that a combination of the two has contributed. In support of the former is the disproportionate rise in those aged >74 years (which could also be due to previous unmet need) and the increase in overall hospital admissions for CAP [1]. The previously unmet need theory is supported by the fall in the proportion that are paralysed and sedated on ICU arrival, perhaps reflecting a recent increase in the use of non-invasive ventilation (although this was not specifically recorded in the database). Statistically significant, but clinically small, rises/changes in admission source, steroid treatment, renal replacement therapy, severe respiratory problems, pneumonia type, and secondary admission reason are unlikely to explain these changes.
The overall ICU mortality of 35% is lower than the figures of 48%, 57% and 58% reported in previous UK studies [3-6], and is more similar to reports from New Zealand and other European countries, where seven studies have reported mortalities of 40% or less [10-16]. This apparent improvement in outcome may be due to improved clinical practice (supported perhaps by the fall in admissions only after CPR, which were 7, 13 and 25% in earlier studies [3,5,6]), but may also be due to differing admission policies and also the potential for bias in the previous UK studies. These studies were all small (maximum 62 cases) and two were based in single centres and may, therefore, not be representative of the UK as a whole. The much higher ultimate hospital mortality emphasises the value of this as a more relevant outcome measure. Nevertheless, that approximately 50% of patients survived to leave hospital vindicates the use of ICU resources in the management of patients with CAP. The much higher ultimate hospital mortality is not unique to CAP [7].
The small rise in mortality between the two periods of the study may be because sicker patients were admitted or because of the increase in the proportion of elderly cases in whom comorbidity may be more common and the decision not to augment treatment might be taken at a different threshold to that in younger, fitter individuals. Hospital care, especially identification of the severely ill patient, prior to ICU admission may affect outcome [17,18]. It is not possible to comprehensively assess pre-ICU care from this database. The fall in the prior CPR frequency and the low CPR frequency compared to previous UK studies [3,5,6], might suggest better pre-ICU management. The increase in the pre-ICU length of stay might, however, suggest delay in ICU referral. The admission of more than half of cases within 48 hours of hospital admission supports previous research suggesting that patients with severe illness are usually severely ill at hospital admission and it is at this time that alertness to severity of illness markers is especially important. In previous studies, the proportion of patients admitted early to the ICU has varied markedly. In a Spanish study [19], 80% reached the ICU within 24 hours of hospital admission, whereas in a British multicentre study [6], only 48% reached the ICU within this time. The latter figure is similar to the present case series and may reflect a difference in practice between countries, but may also be artefactually related to the small number in such previous studies. The higher mortality in those admitted later to the ICU is a cause for concern. The precise reasons for this cannot be identified from this database, but late referral could include failure to recognise markers of severe illness [17,18], high use of intermediate care facilities, high ICU bed occupancy limiting ICU access and shortage of ICU beds in the UK. Some late referrals may have had nosocomial pneumonia, which is discussed in detail later. A detailed analysis of factors, including existing severity scoring models, related to outcome is the subject of a separate paper in preparation.
The main weaknesses of this study are its retrospective design and, in particular, that patients with CAP were not separately identified. While it is not possible to be certain that some patients with other diagnoses (such as, nosocomial pneumonia, pneumonia in the immuno-compromised and non-pneumonic exacerbations of chronic obstructive pulmonary disease (COPD)) might not have been included, we believe that the steps taken to exclude such cases and the large size of the study population mean that any such cases that were mistakenly still included are unlikely to have significantly biased the study findings. In particular, the exclusion of postoperative cases and those with specified diagnoses linked with immunosuppression is likely to have minimised inclusion of nosocomial cases and pneumonia in the immuno-compromised; however, nosocomial pneumonia arising in patients admitted for another medical reason might still have been included, but such numbers are likely to be small. If such cases have been inadvertently included, this is most likely to have been in those only admitted to the ICU after seven days in hospital. That we have been successful in excluding even this group is suggested by the absence of any rise in the frequency of cases with secondary admission reasons of myocardial infarction, cardiac failure and COPD in those admitted to the ICU only after two days compared to the less than two days group. The high frequency of a raised PaCO2 is not in our opinion a sign of misclassification of patients with exacerbations of COPD. In a recent study, 21% of CAP patients had a PaCO2 of >45 mmHg at hospital admission [20], a proportion that is likely to be significantly higher in those admitted to the ICU. The low frequency of 'obstructive airways disease' as a secondary reason for ICU admission also suggests that large numbers of non-pneumonic COPD exacerbations have not been included.
CAP makes up a small, but significant and rising, proportion of adult ICU admissions. Survival of over half of all cases vindicates the use of ICU facilities in CAP management. Nevertheless, overall mortality remains high, especially in those admitted later in their hospital stay. More widespread use of supports to clinical severity assessment, whether disease specific – for example, Confusion, Urea nitrogen, Respiratory rate, Blood pressure, 65 years of age and older (CURB-65) [21] – or generic – for example, Early Warning Score (EWS) [22] – is to be encouraged.
• Community acquired pneumonia accounts for 6% of admissions to adult, general ICUs in the UK.
• Mortality was high, with 35% of admissions not surviving the ICU stay and 49% not surviving to leave hospital.
• Mortality increased with increasing time in hospital prior to ICU admission.
APACHE = Acute Physiology and Chronic Health Evaluation; APS = Acute Physiology Score; CAP = community-acquired pneumonia; CMP = Case Mix Programme; COPD = chronic obstructive pulmonary disease; CPR = cardiopulmonary resuscitation; HDU = high dependency unit; ICNARC = Intensive Care National Audit & Research Centre; ICU = intensive care unit.
The authors declare that they have no competing interests.
CW performed the analyses. MW, CW and DH drafted the manuscript. All authors contributed to the design and interpretation of the study and critical revision of the manuscript, and have read and approved the final manuscript.
Additional file 1 is a table providing a comparison of demographics and physiology for admissions with community acquired pneumonia by length of stay in hospital before admission to ICU.
from Specific conditions in UK critical care units: analyses from the Intensive Care National Audit & Research Centre Case Mix Programme Database.
This study was supported by ICNARC. The authors wish to thank everyone in the ICUs participating in the Case Mix Programme [23]. We acknowledge the Department of Health and the Welsh Health Common Services Authority for the initial, two-year, pump-priming funds in 1994 to establish ICNARC.
PubMed Central Canada is a service of the Canadian Institutes of Health Research (CIHR) working in partnership with the National Research Council's national science library in cooperation with the National Center for Biotechnology Information at the U.S. National Library of Medicine(NCBI/NLM). It includes content provided to the PubMed Central International archive by participating publishers.