Googling the lights fantastic

Thanks to data provided by Steve McIntyre and conversion skills provided by Barry Wise, we now have the first ever interactive global mapping tool for nightlight ratings and GISS stations worldwide that encompass USHCN and GHCN station locations.

How to download: right click, save target as, then save to your disk, double click it to open in Google Earth, (free download here) and follow the instructions below for turning on the city night lights layer in the text below. In the file, the icons are set as follows:

A = dark
B= dim
C= bright

Original post:

Yesterday I had a phone conversation with Steve Mosher. We were talking through some of the puzzlements of the GISS code and how it dealt with city nightlights, urbanization, the USA, and ROW. One of the big puzzles is why GISS uses counts of nightlights for an urbanization adjustment in the USA, but uses population data in the ROW. Why not keep everything on one method for a consistent result?

During the conversation, it occurred to me that there may be a tool available to us that will help us understand how nightlights look from space in relation to the placement of USHCN and GHCN weather stations, so we could check against the GISS lights database. My hunch paid off.

As luck would have it, Google recently partnered with NASA to include “city lights” data. With a little searching, I found I could add a layer to Google Earth (the free download program version, not the web browser version) and see what Imhoff must have seen, when he was researching his paper, except this is newer imagery. According to the source at NASA’s Earth Observatory web page, the visualization date of imagery is 10-23-2000

Here is what they say about it:

This image shows levels of light pollution across the globe. The brightest areas of the Earth are the most urbanized, but not necessarily the most populated—for example, compare western Europe with China and India.

The light pollution data used to create this image was measured by the Defense Meteorological Satellite Program (DMSP) Operational Linescan System (OLS). The OLS was originally built to view clouds by moonlight, but it is also used to map the locations of permanent lights on the Earth’s surface as shown here. This composite shows the data as processed by NOAA’s National Geophysical Data Center and artistically rendered over a false-color night version of NASA’s Blue Marble: Next Generation global map.

It turns out to be quite easy to enable this feature in Google Earth, but it is well hidden, and I only found it after doing a search of their support forums. The screencap below shows how to enable it by simply clicking on the NASA “Earth City Lights” checkbox in the layers control window by opening the “Features” folder and scrolling down.

Once I had this new layer working, it was quick work to add some USHCN stations locations and get some snapshots of what their GISS lights=”X” image actually looks like. Since we have been discussing several stations recently, I thought I’d try those first. Below are some screen captures of Lampasas, TX, Miles City, MT, Mount Shasta, CA, Cedarville, CA, and Moncton, N.B.

I used the GPS coordinates from surveys done by volunteers from surfacestations.org to position the USHCN station markers, and the NCDC MMS database published coordinate for the Moncton N.B. station.

Each image has the NASA GISS lights information added in the lower left corner.

As Steve Mosher pointed out, Moncton, N.B. is in an odd category called “urban dark”. However the picture above certainly doesn’t look “dark” though it does look “urban”. Otherwise there is good agreement between the other GISS lights ratings and the current photos.

I’m still a bit puzzled as to why any adjustments would be needed at all for Cedarville, since it is about as rural as rural can be, “dark”, and lights=0 and that data agrees with the DMSP nightlights image above. Here is the GISS raw versus GISS homogenized data plotted below. Some data is missing for 1915 and 1957 but the station continuity is good otherwise. Click for a larger graph.

Other lights =0 stations, such as Cheesman Lake, CO don’t have adjustments applied at all. It is important to determine what the code criteria is for determining why some lights=0 stations would be adjusted and others would not. There may be a valid method that I and others looking at this don’t yet understand, or it may be an oversight, data error, or bug in the code. Perhaps some of the folks from GISS that have been watching this discussion unfold can shed some “light” on the subject?

Having this Google Earth tool now will allow anyone to check the representivity of a lights=”x” and dark/dim/bright data field value in the GISS dataset. To make such work easier, I hope to update the Google Earth KML file (created by Barry Wise) of all USHCN stations with newer surveyed GPS coordinates, though given the coarseness (2.7 kilometer pixels) of the DMSP imagery, such accuracy may not be needed. Try it out and let me know.

If Steve McIntyre can assist in providing coordinates for other GISS stations for the USA and perhaps the ROW, we could add them to the KML file as well so that it becomes an easy matter to check any station. I hope to include these photos in the surfacestations.org database along with other survey photos since they are key to the GISS adjustment for each station.

Later, if such nightlight data would be useful for further study, perhaps an automated process could be created that would use Google Earth to display centered views for automated export, and the images could be run through an automated pixel histogram process to determine the number of bright/dim/dark pixels in a radius around the station for all ROW stations.

101 Comments

1. If the observation platform is not directly above the light there will an offset if the light is elevated, e.g. a mountain town viewed obliquely. I do not know which corrections if any have been applied to counter this effect. The effect might be negligible depending on actual geometries.
2. A lot of the matching depends on principal pixel size. One would need to know whether there was one set of pixels used for each region or if it was the composite of numerous passes – for both the position of the town and its lights; and one would need to know how multiple passes were mathematically combined.

In the early days the Russians avoided pixels and radio transmission, using film because it had better resolution. I do not know if film is still used for special mapping exercises by anyone.

The examples you give look as if a fair deal of massaging has occurred. Miles City, for example, looks as if a blurring filter has been used to remove jaggies. (Such filters can be dangerous). I can make no comment yet on whether this is good or bad or true.

I cannot follow the adjustment logic, as illustrated by your example of Cedarville. Maybe the data from ROW is sometimes better because it has NOT been adjusted out of reality.

Is there an easy way to determine the amount of “signal” provided by each of the 1221 USHCN data sets to pareto which stations should be examined in more detail initially?

There are many unsurveyed stations yet. The “easy” ones (based on volunteer location/travels, ease of finding and access issues) have been visited. A prioritized wish list for the remaing ones would be valuable to provide incentive to visit some of the “harder” locations.

It might be interesting to do some more urban temperature transects (ala AW’s car based rig) to see how much UHI is associated with locations on the urban periphery, especially associated with the prevailing wind direction. So it would be interesting to do the transects at different times of day in different wind conditions.

Having been to Miles City a couple of times, it’s difficult to believe that the town’s presence has much effect on the Airport’s temperatures. The airport is not typically downwind from town and being located above the valley floor positioning of town, quite a different microclimate.

If it has become urban, the adjustments should logically be the inverse of what we see. 1925 temperatures are currently adjusted downwards by about one degree, but the downwards adjustments should be applied at the recent end of the graph.

Whenever I collate data or metadata, I call the files *.info.dat or details.dat and have a number of such files in ascii format in http://data.climateaudit.org/data . Some of these collations take a lot of fiddly work e.g. a concordance of GHCN and USHCN numbers something which did not exist online anywhere at the outset of this.

re: brightness index
Here is the single reference to a use that I find. In routine padjust.f in STEP2:
if(name(31:32).eq.’ R’.or.name(31:31).eq.’1′) then
Calt if(name(33:33).eq.’A’) then
It’s commented out and the Calt suggest that it is an alternative to ‘ R’ and ‘1’, and ‘A’ means Dark.

All of the lights in my neighborhood are shaded so that little to no stray light goes skyward. At a collection of auto dealers in town it seems that all the very bright lights have no shades on top and send as much light up as down onto the cars. Will this make any difference to how bright a town seems from space? The more I read about how temperature data is massaged for various factors such as brightness and UHI effect the less I have any faith in pronouncements about temperature trends thru time.

ok, sorry. The lights= xxx is actually referred to as brightness-index in the GISS/NASA v2.inv file. It starts in column 83 of the GISS/NASA v2.inv file. Let’s call this the GISS/NASA brightness.

The A/B/C = Dark/Dim/Bright is included in the NCDC/NOAA file. And maybe we should call this the NCDC/NOAA brightness.

And we have the GISS/NASA N=1/2/3 (satellite light data 1995, only near cont. US)

Apparently there are at least three ways to estimate brightness/lights.

I agree with steven and I can’t find, so far, any reference to or use of the GISS/NASA brightness. That brings up an interesting issue. All that hi-tech work with hi-tech images seems to have not been worthwhile. But maybe we’ll find it used somewhere.

So what might be of interest is to look at the relationship between the NCDC/NOAA and GISS/NASA brightness. And between the GISS/NASA brightness and the GISS/NASA n=1/2/3.

Each DMSP satellite has a 101 minute, sun-synchronous near-polar orbit at an altitude of 830km above the surface of the earth. The visible and infrared sensors (OLS) collect images across a 3000km swath, providing global coverage twice per day. The combination of day/night and dawn/dusk satellites allows monitoring of global information such as clouds every 6 hours. The microwave imager (MI) and sounders (T1, T2) cover one half the width of the visible and infrared swath. These instruments cover polar regions at least twice and the equatorial region once per day. The space environment sensors (J4, M, IES) record along-track plasma densities, velocities, composition and drifts.

To get the image you see above, assembly required of cloudless images.

I’m of a strong opinion that there are no angle issues related to imager lookdown at the surface. Looking at my own hometown and comparing lights and road landmarks it fortifies that opinion.

RE8 Steve Mc,
By converting to a grey scale image and then running a pixel under point query I can get a pixel value in terms of brightness B 0-255, here they are
725830060 CEDARVILLE B=34
725920030 MOUNT SHASTA B=100
742300020 MILES CITY FCWOS B=100
717050000 MONCTON,N.B. B=190
722570030 LAMPASAS B=228

This process can easily be automated, I use a graphical scripting language to make all sorts of satellite and radar images every day at my office for use in radio/tv/newspaper like this loop I do for my local radio station

Caveat: This was done on the image Google uses overlaid on NASA blue marble background, there may be a pixel bias associated with that. I am investigating to see if that background layer can be removed and we just get lights only.

Possibly, though this image appears to have been gathered around year 2000 according to the description, so duplication may not be exact. What I don’t know for certain is whether that note refers to the date of satellite passes or the date of compositing of earlier data into the image used by Google.

I expect NASA would be able confirm the age or series of ages for the lights images, but I would venture that the 2000 reported date is a compilation date rather than an image date. This is fairly common for satellite and other image products that cover large areas.

Maybe here’s an idea. While we can’t run the code with the alternative criterion in routine padjust.f mentioned #12 above, we can check the difference in the selected stations.

Using the v2.inv file as used in STEP2, parse it and compile the station list using the ‘ R’ .or. ‘1’ criterion and then parse it using the ‘A’ criterion and compile a second list. The file to be parsed can also be the station_list.txt file. This gives a rough zeroth-order look at what affect the different criterion might make.

My impression is the Steve McIntyre can do the parsing in two heartbeats.

RE 25. SteveMC posted a .dat file a while back that had GHCN brighness and brightness index
BUT not the nighlights (1/2/3) by comparing THAT file to the Station_list.txt I was able to
discover the Nighlights feild.

Amen. Big cities are going to have UHI and well-lit. Smaller towns may still have significant UHI and may or may not be well-lit, depending upon whether the locals decided to pay for street lights.

I wonder if “light pollution” laws could be influencing data, as well. Some large cities in Southern Calif. switched to amber streetlamps away from mercury-vapor to reduce light pollution at Mt. Wilson Observatory. Some other smaller western communities may also have light pollution laws so the residents can see the pretty stars at night. The residents in these communities would still be generating heat, of course.

It makes me wonder what the purpose of the “Nightlights” satellite is and what it’s looking for; all light, or bluish light ?

Isn’t the issue with UHI not that it exists but that it is changing? Therefore, the issue is not whether something is lit or unlit but is the degree of “lighting” changing, particular around the actual weather station. Hence there is a need to understand how lighting is measured and whether change can be detected.

RE 29. JohnV Dark doesnt get you RURAL. DARK gets you dark. There are towns that are small
and dark, and cities like Brantford ON ( population 90,000) which are DARK.
Hansen believes that DARK means NO UHI.

DARK means DARK: BRIGHT means BRIGHT

“The DMSP, on
the other hand, registers only light. All other characteristics-
demographic or otherwise–can only be inferred.
In fact, compared with the Census, the DMSP urban area
is underestimated most dramatically in the northeastern
United States, or New England (Connecticut, Maine,
Massachusetts, New Hampshire, Rhode Island, Vermont).
Many small towns and suburbs in this region were eliminated
from the DMSP/OLS assessment during thresholding.
Many of these towns are counted in the Census.
….
In the western United States, the DMSP/OLS approach
gave larger urban-area estimates than did the
1990 US Census. At this point in time. it is difficult to
differentiate between two probable causes for this discrepancy
(1) the DMSP/OLS may be accurately delineating
urban growth and is “correct”; (2) our DMSP appreach
is indeed overestimating by picking up too much
rural infrastructure and classifying it as urban area.”

Nighltights do not cause UHI. Buildings, pavement and people do.
Nightlights is a PROXY of buildings pavement of people.

Good question, Bender. And there’s more than enough info floating around. The GISS/NASA P=R/S/U flag and the GISS/NASA brightness-index=0-xxx info along with GHCN B=A/B/C and GISS/NASA N=1/2/3 could be checked against population.

Anthony, is the accuracy of station location info and the place locations on Google Earth good enough to get accurate info?

In his comment here, conard noted that the Crawfordsville, IN station was one of those rare, small city stations with dark night lights (S1A). Last summer I had loaded the photos from the Indiana State Climate Office for all sites they surveyed in that state to surfacestations.org, so I went back to look at them and sure, enough, it looks like the station is located on a farm.

Because this station’s record is used in the homogeneity adjustment to adjust nearby urban and small city stations, I wondered if it had always been located on the farm. Not surprisingly, I discovered it was located for over 100 years inside the Crawfordsville city limits, and has only spent the last 15 years on the farm. Using today’s night lights, here is what the historical station positions look like:

Granted, the night lights did not look like that back in 1920 or even 1950, but back then the station was in the heart of Crawfordsville, spending most of the time next to the parking lot between the power plant and the river. The rest of the time it was very near Wabash College (according to the GPS coordinates), and because the coordinates are precise only to a few tenths of a mile, I would not be surprised to learn the station was located on campus. The station went through a number of curators, several of whom are listed as “Prof”.

#32 steven mosher:
#34 Anthony Watts:
Gotcha.
I realized that dark-vs-bright would not help with microsite issues, but it seemed like a good proxy for UHI. There are exceptions (such as the unincorporated towns mentioned by AW) but in general it seems reasonable that brightness should correspond with UHI, *perhaps* better than does population.

As an extreme example, in Northern Alberta there is a municipality called Wood Buffalo with a population of ~70000. This would classify it as urban. However, it has an area of over 63000km^2 (!) so most of it is dark and very rural. In this case, night lights would be a better determinant of UHI than would population.

It may be my bias from growing up in remote locations, but I think brightness is likely a better proxy for urbanity than is population.

I spent quite a bit of time in Crawfordsville one year setting up a weather radar system for the local civil defense there. And I’ve been to Wabash College, your analysis matches my memory of the downtown and campus area.

Clearly, city nightlights doesn’t fully tell the story of this station. Like some early criticisms of photos taken in the surfacestations.org project, “nightlights” only tells the photo story for a moment in time. Thus I don’t think “nightlights” is a good metric by which to adjust an entire station history with. Better station histories are needed to do a valid adjustment for the entire station history, and that is difficult to come by.

IMHO it is better to identify the best stations so that complex adjustments are not required. More data points don’t neccessarily mean “better” data especially if the adjustments are based on a single point in time but the station has moved in time.

The scientific question is this. How well does nighlights correlate with population, and then
how well does population correlate with UHI.

So, if you want to screen for the best stations one could argue this:

1. Population is Rural AND
2. Nighlights is Dark AND
3. No building over 10 meters ( from ushcn station file)AND
4. No impervious surfaces ( nasa satillite product) AND
5. High vegataive index ( gallos study) AND
6. No micro site issue.

The scientific question is this. How well does nighlights correlate with population, and then
how well does population correlate with UHI.

So, if you want to screen for the best stations one could argue this:

1. Population is Rural AND
2. Nighlights is Dark AND
3. No building over 10 meters ( from ushcn station file)AND
4. No impervious surfaces ( nasa satillite product) AND
5. High vegataive index ( gallos study) AND
6. No micro site issue.

“Graph showing the relation between urban-area
measurements calculated from the raw DMSP/OLS data, the
thresholded DMSP/OLS data, and the U.S. Census Bureau
urban areas for the 48 states of the conterminous United
States (48 states plus Washington, DC, for a sample size of
49). Census Bureau numbers represent total urban area for
each state. Raw DMSP is urban area calculated as all grid
cells lit lOO% of the time. Thresholded DMSP represents
urban area only as grid cells illuminated 89-100% of the
time. A t-test indicated no significant difference between the
thresholded DMSP and the Census estimates.”

In other WORDS, the BRIGHTLINE the line between Dim (8-88) and Bright (89-100)
IS NO DIFFERENT than population data.

get it? Nightlights is as GOOD as Census data. But GISS already have census data in
feild 32. Rural/Smalltown/Urban. In fact USHCN supply US census data for the sites
for every decade since 1900.

So what IMHOFF97 proved was this: nightlights is as good as census data in dtermining
URBANITY.

re 26. Nighlights was not the design mission of the satillite or its sensor. The passes where made
by a military satillite to characterize the light reflections (like from the moon) on the tops
of clouds. I will leave it to your imaginations to figure out why.

On cloudless nights these very sensitive sensors could see your porch lights. just kidding
I’ll see if I can find the sensor specs, it wasnt design to do this task, so some magic was applied.

I used the column out of Steve’s file that has a heading of “lights”. It has codes of A,B, or C. the only other column that seems appropriate is one whose heading is “urban” which has codes of U,S,and R. Can you look at that file: http://data.climateaudit.org/data/giss/giss.info.dat and tell me which data you’d like to see? Is there another data source we should be using?

Don’t know if this helps any, but it appears as though northern California was to cold in the past. Page 7 Hansen 2001,

The strong cooling that exists in the unlit station data in the northern California region is not found in either the periurban or urban stations either with or without any of the adjustments. Ocean temperature data for the same period, illustrated below, has strong warming along the entire West Coast of the United States. This suggests the possibility of a flaw in the unlit station data for that small region. After examination of all of the stations in this region, five of the USHCN station records were altered in the GISS analysis because of nhomogeneities with neighboring stations (data prior to 1927 for Lake Spaulding, data prior to 1929 for Orleans, data prior to 1911 for Electra Ph, data prior of 1906 for Willows 6W, and all data for Crater Lake NPS HQ were omitted), so these apparent data flaws would not be transmitted to adjusted periurban and urban stations. If these adjustments were not made, the 100-year temperature change in the United States would be reduced by 0.01°C.

A satellite at 830 km height with a swath of 1500 km would certainly require geometric correction for curvature and terrain altitude, would it not?

I would suggest caution in trying to digitise brightness from these maps. It looks almost as if a set of digital images from little to big was generated to be overlaid on the location of the town. To split the image into factors such as brightness or hue might be quite pointless, if they are synthetic/representational. Also, one does not know if a large city saturates the sensor and if a false colour is added to make the light look more natural. The edges of the images you present have a marked appearacnce of post-collection processed digital blurring and I would not be bothered to work on them until I had comprehensive details of how they were collected and massaged. (e.g. the light pattern does not follow the highways you show as you leave town; also, the appearance of pixel size on the edges of you images is very small, so one would expect much more detail within the lit area unless it is an average of many passes, each with a positional error of the order of, or larger than a pixel).

If lights images were overlaid on towns using a town map position program, then it might also be pointless to correlate position back to a map program. It might be the same map, giving a beaut correlation by cyclic reasoning.

I feel frustrated that I no longer have good access to data or manipulation and that my comments are often negative. I can see problems more readily than I can fix them. Please don’t take this as lack of appreciation of your efforts.

A satellite at 830 km height with a swath of 1500 km would certainly require geometric correction for curvature and terrain altitude, would it not?

Yes for data at the edges, but in the center, it would not require much. It depends on the width of the data when it is applied to the whole collation.

But also remember that this data is mapped back on to the globe in Google Earth, so the eye-view we get is not distorted such as it might be in something like a Mercator projection. Remember you can zoom way out and see this data in globe form or zoom far in and see it presented with a “flat” appearance.

I’ve looked at lights of cities and towns that I’m hands on familiar with, and so far I haven’t seen any mismatches that give me pause, great or small. I don’t have all the answers, but I do have a good feel for how well the data is presented.

Try it “down under” with the new KML file, look at some of yours, turn the overlay on/off from daylight to nightlight and see if the light matches the city limits well or not. Yes, not a perfect test, but it is a good first one to try.

I agree that nightlights doesn’t solve the UHI problem, but the google program could still prove useful because:

a) Nightlights combined with the various other interesting satellite products that Steven Mosher has described above would make an excellent first cut of the stations based on macrosite quality to find those that have the potential to be good stations. If they don’t pass this test then it’s probably not worth expending too much effort investigating microsite issues and station history. For those that do make the cut, surveying them could be given the highest priority.

b) It would be interesting to compare the results of extracting nightlights from Google Earth with the myriad of brightness indices and categories that NOAA and NASA have devised. Some of these are clearly contradictory (Moncton is both bright (NOAA) and dark (NASA)!), so one organization must be wrong. Who needs to fix their algorithm?

c) If the correlation between nightlights and population density turns out to be good, this could be exploited to do spot tests that check the real “ruralness” of stations in other parts of the world where population data is unavailable to us, potentially inaccurate or too coarse. I suspect the relationship would have to be recalibrated though for developing countries where electricity usage per capita is quite different. In any event, I think it’s important not to lose sight of the rest of the world, which is the source of most of the 20th century global temperature anomaly. Unfortunately these stations are suspected to be poorer and in many regions satellite data may be about the only good info on the station we’ll ever have to use.

They look at multiple measures. the LULC the land use land chaarateristcs data.
If you look at the UHSCN data every site has feilds characterizing the land use
( there is also data on building height) in addition gallo easterling look at nightlights
and at population. I’ll finish the paper tommorrow but the interesting thing that they did
was this: UHI urban heat island manefests itself by elevated nighttime temps.

Concrete stores heat and then when the sun goes down the heat gets released, and if
you have buildings blocking wind, you get an urban heat bubble.

Anyay, gallo et all actually see if there Proxies for UHI are any good. I’ll report more later

Thanks, Anthony. I’ll try your suggestions later tonight. First, I’d like to play with your Miles City image, knowing it has been through JPG compression and possibly rescaling to fit the CA page. It is not ideal to work with but it shows some complexities.

It is in the Red-Green-Blue format, RGB. It can be split into these 3 colours and then recombined. There are other formats which can be recombined to make the image after splitting. Here I use HSB for Hue-Saturation-Brightness, which Google defines. Your starting image is below, with a few words added for scale.

Next are Hue, Saturation and Brightness splits in 8-bit greyscale –

Let’s work up from the bottom, brightness. The image looks rather like the original coloured version and does not tell us much.

Next, saturation. This image is full of intrigue. First is the diameter of the roughly circular halo, which almost reaches the map frame. It’s much larger than the “lights” size on the colour map and it therefore suggests caution in saying if a station is lit or unlit on the edge of town, like the USHCN site. Another odd feature is the blocks of colour some tens of pixels in size, as if the image was placed on a textured background, perhaps to give a more natural look. The consequence is that the distribution of yellow might look uniform in places but might have a mosaic under it that makes it hard to be numerically extractive. Third, there is a strange square feature some 100 pixels wide, purpose unknown. Is it a marker used to combine maps from different sources? I know not. The said diffuse border looks artificial, as it does not pick up expected features such as the marked major highways leading from town. It looks like a smeared combination of many maps whose pixels do not coincide when overlaid and might even be of different pixel size.

Finally, Hue. This is a strange image indeed. The black blob in the middle shows none of the diffuse yellow border of the coloured lights map. It even seems to have escaped the obligatory anti-alias technique.

I won’t stretch interpretations further. I’d prefer to seed some discussion as to whether is is possible, desirable or sensible to try to measure a number related to light from an image such as this. I think it would be full of danger. I suspect that at least part of ther colour is synthetic rather than scientific, a cartoon-type representation if you will, rather than a scientific image.

Please use caution trying to correlate factors like population with numbers derived from images like these, especially if you do not have an accurate description of how the images were composed. Danger abounds.

With regards to my post 48, could this not be the reason for the adjustment to Cedarville, Ca. It would be interesting to see how many unlit stations outside of N California were adjusted. As to the accuracy of Google earth nightlights, I would ask anyone to compare and contrast Google earth to the figure on page 16 of Hansen 2001 and tell me which data you would rather use. Lastly, a couple of people wondered why lights are used in US and not ROW. To be fair in the discussion of Hansen 2001, he clearly suggests two improvements to the GISS analysis. First, he would like to break the periurban into two classifications, and secondly, to have the satellite light information for the rest of the world. It seems to me with the release of the old code and partnership with google that we should all expect Hansen 2008 soon, with all new code and bigger and better adjustments.
Steves’ and Anthony, in the interest of full disclosure, should you not tell us which captain of industry is behind this charade?🙂

Ellis:
Re: your point in #48. I am not sure that adjusting for localized climate effects is entirely kosher. It seems like extreme data manipulation, like adjusting outliers. Either you have a generalized variable or you don’t. TOBS make sense. UHI/population/lit makes sense. Adjusting stations because the trend does match the local SST does not make sense.
So if Hansen feels that it was OK to make these adjustments, are there other ad hoc adjustments?

Finally, I am not sure what point you are making in #59. Was it a joke?

I think you’re looking at the result of a series of transparent overlays. Here’s the information tag from Google Earth, note the use of “artistically”.

The light pollution data used to create this image was measured by the Defense Meteorological Satellite Program (DMSP) Operational Linescan System (OLS). The OLS was originally built to view clouds by moonlight, but it is also used to map the locations of permanent lights on the Earth’s surface as shown here. This composite shows the data as processed by NOAA’s National Geophysical Data Center and artistically rendered over a false-color night version of NASA’s Blue Marble: Next Generation global map.

It looks like nitelites sensor was tuned to near IR, which corresponds better to heat
than light. That could explain the reason why you don’t see highway strips like you
think you should if the original data were purely visible light. I agree with your
supposition that you’re looking at false colour overlaying that was processed from
a monochromatic sensor.

CCD sensors are overly sensitive to near IR; when I was doing measurement instruments
in the past with these you always had to (try to) account for this. The response curve
of a CCD in a camera (e.g.) is optimized for what, 550 nm or so, whereas the ones in the
sat may have been tuned more to say the 680 – 720 range (I think IR LEDs are tuned to
about 880 nm.)

Given Mr. Mosher’s contention that the original mission was cloud reflection then it would
make sense that the sensor is IR sensitive (seeing through clouds…)

I don’t know if this helps, but I think the question to ask about the data gathering is
what the response curve of the sensor looked like. This ought to help evaluate what you
are seeing.

Bernie,
The joke is that a scientist would stoop to such vitriol. The man is now clearly a politician and I have no problem with that, the problems start when he wants us to believe he is an objective scientist.

Also, page 19 Hansen 2001, if the 1900-1999 temperature trend in the US is .32 and the total adjustments in the US for 1900-1999 is .30, and you were in charge, would you not expect people to question your methods?

The prototype DMSP/OLS “city lights” image data set
that we worked with in this analysis was prepared by
NOAA/NGDC. The OLS sensor is sensitive to very low
intensity visible and near-infrared (VNIR) light sources,
including, but not limited to, city lights, lightning, moonlit
clouds, fires, and other bright surfaces. The sensor
was originally designed to allow the observation of nighttime
cloud cover for meteorological forecasting.

Imhoff et al. (1997a) and Imhoff et al. (1997b) have
successfully used Defense Meteorological Satellite Program
Operational Linescan System (OLS) data (Elvidge
et al. 1997a) acquired at night to identify ‘‘urban areas.’
The OLS-based estimates of ‘‘urban’ area were within
5% of the area defined by the 1990 U.S. census for the
conterminous United States (Imhoff et al. 1997b). The
OLS radiometer includes two spectral bands. The visible
band ranges from 0.5 to 0.9 mm. The sensitivity of the
radiometer to light at night is four orders of magnitude
greater than that of the National Oceanic and Atmospheric
Administration (NOAA) Advanced Very High Resolution Radiometer (AVHRR) or Landsat-TM (Elvidge
et al. 1997a).

Other studies such as GALLO ( google LULC UHI or LULC Gallo) used: A snippet

The daily scenes were composited on a biweekly basis
such that the data value from the scene that exhibited
the largest value of the normalized difference vegetation
index,
…was retained for each grid cell in the composite product.
The visible (0.58–0.68 mm), near infrared (IR) (0.72–
1.1 mm), and thermal IR associated with the maximum
NDVI value were also retained in the composite product.
Calibrated thermal IR data of channels 4 (T4; 10.3–
11.3 mm) and 5 (T5; 11.5–12.5 mm) on the National
Oceanic and Atmospheric Administration (NOAA)
AVHRR were used to estimate apparent surface temperature
(Tsfc , 8C) as
Tsfc 5 T4 1 3.3(T4 2 T5), (2)
for a surface emissivity of 1.0, as described by Price
(1990). The emissivity values associated with the urban
and rural AVHRR samples of T4 and T5 data were not
available. Oke (1987) cited emissivity values for vegetation
(agricultural crops, deciduous and coniferous
forests) that ranged from 0.90 to 0.97, while values for
urban materials (concrete, asphalt, and stone) ranged
from 0.71 to 0.95. Roth et al. (1989) estimated that the
suppressed differences in Tsfc , due to unaccounted differences
in emissivity, could be as much as 1.58C. The
visible, near-IR, and thermal IR data were not adjusted
for atmospheric properties because relative, rather than
absolute, differences in the vegetation index and Tsfc
were computed.”

I think the best study I’ve seen is Gallos study, where he combines VEGATATIVE INDEX
with NIGHTLIGHTS and POPULATION to look at the difference in trend between Urban and Rural.

So some smart grad student could head off and do some very interesting work, but for now
we audit the past. By the Time H2001 was being done, you had Easterling, Gallo and Owen
Apparently suggesting an approach that combined all of our best knowledge ( population,
land use and land cover, nighlights) to identify Rural verus Urban and H2001 settle on an
approach that is less accurate. Maybe Gallos work wasnt published at the time?

The OLS instrument consists of two telescopes and a photo multiplier tube (PMT). The visible telescope is sensitive to radiation from 0.40 – 1.10 um (0.58 – 0.91 um FWHM) and 10-3 – 10-5 Watts per cm2 per sterradian. The infrared telescope is sensitive to radiation from 10.0 – 13.4 um (10.3 – 12.9 um FWHM) and 190 to 310 Kelvins. The PMT is sensitive to radiation from 0.47 – 0.95 um (0.51 – 0.86 um FWHM) at 10-5 – 10-9 Watts per cm2 per sterradian. The detectors sweep back and forth in a “whisk broom” or pendulum-type motion. The continuous analog signal is sampled at a constant rate so the Earth-located centers of each pixel are roughly equidistant, i.e., 0.5 km apart. 7,325 pixels are digitized across the 1080 swath from limb to limb. The instruments are built by Westinghouse Corporation. DMSP satellites are in a sun-synchronous, low altitude polar orbit.

NGDC serves as the long-term archive for DMSP data and has data holding extending from 1992 to
the present. In the standard collection mode the OLS gain is turned up quite high for the detection of
moonlit clouds. Under these operating conditions the nighttime visible data are saturated (DN=63) in
urban centers. In addition, during the operational collections the system gain can vary, modified by
on-board algorithms. It is possible to turn the gain down to avoid saturation in urban centers, but then
the dimmer lights are not detected due to the limited dynamic range of the sensor. To overcome these
limitations, NGDC requested the data collection in three overlapping fixed gain settings (low, medium
and high) on alternating 24-hour periods. The low gain collections provide unsaturated data of bright
urban cores. The high gain setting data provide detection of dim lights – but has saturation in the
bright urban cores. The medium gain setting covers the gap between the detections achieved with the
high and low gain settings. The digital numbers from the low and medium gain settings were
converted to the radiance units of the high gain data based on the sensor’s preflight calibration. Each
orbit was processed with automatic algorithms that identify image features (such as lights and clouds)
and the quality of the nighttime data. The basic algorithms have been described in references
17,18,19. The following criteria identified the best nighttime lights data from the set of fixed gain
collections:
1. Center half of orbital swath (best geolocation and sharpest features).
2. No sunlight present.
3. No moonlight present.
4. No solar glare contamination.
5. Cloud-free (based on thermal detection of clouds).
6. No contamination from auroral emissions.
The unsaturated nighttime data from each individual orbit meeting the above criteria were averaged
in a 30 arc second grid (Platte Carree projection) using the radiance increment for a single DN from
the highest gain setting of the collections (1.35 x 10-10 watts/cm2/sr). Masking was used to remove the
lights from natural gas flares present on land areas in places such as Nigeria and Russia

Yes the image is imperfect, but we don’t need a perfect image to tell if the station location is bright/dark/dim etc.

I don’t claim this to be a perfect method by any means, but it is at the moment, the only one available to us.

I see it better used for qualitative than quantitaive analysis, much like surfacestations.org photos are used for. I think that location and distance info derived from these images would be reasonable, but given the 2.7 KM coarseness of measuremnet by DMSP and the overlay manipulations/scaling in Google Earth, the original pixels are obscured behind a series of adjustments, much like our surface temperature record.

If a better DMSP image can be found that is lat/lon regsitered, we can create our own overlay in Google Earth to minimize the issue.

I took a quick look at the Mona Loa volcano with the nightlights image on since I would think that would radiate a fair amount of IR even when not erupting. If the imaging was IR I would expect to see some brightness but I don’t. Areas like the university of HW show up as bright spots. It’s an interesting idea though if you had an IR image you could use it to identify UHI.

Thank you for this information. It is interesting that looking at the dark (A) GISS sites for New Zealand: one (Invercargill) appears to be located in the sea some distance for Invercargill city (I’m pretty sure is actually at Invercargill Airport); and the Dunedin Aerodrome site is located on the outskirts of Dunedin (one our largest cities).

I’ve had a quick look at my nearest station, Cockle Park UK. It is classed as bright and appears lit on the night light view even though it is about 1 mile from town limits. There seems to be a lot of flare on the night light images.

Just because you’re using an IR sensor doesn’t mean you’d detect UHI – it’s going to depend on data filtering and scaling of output data. Consider that the difference between the city surface temp and surrounding rural environs is, perhaps, 300K vs. 290K – assuming a large UHI delta T. The streetlamps and other forms of visual illumination will have temps in the 1000s of K. They could easily hide an hypothetical 10K delta T from UHI, when the output images are scaled to saturate at the temp of a tungsten filament. I think this goes along with what Geoff Sherrington is talking about, above. [#56]

We have satellites which could resolve UHI; I’ve seen the photos in Nat’l Geographic. These images look like they may have been tuned to look for visual illumination – albeit in the IR range.

If the output image is tuned to saturate from a 10,000 K heat source, then I go back to my comment #28 and Sean Egan’s #55. Local light pollution laws could have a big impact on how a community appears on the Nightlights image. Correct me if I’m wrong, but don’t cars in Europe have slightly amber (~6000 K ?) vs. American blue-white (~10,000 K ?) headlamps. The same may be true for streetlamps if, as Sean discusses, they are used at all.

Could this be a rationale for not using the same Nightlights data as in the US as a proxy for urbanization adjustment – it would tend to make European cities appear smaller / cooler than their US counterparts ?

Looking further at the image and sites for New Zealand. None of the GISS sites are plotting at their actual locations. Most are out by about 1-2 km, but some are up to 50 km out (See Campbell site).

The image is also showing brightness due to surf breaks, snow (see coast and centre of North Island), and other reflective surfaces. These are brighter than artificial sources of light nearby. Does this complicate UHI assessment?

The local weather forecast gives the location of the nearest weather station as 55 25′ 1 36′. This makes sense as this is a military base although the sensor appears close to a heliport. The map shows this area as dark. The Google map shows the location as 55 12′ 1 36′ and shows this as bright.

I agree that the “brightness” maps could be used semi-quantitatively for comparison with other factors in the way that “adjusted” surface temperatures are used. You could semi-quantitatively split them into 1, 2 and 3, being tiny, medium and quite large. My purpose was to head off at the pass any excitable person starting out to measure “brightness” however defined from the maps and make correlations with factors like population.

I was also concerned with positional accuracy – see now # 76 Willem de Lange for New Zealand. Finally, it would be a huge task to check each blob for positional accuracy, there are so many. So inaccuracies would be the expectation.

Thank you to those inputting tech specs – this helps understand what was done and explains some of the image features. But I suspect that in the long run they will be moved aside to a cartoon-type illustration role and will have no use as a serious research tool. There are too many noise factors, some already identified above, and the required accuracy will not be there.

I do not propose to do more detailed forensic analysis of the images so bandwith here is not going to be overused again. Thanks for listening Geoff.

re 74. There is no such explaination given in the text. In 1999 Hansen was looking at Gallos
work using a vegatative index to characterize UHI. Imhoff also was using nighlights to characterize
the urban enviroment. As was Owen. Gallo, Owen and Easterling decided on using a Combination
of metrics to define the urban rural boundary. LULC, Population, and nighlights. Typically these approaches get tested out in the US… and then?????

Here is the bottomline that bears repeating. Nighlights doesnt work any better than population.
Its a funny point lost on everyone. If you have a big city, the census take will find the people.
and yes, nighlights is a crude cut at “urban” it gets urban MORE WRONG than census takers. because no census taker would mistake moncton for rural. And Nighlights doesnt work any better for defining rural than a census taker.

Reading H2001 you will see that hansen is interested in the Density of population. There are are
products published by columbia that give you population densities for the entire globe at resolutions greater than nighlights. Shrugs?

Using the DMSP .kmz nightlights file I linked to above, when viewing the individual images for each satellite and each year in Google Earth, there is considerable variation between the images.

To get a feel for how different years are presented for Miles City, as a single RGB image I combined sat F10 1993 (red), F14 1996 (green), and F15 2003 (blue), and pushed saturation to the max to show the colours better:

Direct URL:

One thing that stands out is that the older (red) image has a greater extent then the newest (blue) image – it seems as if the suburban extent of Miles City is shrinking!

Those google earth kml files impressed me so much I made one up for all the GISS stations in use during 2006-2007. It’s less crowded than the full GISS list. Separate sub-folders for those that stayed active into 2007 and those that went inactive after 2006. Green pins are stations designated by GISS as rural. Red pins are non-rural.

The .kml file is available here. If you open it, it will start google earth immediately or load into it if it is already running. If you save it instead, you can click on it later to put it in google earth. It will reside as a sub-folder under temporary places.

Here is the first thing I noticed.
The locations of three Canadian stations.
Churchill, Man. 58.7N -94.1W
Fort Nelson, B 58.8N -122.6W
The Pas, Man. 54N -101.1W
These stations create a triangle with sides of 677 km, 1418 km, and 1631 km.
Since the temperatures of these stations are used even outside that triangle, I think it is fair to say these stations provide GISS with the temperature for what looks to be about 20% of Canada.
That’s some pretty fancy temperature interpolation they’re doing.

There was a truly amazing drop in US stations between 2006 and 2007. Almost all the rural ones are gone in addition to a good chunk of the non-rural.

Steve: This is just a reporting delay for the USHCN network. Early returns are mainly from MCDW.

Perhaps this is one of those situations where NASA spent $50 Million to generate “Nightlights” satellite capability, and there is now considerable internal pressure within the agency [on Hansen…] to publish academic articles that make use of the marginally useful data so future earmark requests are not jeopardized. The more bang for the buck from current technology, the more likely Congress will approve new gadgets…

OK, I have uploaded a 1.4 MB QuickTime movie of all the nightlight images for Miles City here.

Looking at the variation between years, and the variation between satellites in the same year (coloured frames – older sat brighter as reddish, newer sat brighter as bluish, both the same as grey shades), it seems to me that it is a hard ask for this method to give consistent results until the technology improves.

Perhaps it will be more useful in some kinds of environments and less so in others.

I think the datum problem is probably swamped by the resolution of the lat longs in the source file. They’re only out to the hundredths of a degree. Which gives you up to a 1.1 km error in the lat direction,

Correct me if I’m wrong, but wasn’t Moncton classified as urban dark? I realize that designation seems odd, and it is indeed unusual, but some cities are geographically large and have dark/rural areas within their boundaries. In these cases, the nighlights designation may be more useful than the census.

Admittedly, if the goal is to find only the best rural stations, then both urban and non-dark stations should be discarded so it’s a bit of a moot point.

There are are products published by columbia that give you population densities for the entire globe at resolutions greater than nighlights

I know the Moncton area quite well. By the map provided, it looks like the station is at the airport which is on the eastern edge of town. The area adjacent to the west side of the airport is an industrial park, while the area adjacent to the east side is mostly forest. “Urban Dark” appears to be a fitting term for the area.

Also, the greater Moncton area has a population of >100K. The census data seems to include only the city proper.

New York City, like other large cities, is warmer than surrounding areas due to the urban heat
island effect (UHIE). This phenomenon occurs when naturally vegetated surfaces are replaced
with impervious surfaces that absorb, retain, and reradiate more solar energy than do grass and
trees. The rate at which energy is absorbed and reradiated depends on the physical properties of
different surface types, their configuration within the urban fabric, regional meteorology,
localized microclimate, and the addition of anthropogenic heat into the urban atmosphere.
Currently, New York City’s summertime temperatures average 7.2ºF (4ºC) warmer than
surrounding suburban and rural areas. A recent study (Laurie Kerr & Daniel Yao, 2004)
determined that for each degree Fahrenheit of temperature increase during the summer, the City
consumes an average of 3,300 MWh more energy per day for cooling. Given an average cooling
season of 150 days, the annual energy savings for each degree of UHIE reduction would be
roughly 495 million KWh. While this analysis points to significant financial implications for the
City, it nevertheless understates the overall value of heat island mitigation, because substantial
additional savings would result from expected improvements in public health and environmental
protection.

……

The project established seven case-study areas for which temperature data were obtained during
three heat waves in the summer of 2002. A remote-sensing and geographic information system
(GIS) data library was developed to characterize the numerous dimensions of New York City’s
heat island, including land surface, urban morphology, and urban function information, as well as
base map layers including streets, hydrology, open space, block groups, and land cover.

I think the datum problem is probably swamped by the resolution of the lat longs in the source file. They’re only out to the hundredths of a degree. Which gives you up to a 1.1 km error in the lat direction,

Which make the accuracy of the city lights classification a bit futile for small towns, doesn’t it?

After playing with Miles City USA, I was going to see how the Google Earth ‘DMSP nightlights’ layers performed in a remote area of Western Australia, and thought as a starting point I would use Port Hedland, the largest tonnage port in Australia and a city of around 15,000 (incl. South Hedland), as it is large enough to have a good light footprint while also having many small mining towns and cattle stations nearby, thinking this may be a good test for nightlights.

Anyway, when zooming into NW WA on Google Earth, I noticed another mystery – there were two Port Hedlands a long way apart! One is the real Port Hedland, the other exists in only in the GISS database:

So anyway, having spent time in Port Hedland, I know that the BoM office is at the airport, as anyone can check to make sure here where it says:

For further reference, I looked at the station photos available from this site, and after looking at the photos for a bit to imagine what the place looked like from above, I zoomed in on the Port Hedland Airport.

After finding the BoM building, there was the instrument enclosure plain as day, so I marked it with a pin:

I then zoomed out to see where it is in relation to the airport:

And again to see proximity to Port Hedland:

And finally to see how far away NASA GISS thinks it is:

– which turns out to be a little over 100 km away from the real Port Hedland AWS!

IMHO the Google feature is as misleading as the NASA pictures are. Everywhere we are told, these images show the “light pollution” around the earth, but they do not:

The so called “light pollution” images are over-over-overestimated (especially the low resolution pictures like the NASA .gif images). Around each city there is a bright spot showing the “light pollution”, but if you compare these images with reality (remember a night flight over these areas), you realize that the real “light pollution” is rather small compared to the brightness of the images provided. All these images look like a 60+ seconds exposure of the night sky (which sums up to a “blue sky exposure”) with very bright stars (much brighter than they are in reality).

For example the Canary Islands are rather sparse inhabited and have some hotels in some regions, but in the NASA images there are large bright spots over almost all islands as they are drawn in Central Europe or the USA (here the Google layer is less misleading due to the higher resolution, but the brightness is still to high). On the other hand North Korea is almost not polluted (do they have no electricity there?) while the sea between South Korea and Japan is…

So all these images as well as the Google Earth layer do not show the real extent of “light pollution”, because they are overexposed images (just cartoons showing the intention of the cartoonist camouflaged by NASA graphics) with some qualitative but no quantitative information.

4 Trackbacks

[…] with the adjustment methodology used by NASA GISS. One of the issues being discussed is the application of city nightlights (used as a measure of urbanization near the station) as a proxy for UHI adjustments to be applied […]

[…] one source image from the Defense Meteorological Satellite Program from 1995. You can do the same yourself in Google Earth. Clearly from this example, GISS should be updating that source image if they are to get anything […]

[…] are calculating UHI for stations by looking at satellite images of nightlights, like GISS does (see my post on it at CA) , you’ll find that there’s generally no city lights in the water, leading you to think […]

[…] homogeneity adjustment doesn’t take Tucumcari’s declining population into account, it only uses nightlights, and while the population may dwindle, town infrastructure usually doesn’t; streetlights […]