Aileen Buckley, at the ESRI Mapping Center, has released a new tutorial on creating what she calls “sun glints”. These are the reflective glows you sometimes see on water bodies in aerial photographs. In essence, it’s a tutorial on creating gradient fills in ArcMap – a neat effect that cartographers don’t get to use very often.
The tutorial also includes instructions for randomizing the direction of the glints using Python. While her scientific explanation for this is sound, I’m not sure it’s really necessary. Personally, I wouldn’t use this effect to make a map more realistic, I’d use it to make a map look cool. Regardless, it’s a useful tutorial on a basic application of Python scripting.
This afternoon, I opened QGIS, and was greeted with a strange error message:
Couldn't load PyQGIS. Python support will be disabled. Traceback (most recent call last): File "", line 1, in RuntimeError: the PyQt4.QtCore module is version 1 but the qgis.core module requires version -1
QGIS will still open if Python support is disabled, but none of the geoprocessing tools will be available. The Vector menu won’t even show up! Clearly, this is bad news.
After a little research, I concluded that this version of QGIS (1.7.1 “Wroclaw”) was incompatible with the new PyQt4 library, which was passed to me as an update over the weekend. Eventually, a version of QGIS that is compiled against the new library will be available to install (and I hope that’s soon). But for now, the only solution for me is to downgrade my PyQt4 library from 4.8.6-1 to 4.8.3-2. This can be done from the command line:
sudo yum downgrade PyQt4
And QGIS is back to normal!
Note: this will only work in Fedora, RedHat, and other distros that use yum and rpm. For Ubuntu, you will have to use apt to perform the downgrade. See here for an example.
Slashgeo linked to an incredible web-based QGIS tutorial last week. I’ve been looking through it, and so far it is excellent. It’s not really designed for total newbies – there are other places to get your very first introduction to a desktop GIS. Instead, this tutorial seems to be aimed at those who are familiar with the concepts of GIS, but not yet familiar with QGIS. If that’s you, this tutorial will quickly get you up to speed. The combination of text instructions, screenshots (from a Mac!), and YouTube videos creates a fun and fast-paced learning environment which is ideal for people who want to get started quickly.
The only obvious drawback to QGIS presented here is the installation of the required software. Since QGIS was designed as a package for the Unix environment, it has a lot of dependencies on other projects, notably GDAL. And those packages have to be in place for the installation to proceed. Even the installation of Mac binaries requires a lot of hair-pulling. Windows standalone binaries are easy enough to install, but a static installation is not as extendable as a linkable application framework. So if you really want to capture the full power of QGIS, some assembly required.
My experience installing QGIS, in Fedora Linux 15 (and 12-14) has been much simpler. Fedora has a graphical package manager that lets me install QGIS just by checking a box (Figure 1). All the dependencies and prerequisites are taken care of by the package manager behind my back. Installing software doesn’t get much easier than that. Installing QGIS in Linux Mint, a spinoff of Ubuntu, was similar. Linux is still not the easiest OS to work with, but some distros have made a lot of progress on the package management front.
Figure 1: QGIS in the Fedora package manager
Link to the tutorial:
http://cga-5.hmdc.harvard.edu/qgis/
Chances are, you’re not familiar with OpenStreetMap. The market saturation of Google Maps and Bing Maps is such that they’re the tools we turn to, sometimes without even thinking. Both these services have an incredibly beatiful and usable design aesthetic, which makes them easy to read. There’s no real reason why you’d need a third web mapping service – unless you really wanted one. If you like having alternatives, especially alternatives that are open source, crowd-editable, and not controlled by nearly monopolistic companies, then OpenStreetMap may be the project for you.
The best thing about OpenStreetMap, besides the ability to add your own map features via a wiki-like interface, is the wide variety of viewing options. As you may know, web mapping services work by serving image tiles to your browser. The visual aspects of the map are dependent on which set (‘deck’) of tiles you are loading. Google Maps, for instance, has map tiles, satellite tiles, and topography tiles. OpenStreetMap currently has five different tile decks to choose from, using the handy blue layer switcher in the upper right corner of the map. Each deck has some overlap in functionality, and each has its benefits and drawbacks. To illustrate the differences, I’ve included links to the main campus of the University of Colorado at Boulder, my alma mater.
Mapnik
This default deck features the same basic light gray and muted yellow as a Google map, but with a wider pastel pallete to color-code buildings and street types. The inclusion of sidewalks is a nice touch, but the map feels like maybe it needs a key. One major drawback of this deck is that the campus boundary isn’t as clear as in other decks.
http://www.openstreetmap.org/?lat=40.0057697296143&lon=-105.267241001129&zoom=16
Osmarender
The streets and street names in this deck pop a little bit more, but the strong purple background on the campus makes it feel too busy. In some cases, the placement of the street names makes the map harder to read, especially where a trail runs close to the street, or a highway has a local name. But in less dense areas, these tiles are very easy to read, and would look great on a small screen.
http://www.openstreetmap.org/?lat=40.0057804584503&layers=O&lon=-105.267241001129&zoom=16
Cycle Map
Although the functionality of this deck caters to the needs of cyclists, it also looks great as a car or pedestrian map. The bluish gray looks excellent as a base color, which makes the bike lanes and paths easy to find. This map still feels like it needs a key, though: it’s not easy to tell that the yellow areas represent auto parking lots (which are bicycle hazards).
http://www.openstreetmap.org/?lat=40.0057697296143&layers=C&lon=-105.267241001129&zoom=16
Transport Map
Everything fades into the background except for the bus routes. It’s not easy to tell which buses follow which routes, but there is no question which streets have bus traffic. Some of the major intersections are labeled. which can be convenient for visiting riders. Personally, I would like to see the actual stop locations. Assuming that the tiles are constructed using open transit data (some of which uses the Google transit specification), that should be easy enough to add.
http://www.openstreetmap.org/?lat=40.0057911872864&layers=T&lon=-105.267241001129&zoom=16
MapQuest Open
Recently, MapQuest opened up their tiles to OpenStreetMap, and the result looks fabulous. This deck may not be as informative as other decks, but it is simple enough that it doesn’t really need a key. I’ve been having trouble with the tiles loading slowly, but that’s my only complaint. A very usable map.
http://www.openstreetmap.org/?lat=40.0058019161224&layers=Q&lon=-105.267241001129&zoom=16
Google Map for comparison:
http://maps.google.com/maps?hl=en&ll=40.005544,-105.267498&spn=0.0119,0.021865&vpsrc=6&t=m&z=16
One of the keys to good database planning is correctly anticipating what type of data each field will hold. If a field is expected to only hold integers, it is best to make it an integer field instead of a float or double field. Doing so will save space and time, and the database structure will be more intuitive. Likewise, it is a good idea to fix the number of characters allowed in a string field ahead of time.
Sometimes, though, you may run across data that is incorrectly typed. The most common error is a value that looks like a number but is actually a string of numeric characters – they may look the same, but they certainly don’t act the same. Models, scripts, and geoprocessing tools will go nuts when they try to do math on mistyped fields, and the vague errors thrown may take hours to decode. (I suspect these errors are caused by importing incorrectly typed MS Excel spreadsheets into a database, but don’t quote me on that.)
Fortunately, there’s an easy way to dynamically change the field type in ArcGIS. This is called typecasting, and it is a common feature in untyped programming languages like Python and JavaScript. To typecast a numeric string as an integer, create a new integer field. Then use the field calculator to multiply the numeric string by 1. The field calculator is “smart” enough to know that it can’t multiply a string, so it multiplies the number value of the string instead. The new field will then be populated with integer values that you can manipulate to your heart’s content. The same procedure can also be used to typecast numeric strings (or integers) as floats or doubles. Just multiply the values by 1.00 instead of 1.
Part of the GIS Day 2011 festivities at Texas A&M University included a student poster display in Evans Libary on the main campus. There were 29 posters covering a wide range of GIS topics from various classes and departments. All 29 posters contained a scalebar, a legend, and a North arrow. But only three contained any sort of projection information.
GIS analysis, by definition, requires 2 or more datasets – one dataset is just a map, right? For a spatial analysis to make any kind of sense, there has to be some level of confidence that the datasets are correctly aligned with one another. That confidence is easiest to achieve by providing projection information. Claiming that a map uses Wisconsin Central State Plane doesn’t prove that the data is properly aligned. But it does imply that the cartographer has taken every possible step to insure the correctness of the map. Knowing the projection is the first step toward correctly managing projections.
I wonder if providing projection information would be more common if desktop GIS layout programs used a simple function to insert it into the map? ArcGIS has made adding a North arrow to a project so easy that most users don’t bother to stop and consider whether it is functionally necessary. I’d like to see similar functionality with regard to projections.
Happy GIS Day everyone! GIS Day (and Geography awareness week) has certainly gotten more popular over the past year. The first time I ever heard of it was last year, when a former professor mentioned it on Facebook. This year, it seems like everyone is celebrating. Universities are hosting student poster displays, and local governments are hosting open houses and demonstrations. The economy has not been kind to the industry, so it’s nice to see that everyone has been able to keep their spirits up.
GIS Day is a special day at Northwest Spatial Solutions – it’s a birthday. One year ago today, I registered this domain and made my first post. At that time, I had been living in Hawaii for five months, and had just finished writing my first web-based geocoding application. I wanted to share it with someone, and I decided the internet would make a fantastic audience. (That geocoding application has also gone through some changes, and the newest version will be rolled out sometime next week.) And now here we are, 30 posts and 365 days later.
I guess this is as good a time as any to let everyone know that Northwest Spatial Solutions will be heavily revitalized over the next few weeks. You may have already noticed the new, cleaner look to the site – that’s just the beginning. Starting next week, I will be submitting two posts per week, most likely on Mondays and Wednesdays. I will also be promoting new posts via Twitter, Facebook, Google+, and LinkedIn. Over the next few months, the Portfolio section of this site will be filled in, allowing anyone to keep an eye on what I’ve been up to.
Today I’ll be attending a few conferences on the Texas A&M University campus. I hope you’ve got some celebrations planned as well.
When loading data into a PostGIS database using shp2pgsql, have you ever gotten this cryptic error?
Unable to convert data value to UTF-8 (iconv reports "Invalid or incomplete multibyte or wide character"). Current encoding is "UTF-8". Try "LATIN1" (Western European), or one of the values described at http://www.postgresql.org/docs/current/static/multibyte.html.
This means that there is an invalid or incomplete multibyte character in your data somewhere. Unfortunately, the error message figures the only possible reason for this is that you have got the character set of your database wrong. There is another possibility, though – that you got the character set right, and your shapefile contains an invalid multibyte character. Which is unfortunate, but easy enough to fix.
Open QGIS, and add your shapefile, making sure to set the file’s encoding to UTF-8. Now open the attribute table and start scrolling through the records. If it was displayed in octal or hex form, the invalid character would be impossible to find simply by scanning. Fortunately, QGIS displays invalid characters using an easily recognizable dingbat:
Once you’ve found the offending character(s), simply delete it, and replace if necessary. After closing QGIS, the shp2pgsql operation should complete without errors.
If you divide your time between ArcGIS and QGIS, you may have noticed that they don’t use the exact same projections. For Northern Colorado, for example, ArcGIS offers “NAD 1983 HARN StatePlane Colorado North FIPS 0501”, while the best QGIS can do is “NAD83(HARN) / Colorado North”. When projection information doesn’t agree exactly, I get very nervous.
Fortunately, we can ask our friends at SpatialReference.org for further information. By browsing through the list of projections for Colorado, we find that the ESRI projection above is called “ESRI:102253”, and the QGIS equivalent is called “EPSG:2772”:
http://spatialreference.org/ref/?search=colorado
EPSG stands for European Petroleum Survey Group, an international organization which collects and catalogs spatial data parameters. Their projection definitions are the world standard for everyone except ESRI, who prefers to keep their projections proprietary. So the question is, what’s the difference? Which one should I use for my project? Which one is “better”?
To find the answer, click through to a projection from the above list, and then click on the Proj4 definition, which shows you its parameters quite clearly. This is the Proj4 statement for EPSG:2772:
+proj=lcc +lat_1=40.78333333333333 +lat_2=39.71666666666667 +lat_0=39.33333333333334 +lon_0=-105.5 +x_0=914401.8289 +y_0=304800.6096 +ellps=GRS80 +units=m +no_defs
And this is the Proj4 statment for ESRI:102253:
+proj=lcc +lat_1=39.71666666666667 +lat_2=40.78333333333333 +lat_0=39.33333333333334 +lon_0=-105.5 +x_0=914401.8289 +y_0=304800.6096 +ellps=GRS80 +units=m +no_defs
As you can plainly see, the definitions are largely identical. The only difference is that the 2 standard parallels that define the Lambert Conformal Conic projection are swapped. This doesn’t seem like much, but the transformation equations suggest that it may lead to sub-millimeter differences in the calculated y-coordinates. For most applications, these differences are not worth worrying about.
However, if you do feel like worrying, it’s very easy to teach new projections to QGIS. Copy the Proj4 definition from SpatialReference.org and paste it into the Settings > Custom CRS dialog. Give it a name and hit the tiny save icon beneath the parameters box. Now you can access your ESRI projection in QGIS.
Adding EPSG coordinate systems to ArcGIS is also possible, but requires a couple more clicks. In ArcCatalog, right click a shapefile with the ESRI projection to be modified. On the XY Coordinate System tab, click the Modify button. Now you can give your projection a new name and swap the values for Standard_Parallel_1 and Standard_Parallel_2. Click okay to save the file with the modified projection. This returns you to the shapefile properties dialog, where you can click Save As to save the projection for use with other files.
Code: The Hidden Language of Computer Hardware and Software
by Charles Petzold
Microsoft Press, 2000
ISBN-13: 978-0735611313
The fact that you’re reading this suggests that you know how computers work. But that’s only at one level of abstraction. In the case of the touch/point/click interface, that can be ten or more levels of abstraction above the electric signals flowing through your hardware. Most people don’t know how logic gates and memory circuits work – and they usually don’t have to. However, if you’re really interested in how those first few layers of abstraction work, read this book. There’s no better introduction.
The book begins innocuously enough with discussions on Morse Code, Braille, and simple electric circuits. And then things begin to ramp up with chapters about logic gates and binary math. After a brain-twisting chapter on memory circuits, the author has built a theoretical machine similar to the Altair using millions of 19th century telegraph relays. From there, technology advances to the age of integrated circuits, and the author uses an Intel 8080 to introduce memory pointers and op codes. Without even knowing it, you’ve just learned assembly language! Eventually, the book catches up with “modern” (late 90s) technology, and the final chapter on graphics systems should be mostly review to anyone who knows the difference between vector and raster.
Although there’s a lot of information presented here, it makes for very easy reading. The first five chapters just breeze by. The book can get quite tough towards the middle, especially the chapters on flip-flop circuits and memory (14 and 16 respectively). But don’t worry if you get lost – the technical details become unimportant as the abstraction level gets ramped up. After all, this is only for fun – only a handful of highly paid engineers have to worry about XOR gates or processor opcodes anymore.
