TileMill/MapBox Test

I’ve just today learned how to make a map with TileMill, using the tutorial found at http://mapbox.com/tilemill/docs/crashcourse/introduction/, and uploaded it to MapBox. I also installed a plugin that allows me to display MapBox maps within WordPress. You can use the mouse to pan and zoom from within the post. I’m looking forward to experimenting further with this technology, it seems really simple to use.

One problem I’ve noticed is that sometimes the symbol markers are cutoff. You can see an example of this if you pan the map to the area north of New Zealand. The problem has persisted through this project, from desktop to web. I assume that one of the tile seams is cutting off the marker symbols.


Ubuntu GIS from Scratch: Step Six – QGIS

Welcome to Step Six in our exploration of open source GIS using Ubuntu. In the previous installments, we uploaded some spatial data into a PostGIS database, and experimented with spatial SQL. Finally, we’re ready to install QGIS, the most popular open source GIS desktop available. QGIS is a lot like ESRI’s ArcMap, in that it presents the users with a familiar environment of vector layers, rasters, and attribute tables in which to do analysis. QGIS also has its own Python-based scripting API, and is extensible through plug-ins and code samples. Best of all, QGIS can connect to a wide variety of databases, including those running PostgreSQL and PostGIS.

1. QGIS is fairly easy to install, but we have to get our hands on the packages first. Since we’ve gone through all the work to install GDAL and GEOS, and know how to build from source, it would make sense to just download a tarball and install it by hand. But QGIS has an unbelievable number of package dependencies, and many of them are different versions from the ones we installed earlier. Instead, we’re going to install a pre-built package, which comes with a set of pre-compiled dependencies. This package is provided by QGIS, so we know it will be complete. In order to install packages from QGIS, we have to add their servers to our package installation database. This is easy to do. Start the terminal, and then execute:
sudo gedit /etc/apt/sources.list

This will bring up the sources.list file in a text editor, ready for modification. Go down to the very end of the document and add the following lines:
## Sources for QGIS
deb     http://ppa.launchpad.net/ubuntugis/ppa/ubuntu precise main
deb-src http://ppa.launchpad.net/ubuntugis/ppa/ubuntu precise main

These are the package servers for Ubuntu Precise, which is the codename for 12.04. Save the file and then exit gedit to be returned to the terminal.

Next we have to modify our package security settings in order to accept packages from this new source. Basically, we are adding a public key to our security keyring, which lets the system know that we trust packages from this server, as long as they match our key:
sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys 314DF160
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /tmp/tmp.hV5DAdIWJ2 --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv-keys 314DF160
gpg: requesting key 314DF160 from hkp server keyserver.ubuntu.com
gpg: key 314DF160: public key "Launchpad ubuntugis-stable" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Well, that’s that. Now we can start installing:
sudo apt-get install qgis
This will add a ton of packages to your system, and it will take close to half an hour. But when it is all done, QGIS 1.8 (codename: Lisboa) will be installed!

2. During the install process, a custom directory is created in your home folder. This directory contains a database where all your customizations and add-ons will be stored. Unfortunately, this directory is read-only after installation. This is a bug that has been dealt with in version 1.9, but we’re stuck with it for now. However, it’s easy to fix – just delete the folder. To find the folder, change to your home directory:
cd ~
If you list the contents of this directory, you won’t see the .qgis directory, because it has a . at the beginning of its name, and ‘dotfiles’ are hidden by default. So we need to list everything. The -la flag shows you everything in convenient table form:
ls -la
There are probably a lot of hidden folders in your home directory – that’s where applications store your personalized settings. The permissions string of .qgis will look like:
drw_r__r__
And its owner and group will be root. This means that root can read and write in this directory, but all other users, including users in the root group, can read only. You could change the settings yourself, by running the chmod, chgrp, and chown commands, and that’s what you’d do if there was anything stored in the directory that you wanted to save. But it’s empty, so there’s no reason to keep it around:
sudo rm -Rf .qgis
rm is the remove command. -R makes the command run recursively, deleting any folders inside the folder, and -f forces the command to override any failsafes. This is a very powerful command, you do not want to use it lightly!
The next time you start QGIS, this folder will be recreated, but with you as the owner and group, and with a permission string of:
drwxrwxr_x
Which means that you and any member of your group can read, write, and execute, while other users can read and execute. One could write a whole book on managing permissions alone: all you need to know is that you can now use this directory to store customizations. Which will come in handy later.

That’s all there is to it. Feel free to play around with QGIS – you should be able to load many shapefiles, KML files, rasters, and even non-spatial tables, right out of the box. But the real power lies in connecting to a spatial database, which we will get to in Step Seven, the final installment in this series.


Ubuntu GIS from Scratch: Step 5 – Spatial SQL

Welcome to Step Five in our exploration of open source GIS using Ubuntu. In the previous installment, we uploaded some spatial data into a PostGIS. database. The next logical step is to install QGIS and add start visually analyzing this data, which we’ll get to in Step Six. But before that, I wanted to take a short time out and give a quick demonstration of some of the spatial analysis functions PostGIS can perform on its own through SQL database queries.

Warning: If you haven’t completed the previous steps, installing and configuring PostgreSQL and PostGIS, and uploading Portland streets data, none of the following will work at all – you’ll get all kinds of crazy errors. So unless you have these programs and datasets installed and working on your machine already, you’ll need to go back and do that before proceeding.

1. Connect to the Database (using the terminal):
psql -d portland
In the previous step, we added Portland’s custom coordinate system definition (SRID 6856) to the spatial_ref_sys database. However, that definition was originally missing a piece of information – the proj4 text. Proj4 is a reprojection system used by PostGIS to transform coordinates. If we add the proj4 text, we’ll be able to project coordinates in the terminal. I went back and updated the previous step to include the proj4 text, but if you missed that, here’s the instructions to do it now:
UPDATE spatial_ref_sys SET proj4text=’+proj=lcc +lat_1=44.33333333333334 +lat_2=46.0 +lat_0=43.66666666666666 +lon_0=-120.5 +x_0=2500000 +y_0=0 +ellps=GRS80 +units=ft +no_defs’ WHERE srid=6856;
UPDATE 1

2. Simple Selection
The first thing to try is a basic selection. Here’s the code to select SE 4th Street:
SELECT prefix, streetname, ftype FROM public.streets WHERE streetname=’4TH’ AND prefix=’SE’ AND ftype=’ST’;

prefix | streetname | ftype
--------+------------+-------
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
SE     | 4TH        | ST
(18 rows)

There is only one SE 4th Street, but it is stored in the database as 18 segments, extending from intersection to intersection. This makes it easier to do network analysis and geocoding.

3. Finding an Intersection
Since each street segment has its own database entry, every two street cross intersection has four intersection points, all of which have the same coordinates. And it’s likely that each street will include ten or more segments. This requires a complicated SQL statement that will Select each street as a subtable, and then Select the intersection from the subtables. For this example, we’ll be selecting the intersection of NW Lovejoy Street and NW 23rd Avenue.
SELECT distinct(AsText(intersection(b.geom, a.geom))) AS the_intersection FROM (SELECT geom FROM public.streets WHERE streetname=’LOVEJOY’ AND prefix=’NW’ AND ftype=’ST’) a, (SELECT geom FROM public.streets WHERE streetname=’23RD’ AND prefix=’NW’ AND ftype=’AVE’) b WHERE a.geom && b.geom AND intersects (b.geom, a.geom);
the_intersection
------------------------------------------
POINT(7638738.59481627 686914.222769022)
(1 row)

The results are in projection units – feet. That might be useful in coordinate space, but perhaps latitude and longitude might be more useful. So here’s the same query again, with an added transformation function that reprojects to SRID 4326, which is unprojected WGS84 coordinates.
SELECT distinct(AsText(ST_Transform(intersection(b.geom, a.geom),4326))) AS the_intersection FROM (SELECT geom FROM public.streets WHERE streetname=’LOVEJOY’ AND prefix=’NW’ AND ftype=’ST’) a, (SELECT geom FROM public.streets WHERE streetname=’23RD’ AND prefix=’NW’ AND ftype=’AVE’) b WHERE a.geom && b.geom AND intersects (b.geom, a.geom);
the_intersection
-------------------------------------------
POINT(-122.698608851154 45.5297891204646)
(1 row)

You can check this result by transposing the numbers (so that the latitude is first) and then pasting them into the search bar in Google Maps:
45.5297891204646, -122.698608851154

4. Geocoding
Each street segment in the database contains 4 numbers that describe the address ranges used by the segment. Leftadd1 is the lowest number on the left, and rgtadd1 is the lowest number on the right; while leftadd2 and rgtadd2 are the highest numbers. To geocode a street address, the first step is to find the street segment containing the address. As an example, let’s search for the House of Ramen restaurant in downtown Portland, which is located at 223 SW Columbia Street.
SELECT gid, prefix, streetname, ftype, leftadd1, leftadd2, rgtadd1, rgtadd2 FROM public.streets WHERE streetname=’COLUMBIA’ AND prefix=’SW’ AND ((leftadd1 < 223) OR (rgtadd1 < 223)) AND ((leftadd2 > 223) OR (rgtadd2 > 223));
gid  | prefix | streetname | ftype | leftadd1 | leftadd2 | rgtadd1 | rgtadd2
-------+--------+------------+-------+----------+----------+---------+---------
40183 | SW     | COLUMBIA   | ST    |      200 |      298 |     201 |     299
(1 row)

Based on this info, it looks like our odd-numbered address will be on the right side of the street. Take note of the gid, the unique number we can use later to isolate this segment. But first, we have to determine how far down the street 223 is likely to be. We could estimate that it will be about 22% of the way down, and that’s probably close enough. But there’s a mathematical way to get a more precise value.
proportion = (number – rgtadd1)/(rgtadd2 – rgtadd1) = (223-201)/(299-201) = 22/98 = 0.2245
Therefore our query will be searching for the coordinates of the point that is 22.45% of the way down street segment 40183:
SELECT ST_AsText(ST_Line_Interpolate_Point(geom, 0.2245)) FROM public.streets WHERE gid=40183;
st_astext
------------------------------------------
POINT(7644036.17590994 680787.382718017)
(1 row)

Again, this is in projected units, so let’s use an on-the-fly transformation to get an answer in decimal degrees:
SELECT ST_AsText(ST_Transform(ST_Line_Interpolate_Point(geom, 0.2245),4326)) FROM public.streets WHERE gid=40183;
st_astext
-------------------------------------------
POINT(-122.677299994613 45.5133847845873)
(1 row)

If you transpose these results and search for them in Google Maps, you will see some interesting results. There will be a green arrow on the map showing a point that is exactly 22.45% of the way down the street. However, it shows House of Ramen being much further down the street. Google Maps also assumes that you’re looking for something else, perhaps in the building on the other side of the street, because its search algorithm is trying to anticipate your needs by choosing the nearest business location. What we’ve done is mathematically accurate, but it doesn’t actually reflect the real-life addressing of SW Columbia Street.

Well, that’s just a quick demonstration of how SQL can be used to do spatial analysis on a PostGIS table. This may not be helpful for desktop analysis, but it is an important part of developing spatially enabled web applications. In Step Six, we’ll be installing QGIS, and that’s where the real fun begins.


Ubuntu GIS from Scratch: Step 4 – Loading Data.

(Note – this post was updated on October 28 to include new information!)

Welcome to Step Four in our exploration of open source GIS using Ubuntu. In the previous installment, we spatially extended our PostgreSQL database server with PostGIS. Now we’re ready to load some geographical data into our database. PostGIS includes a tool called shp2pgsql which allows us to prepare a standard ESRI shapefile for upload. The first step is to find some data.

Warning: If you haven’t completed the previous steps, installing and configuring PostgreSQL and PostGIS, none of the following will work at all – you’ll get all kinds of crazy errors. So unless you have these programs installed and working on your machine already, you’ll need to go back and do that before proceeding.

1. The datasets I have chosen to use in this example describe urban features in Metro Portland, OR. The Metro GIS data portal is located at:
http://rlisdiscovery.oregonmetro.gov/
We’re interested in two datasets here. The first is collection of city boundary polygons called ‘City Limits (poly)’, which can be found on page 3. And the second is the street network, which is called ‘Streets’ and can be found on page 10. Download both of these files to your Downloads folder.

2. To unpack the data, open the Downloads folder and double-click streets.zip, which will open the zipfile in the Archive Manager. In the Archive Manager window, you will see that this zipfile contains 25 objects. When you click the Extract icon, a dialogue will appear, asking you where you want to store the contents. In order to make it easier to move the files around, you should create a folder called ‘streets’ before clicking the extract button. Do the same for city_fill.zip, creating a folder called ‘city_fill’ to store its contents. Next, create a new folder in your home folder called ‘GIS’ and drag the streets and city_fill folders into it.

(Note: sometimes you will only see one object in the Archive Manager window, a folder. In this case, you do not need to create a new folder before extracting.)

3. Uploading data into a PostGIS database requires you know what the SRID is – the spatial reference ID, a number corresponding to the projection of your data. This is not always easy, because many SRIDs are very similar, and the wrong one can lead to data errors. The first thing to do is to check the shapefiles themselves for projection data. The Portland streets dataset includes a file called streets.prj. If you open this file in a text editor, you will see the name of the projection on the first line:
NAD_1983_HARN_StatePlane_Oregon_North_FIPS_3601_Feet_Intl
Once you know the name, it’s fairly easy to find the right SRID. Go to the website:
http://spatialreference.org/
and do a search for ‘oregon north’. There will be 37 results, and 2 of them match our projection exactly. Both of them are exactly the same (a very uncommon happenstance), so it doesn’t matter which one we use. For demonstration purposes, I’m going to arbitrarily choose ‘SR-ORG:6856’. Click the link for this projection, and then click the link to the ‘PostGIS spatial_ref_sys INSERT statement’. You’ll see a long line of text beginning with the word INSERT. Make a note of this location, because we will need this insert statement later.

4. Unfortunately, PostGIS doesn’t know about SRID 6856 yet. By default, it only comes with a limited number of SRIDs, and most of them correspond to projections defined by the EPSG, a European geodetic standards organizer. So we will have to add this SRID to the spatial_ref_sys table in template_postgis. This requires a trip to the terminal. The first command will unlock template_postgis, allowing us to connect with it:
psql -d postgres -c “UPDATE pg_database SET datallowconn=’true’ WHERE datname=’template_postgis’;”
The response will be:
UPDATE 1
Now we can connect:
psql -d template_postgis
As before, this will drop us into the PostgreSQL command shell, and the prompt will change. From here we can verify that SRID 6856 is not in the spatial_ref_sys table using an SQL select statement:
SELECT srid, auth_name, auth_srid FROM spatial_ref_sys WHERE srid = 6856;
(0 rows)
Since the select statement found 0 rows, SRID 6856 must not exist in the table.

At this point, recall the insert statement found on Spatial Reference in the previous step. If you know anything about SQL, you will see that the first number in this statement is the SRID – in this case, it is 96856. Spatial Reference likes to prefix a 9 to its SRIDs so that there is no chance of overwriting an SRID that is already in the table. This is essential because the SRID is the primary key of the spatial_ref_sys table, and must be unique. However, since we know that there is no 6856 in the table already, the extra 9 is just confusing. For the sake of convenience, we want to delete the 9 from the insert statement. Here’s the complete command, without the extra 9, which you can run now:
INSERT into spatial_ref_sys
(srid, auth_name, auth_srid, proj4text, srtext) values ( 6856, ‘sr-org’, 6856, ”,’PROJCS[“NAD_1983_HARN_StatePlane_Oregon_North_FIPS_3601_Feet_Intl”,
GEOGCS[“GCS_North_American_1983_HARN”,DATUM
[“D_North_American_1983_HARN”,SPHEROID[“GRS_1980”,6378137.0,298.257222101]],
PRIMEM[“Greenwich”,0.0],UNIT[“Degree”,0.0174532925199433]],PROJECTION
[“Lambert_Conformal_Conic”],PARAMETER[“False_Easting”,8202099.737532808],
PARAMETER[“False_Northing”,0.0],PARAMETER[“Central_Meridian”,-120.5],
PARAMETER[“Standard_Parallel_1”,44.33333333333334],PARAMETER
[“Standard_Parallel_2”,46.0],PARAMETER[“Latitude_Of_Origin”,43.66666666666666],UNIT[“Foot”,0.3048]]’);
INSERT 0 1
(Don’t worry that this command spans more than one line – the PostgreSQL shell doesn’t execute the command until it sees a semicolon at the end.)
Now you can check your work using the same select statement from above:
SELECT srid, auth_name, auth_srid FROM spatial_ref_sys WHERE srid = 6856;
(1 row)

**********************************
EDIT: in order to use the built-in projection and transformation functions in PostGIS, we need to include the proj4 defintion for this projection in the database. The insert statement we copied from spatialreference.org does not include the proj4 definition, and I haven’t seen it anywhere online. Fortunately, I figured out how to make one myself. To add this information to spatial_ref_sys, give the following command:

UPDATE spatial_ref_sys SET proj4text=’+proj=lcc +lat_1=44.33333333333334 +lat_2=46.0 +lat_0=43.66666666666666 +lon_0=-120.5 +x_0=2500000 +y_0=0 +ellps=GRS80 +units=ft +no_defs’ WHERE srid=6856;
UPDATE 1
**********************************

That’s done, so we can exit from the PostgreSQL command shell:
\q
And lock up template_postgis again:
psql -d postgres -c “UPDATE pg_database SET datallowconn=’false’ WHERE datname=’template_postgis’;”
UPDATE 1

5. Now we will leave the terminal for a moment and open pgAdmin. You should have a PostgreSQL server instance called ‘pgtest’ that has two databases in it – postgres and template_postgis (there may be more if you took my previous advice to try creating and exploring databases). Let’s create a brand new database to load our data into. Right-click on Databases in the left frame and choose New Database. On the Properties tab of the New Database dialogue box, set the name to portland, and choose yourself as the owner. Then on the Definition tab, set the template to template_postgis, then click OK. This creates a new database that has the PostGIS extensions, and a table in the public schema called spatial_ref_sys. Thanks to our work in the previous step, the spatial_ref_sys table includes SRID 6856. Now close pgAdmin and return to the terminal.

6. When we installed PostGIS, the upload tools were installed to a very inconvenient location that is not in your system path. We could move the tools to more convenient location, but it’s better to make a link to the tools instead. That way, if the original tool is changed, we don’t have two versions of the tool in different locations.
First, switch into the main program directory:
cd /usr/bin
Next, make a link for shp2pgsql, the shapefile loader:
sudo ln -s /usr/lib/postgresql/9.1/bin/shp2pgsql shp2pgsql
While we’re here, we might as well do the same for raster2pgsql (the raster loader), and pgsql2shp (the shapefile unloader):
sudo ln -s /usr/lib/postgresql/9.1/bin/raster2pgsql raster2pgsql
sudo ln -s /usr/lib/postgresql/9.1/bin/pgsql2shp pgsql2shp
Now we can call these programs from anywhere in the file tree.

7. Finally, we can begin uploading data. The commands will be simpler if we do the uploading from the location of the shapefile, so let’s change directories to the streets folder:
cd ~/GIS/streets
We want to upload the streets.shp file into a table in the public schema called streets. We know the SRID of the data is 6856. Before we can upload the data, we have to package it. The shp2pgsql tool will create an SQL package from a shapefile. Here’s the complete command:
shp2pgsql -S -s 6856 streets.shp public.streets > streets.sql
Shapefile type: Arc
Postgis type: LINESTRING[2]

All the records in this particular shapefile have the same geometry, which is a WKT LineString composed of 2 points. The -S flag forces shp2pgsql to use only simple geometries, instead of creating multi-part geometries. Now we can upload the SQL package we just created into the database:
psql -d portland -f streets.sql
This will take a few minutes. First the following lines will appear momentarily, so quickly that you may not even see them:
SET
SET
BEGIN
psql:streets.sql:24: NOTICE:  CREATE TABLE will create implicit sequence "streets_gid_seq" for serial column "streets.gid"
CREATE TABLE
psql:streets.sql:25: NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index "streets_pkey" for table "streets"
ALTER TABLE
addgeometrycolumn
-------------------------------------------------------
public.streets.geom SRID:6856 TYPE:LINESTRING DIMS:2
(1 row)

And then you’ll see over 100,000 lines of this, one for every street:
INSERT 0 1
At the end, you will finally see:
COMMIT
Before we open pgAdmin to admire our handiwork, let’s upload the municipal boundary file too:
cd ../city_fill
shp2pgsql -s 6856 cty_fill.shp public.cty_fill > cty_fill.sql
This time, our shapefile includes some multi-part polygons, so we can’t use the -S flag or there will be an error.
psql -d portland -f cty_fill.sql
The upload process for this file is similar, except it won’t take nearly as long.

Once the upload is finished, reopen pgAdmin. Now you will see two new tables in the public schema – streets and cty_fill. Coming up in the next installment, we’ll use the PostgreSQL command shell to run a few spatial queries. After that, we’ll be ready to tackle QGIS.


Ubuntu GIS from Scratch: Step 3 – PostGIS

Welcome to Step Three in our exploration of open source GIS using Ubuntu. In order to grasp the full power of GIS, we need to spatially extend our database (PostgreSQL, which was installed in the first step). The spatial extension for PostgreSQL is PostGIS. In the second step, we installed GEOS and GDAL, two major dependencies of PostGIS. Now we’re ready for the real thing. Once again, the latest version of PostGIS is not available from the Ubuntu Software Center, so we have to install from source.

Warning: If you haven’t completed the previous steps, installing and configuring PostgreSQL, GEOS, and GDAL, none of the following will work at all – you’ll get all kinds of crazy errors. So unless you have all three installed and working on your machine already, you’ll need to go back and do that before proceeding.

1. The first thing to do is download the PostGIS tarball, which can be found here:
http://postgis.refractions.net/download/
The latest version as of this writing is postgis-2.0.1.tar.gz
You may notice that the PostGIS downloads site refers to this release as a “stable” release. A stable release is a version of the software that the development team has decided is bug-free and ready for general use. When a release becomes stable, development is frozen – any further development becomes part of the next release. This is in contrast to an “unstable” or “testing” release, which is undergoing current development, and may be changing on a daily basis. Unless you have a specific reason to do otherwise, always choose a stable release when downloading a source package.

2. The source package is a gzipped tarball, so switch to the Downloads directory and issue the appropriate tar command.
cd ~/Downloads
tar zxf postgis-2.0.1.tar.gz
cd postgis-2.0.1
In the previous post, we used two configure apps, geos-config and gdal-config, to get information about the GEOS and GDAL installations. Now we’re going to feed the output of those apps into the configure statement of PostGIS. This is not usually required, but it makes the installation go a lot easier. The apps were installed in /usr/local/bin.
./configure –with-geosconfig=/usr/local/bin/geos-config –with-gdalconfig=/usr/local/bin/gdal-config –with-topology –with-raster
We’re also telling configure that we are definitely interested in having topology support and raster support available in the database. When configure is done, it is time to build and install the software:
make
sudo make install

3. As with our previous source installations, it is a good idea to update the linker database after installing new libraries:
sudo ldconfig
At this time, let’s restart the PostgreSQL database, so that PostGIS will be running:
sudo service postgresql restart

4. During the PostGIS installation, a database template called template_postgis was created. However, it isn’t ready to use yet, because it doesn’t have the spatial extensions enabled. So let’s go ahead and do that. First is the main PostGIS extension:
psql template_postgis -c “create extension postgis;”
After running this command, you will see the message:
CREATE EXTENSION
And now to enable the topology extension:
psql template_postgis -c “create extension postgis_topology;”
CREATE EXTENSION
Although the newest version of PostGIS is implemented through extensions, it would also be nice to have backwards compatibility with databases managed by previous versions of PostGIS. So we’ll install the legacy functions too:
psql template_postgis -f /usr/share/postgresql/9.1/contrib/postgis-2.0/legacy.sql
This command will run quickly and generate a couple pages of output, most of which will look like:
CREATE FUNCTION
CREATE AGGREGATE

5. The public schema in template_postgis now has 4 table views, called geography_columns, geometry_columns, raster_columns, and raster_overviews. There is also a table called spatial_ref_sys where spatial reference system definitions are stored. We want all five of these to be accessible by any PostGIS users, which requires 5 different commands. Each command will be confirmed by “GRANT”:
psql -d template_postgis -c “GRANT ALL ON geometry_columns TO PUBLIC;”
GRANT
psql -d template_postgis -c “GRANT ALL ON geography_columns TO PUBLIC;”
GRANT
psql -d template_postgis -c “GRANT ALL ON raster_columns TO PUBLIC;”
GRANT
psql -d template_postgis -c “GRANT ALL ON raster_overviews TO PUBLIC;”
GRANT
psql -d template_postgis -c “GRANT ALL ON spatial_ref_sys TO PUBLIC;”
GRANT
Now, that the permissions and extensions of the template are set up just the way we like them, we want to make it difficult for anyone to tamper with it. This is accomplished by not allowing users to connect to the template directly:
psql -d postgres -c “UPDATE pg_database SET datallowconn=’false’ WHERE datname=’template_postgis’;”
UPDATE 1
(later, if you want to modify the template, you must run this command again, but setting datallowconn to ‘true’ instead of ‘false’)
One last configuration command keeps anyone from accidentally erasing the template:
psql -d postgres -c “UPDATE pg_database SET datistemplate=’true’ WHERE datname=’template_postgis’;”
UPDATE 1
(later, if you want to erase the template, you can run this command again, but setting datistemplate to ‘false’ instead of ‘true’)
Finally, restart the PostgreSQL server again to reflect our changes:
sudo service postgresql restart

6. At this point, we can look over our work in pgAdmin. Upon opening and connecting, you will see that there is a new database called template_postgis that you cannot connect to. However, it is quite simple to create a new instance of the template to explore. Go to Edit > New Object > New Database; or right click the template_postgis icon to select New Object > New Database. In the New Database dialog, on the Properties tab, give your new database a name and select an owner from the dropdown box. Now, on the Definition tab, use the Template dropdown box to select template_postgis, and then click OK. Your new database will appear in the left column, and you can use the [+] icons to expand and browse the database tree. You will see two schemas, one called public, and one called topology. A schema is the top level of structure in the database, which you can use to organize tables logically or conceptually. All of the main PostGIS functions and definitions are stored in the public schema, while the topology-related functions are in the topology schema. Later, when we upload data to the database, we’ll create a new schema, to keep our data and functions separate, which makes them easier to move and delete.

At this point, you have a fully functional spatially-enabled database system running on your computer. Coming up in the next installment, we’ll be uploading a shapefile into the database and exploring it. Later, we will install QGIS and load data from a PostGIS database directly into a graphical GIS platform.


Ubuntu GIS from Scratch: Step 2 – GEOS and GDAL

Welcome to Step Two in our exploration of open source GIS using Ubuntu. In order to grasp the full power of GIS, we need to spatially extend our database (PostgreSQL, which was installed in the previous step). The spatial extension for PostgreSQL is PostGIS, which we will eventually be installing. But before we can get to that, there are a lot of prerequisites to install, and not all of them are available in the Ubuntu Software Center.

Warning: If you haven’t completed the previous step, installing and configuring PostgreSQL, none of the following will work at all – you’ll get all kinds of crazy errors. So unless you have PostgreSQL installed and working on your machine already, you’ll need to go back and do that before proceeding.

1. Before we can start installing the dependencies of PostGIS, we have to install the dependencies of the dependencies of PostGIS. This could be done using the Ubuntu Software Center, but that would take forever, because you would have to search for each package by name and then install it. So I’m going to introduce a new way to install software – apt-get. Apt-get is the main engine behind the Ubuntu Software Center, which we can access directly by using the terminal. Enter the following commands into the terminal, providing your password when prompted:
sudo apt-get install postgresql-server-dev-9.1 proj libjson0-dev xsltproc docbook-mathml docbook-xsl
(this command will prompt you to install these 6 packages, plus 7 or more additional packages)
When that’s done, there are four more apt-get commands to run:
sudo apt-get install gettext dblatex openjdk-7-jre openjdk-7-jdk icedtea-7-plugin libxml2-dev
(this command will prompt you to install these 6 packages, plus 53 or more additional packages)
(note: the icedtea plugin package is not required for this tutorial, but you might as well get it now so you can use this version of Open Java (openjdk) in your web browser)
sudo apt-get install swig php5-dev phpunit openjpeg-tools libopenjpeg-dev libkml-dev
(this command will prompt you to install these 6 packages, plus 5 or more additional packages)
sudo apt-get install libpoppler19 libpoppler-dev expat libexpat1-dev libxerces-c-dev libcurl4-openssl-dev
(this command will prompt you to install these 6 packages, plus 4 or more additional packages)
sudo apt-get install g++ python-dev ruby
(this command will prompt you to install these 3 packages, plus 3 or more additional packages)

2. Now we can begin by downloading and installing GEOS. GEOS, or Geometry Engine Open Source is a library of routines for handling spatial geometry and topology. An older version of this package could be installed using apt-get or the Ubuntu Software Center, but we need the latest version, so we have to download the source package, which can be found at:
http://trac.osgeo.org/geos/
The latest version as of this writing is geos-3.3.5.tar.bz2, which you should save to the Downloads folder on your hard drive.

3. Installing GEOS is fairly simple. In the terminal, change the directory to the location where you saved the file:
cd ~/Downloads
In this case, the ~ is short for your home directory. The first thing to do is uncompress the downloaded file. Recall that it had an extension of .tar.bz2. Tar is an archive format, and bzip2 is a compression algorithm – you might say that the file has been compressed two ways. It only takes one command to decompress the file:
tar jxf geos-3.3.5.tar.bz2
tar is the decompression program, j tells it to use the bzip2 algorithm, x is to extract, and f means file. This will create a directory called geos-3.3.5, and you can delete the tar.bz2 file (also called a tarball) if you want. Now we’ll change into that directory:
cd geos-3.3.5
Before the installation, we need to set up which options we will use. This is done using a script called ‘configure’, which can be found in this directory. Running the script with the -h flag will provide us a list of options, so let’s do that first.
configure -h
Uh-oh! You probably got an error there:
configure: command not found
That’s because this directory is not part of the system’s path, which is a list of locations where executable programs live. To see you path, enter this command:
echo $PATH
We could add the current directory to the path, but that’s a lot of work. It’s much easier to prepend the command with a ./, which means to “run from here”:
./configure -h
You’ll see a long list of configuration options fly by, giving you an idea just how much customization can be done. I’ve selected the most useful options for you already:
./configure –enable-python –enable-php –enable-ruby
This will take a couple minutes, but don’t get up and stretch your legs just yet. When configuration is complete, without any errors, it’s time for the really long command:
make
Now is the time to get up and stretch, because you’re compiling software from source, which will probably take about 15 minutes. When it finishes, you can install the new libraries into their proper locations by running:
sudo make install
One of the things you have just installed is a small utility app that provides some information about GEOS to the system. You can use this app to check the installation by entering:
geos-config –version
If everything went well, you should see:
3.3.5
4. Now it’s time to download GDAL, the Geospatial Abstraction Data Library. This library is the backbone of open source GIS. You can find it here:
http://trac.osgeo.org/gdal/wiki/DownloadSource
The latest version as of this writing is gdal-1.9.1.tar.gz, which you should save to the Downloads folder on your hard drive.

5. Installing GDAL will be very similar to installing GEOS, but with a couple exceptions. The first is that the GDAL tarball uses gzip compression instead of bzip2 compression, so we’ll call tar with the z flag instead of the j flag. Also, the number of configuration options is considerably longer. But otherwise, the process is the same. Here’s the complete list of commands to run:
cd ~/Downloads
tar zxf gdal-1.9.1.tar.gz
cd gdal-1.9.1
./configure –with-python –with-poppler –with-pg –with-curl –with-openjpeg –with-geos –with-geotiff –with-jpeg –with-png –with-expat –with-xerces –with-java
make
sudo make install
Afterwards, you can check the installation using gdal-config:
gdal-config –version
Which should let you know that we have successfully installed:
1.9.1
6. We’ve just installed a ton of new libraries and header files. Now we need to add all these libraries to an indexing database so that the dynamic linker (which is called ld) can find them. This is done automatically after running apt-get, but it has to be done manually when installing software from source. Fortunately, it’s a simple command:
sudo ldconfig
Once you have done this, you can delete the tarballs you downloaded earlier, and the install directories from your Downloads folder.

Coming up in Step Three, we’ll be downloading, installing, and exploring PostGIS. The process will be very similar to the one we used here, so if you’ve made it this far, you should have no trouble with any of the other installation tutorials I have planned.


Ubuntu GIS from Scratch: Step 1 – PostgreSQL

After a recent hard drive crash, I decided to try out Ubuntu, the most popular consumer Linux distribution. Ubuntu has become popular because it is more lenient than other distributions with regard to proprietary software – it can use Flash and play mp3s out of the box, without installing additional software. Ubuntu, like other Linux distros, is also a great platform for the exploration of open source GIS. This post is the first in a series of tutorials on how to get some of the basic GIS system tools up and running. For the demonstration, I’ll be using a default installation of Ubuntu 12.04 LTS 32-bit desktop.

The first step is to install PostgreSQL. PostgreSQL is a free, open source, object-oriented database that can be spatially extended.

1. Open the Ubuntu Software Center. On the dash bar (the toolbar running down the left side of the screen), its icon looks like an exploding shopping bag. In the Software Center window, enter ‘postgresql’ in the search box near the upper right corner. The first result should be “Object-relational SQL database (supported version)” – that’s the one. Click on it; an install button will appear. Click the install button, and wait a few minutes while the package is downloaded and installed. It will probably ask for your password, so be prepared to enter it.

2. Now do a search for ‘postgresql-contrib’. There should only be one result, “Additional facilities for PostgreSQL (supported version)” – go ahead and install it. Next do a search for “pgadmin” – this will also only yield one result, pgAdmin III, a graphical database management application. Install this as well. When you’re done, close the Software Center.

3. At this time, launch the Terminal application by clicking the Dash Home button at the top of the dash bar. The window that appears will have a search box at the top of it – if you type ‘terminal’, it will find the Terminal application for you. Enter the following command (press enter after typing all commands):
cd /etc/postgresql/9.1/main

cd = change directory; this will take you to the location where the PostgreSQL configuration files are stored. You’ll need to edit the one called postgresql.conf. Enter the command:
sudo gedit postgresql.conf

and enter your password when prompted. This will launch a graphical text editor. There are two lines in this file that need to be changed. The first is line 59, which says:
#listen_addresses = 'localhost'
Lines starting with # are comments – deleting the # forces the interpreter to pay attention to this line. So go ahead and delete it. Now find line 84 which says:
#password_encryption = on
and delete the # from this line as well. Very easy, now save the file, and then close the text editor window, which will return you to the Terminal.

4. Now it’s time to start the PostgreSQL server. Enter the following command in the Terminal:
sudo service postgresql start
and press enter. If prompted for your password, enter it (sudo invokes the superuser account, and you will have to enter your password if you haven’t done so in the past few minutes). You will see the following on the screen:
* Starting PostgreSQL 9.1 database server     [ OK ]
Congratulations, you now have a database server running on your computer! Unfortunately, you’re not quite done yet – you still need to add a user who can work with the database.

5. Enter the following command in the Terminal:
sudo -u postgres createuser
You’ll see the following prompt:
Enter name of role to add:
Enter the name of the new user. For the purposes of this exercise, I’ll be using ‘newusername’, but you should use something more relevant, like ‘dbuser’ or your own username. Next you see:
Shall the new role be a superuser? (y/n)
Not every user should be a superuser, but you need at least one person who has total access to the system, and it might as well be yourself. So type ‘y’ here and press enter.

6. Now your new user needs a password. To assign one, you’ll have to log into the database:
sudo -u postgres psql
You will see the following prompt:

psql (9.1.4)
Type "help" for help.

postgres=#

You are now inside the database, giving commands directly. Let’s not spend too long in here right now. There’s just one command to give. In my example, the username is newusername, and the password is newpassword:
alter user newusername with encrypted password ‘newpassword’;
Don’t forget the semicolon at the end – all SQL statements are terminated with a semicolon. If you forget it, the interpreter will think you’re adding more lines of code. When the job is done, you’ll see this message:
ALTER ROLE
You’re all done in here, so type:
\q
to exit the database. Restart the server by typing:
sudo service postgresql restart
and then close the Terminal.

7. Now you can get inside your database and look around using a graphical management client. During the install process, pgAdmin III should have been added to your dash bar – its icon is a blue elephant. Open pgAdmin and expand its window. On the far left of the toolbar is the Add Connection button, which looks like an electrical plug. Click that button to bring up the New Server Registration dialogue. Here’s the information you need to enter:
name: pgtest
host: localhost
port: 5432
maintenance db: postgres
username: newusername
password: newpassword
store password: checked
Use the following diagram as a guide:

Click OK when you’re done. You will probably see a warning message about stored passwords – that’s okay, because we told it to encrypt the passwords (using an md5 hash) back in step 3. You may also see a warning about server instrumentation, that’s okay. If you see a warning telling you to install an admin-pack or postgresql-contrib, that’s fine – we actually did that already, back in step 2.

At this point, your database installation is complete. Right now, you only have access to one database, called ‘postgres’. Double-clicking on the icon will connect to it, allowing you to browse through its structure of tablespaces, schemas, and catalogs. At the bottom of the tree, you should see login roles, which includes the user we created in step 5. Feel free to browse around the postgres database, but don’t modify it in any way – it’s the maintenance database, not meant for user data. You could add a user database at this point, but this tutorial won’t be getting into that until part 3. At this time, you may wish to browse the help documentation and familiarize yourself with pgAdmin.

Thus concludes this installation of the PostgreSQL database. Coming up in Step 2, we will install GEOS and GDAL, some important dependencies of the spatial database extender PostGIS.


Favor the World File

A world file is a text file that is distributed with many raster datasets. It contains six lines of instructions which describe the location, scale, and rotation of the raster image. ArcGIS can read display information from the world file – but by default, it does not. Instead, ArcGIS prefers to read this information from the raster’s spatial header, if it exists. Many of the common raster formats, including as TIFF and IMG, have spatial headers, making the world file redundant.

If you really, really want to, you can force ArcGIS to always use the world file, if it is available. You may wish to do this because you want to standardize the spatial data associated with each raster, regardless of format. A text file is easier to standardize than the spatial headers of disparate image types. However, you have to be careful about this, because the world file and the spatial header info are not always the same. Choosing to favor the world file could change the spatial properties of your archive data.

In ArcGIS 10, all you have to do to favor the world file is tick a checkbox in the ArcMap (or ArcCatalog) options. It’s a bit trickier in ArcGIS 9.x. You have to enter the Windows registry and search for the correct key. For Windows XP Professional SP3, the key is located at:
HKEY_LOCAL_MACHINE\SOFTWARE\ESRI\Raster\Preferences\filepref.favor_world_file
The default value for this key is false. Setting it to true will force ArcGIS to favor the world file. Double-click the key to change its value.


Full Color Shaded Relief

Contour maps have essentially gone out of fashion, replaced by shaded relief. There are still plenty of uses for contours, but they just don’t look as cool as a well-drawn hillshade. A colored hillshade is even more exciting. Traditionally, the easiest way to do that in ArcGIS was to place a color stretched DEM over the hillshade, and then reduce the transparency of the DEM. There are two major problems with this solution. One problem is that the transparency washes out the colors in the DEM, making the image too pale and bland. (This can be alleviated by choosing extremely bright colors, or adjusting the number of standard deviations in the stretch, but never quite solved.)The other is that when a transparent DEM over a hillshade is exported as a PDF (so that it can be edited in Adobe Illustrator), the alpha values are recalculated, making the image significantly darker.

Now, through the use of some deep image processing techniques, it is possible to brighten up your DEM overlays. The trick is to use pansharpening. Pansharpening uses a high resolution greyscale image to increase the color resolution of a multi-band color image. Don’t worry if you don’t exactly understand what’s going to happen – the wonderful folks at the ArcGIS mapping blog have put together a great tutorial. Within ten minutes, you’ll be wowing yourself with unbelievable shaded relief – without the hassles of transparency.

http://blogs.esri.com/esri/arcgis/2012/03/08/an-alternative-to-overlaying-layer-tints-on-hillshades/


Cartographic Criticism

Unless you’re working on a hobby project, as a cartographer, you will eventually be expected to show your work. And sharing means making yourself vulnerable to criticism. That word itself, ‘criticism’, sometimes conjures images of personal attacks and low self esteem, but that’s not always the case. Often, a criticism can be extremely helpful, both to you and to your organization. There are many different types of criticism, but only three of them will likely apply to your cartographic design: negative, practical, and constructive.

Negative
Example: “The legend looks terrible.”
Negative criticisms are the ones we’re really afraid of. Although they’re usually not meant to be personal, they can easily feel that way. Usually, you will only hear truly negative criticisms from people who actively dislike you, and people who are socially blunt. Most people have enough tact to offer practical or constructive criticisms. But don’t think the opinions of the tactless should be brushed away lightly. Even spiteful criticism can be helpful, if you can isolate the criticism from the spite. Poor feedback is better than nothing at all, because you can’t learn anything from nothing. For example, if someone snubs your legend, it might be a good idea to polish the legend a little, just to be safe. There’s always room for improvement, and a negative criticism can tell you where to focus for the maximum return on effort.

Practical
Example: “The legend is hard to read.”
Practical criticisms aren’t criticisms of quality, but of practicality. In a map, practicality is a function of readability and usability, so most practical criticisms will be very useful. As a cartographer, the map and the data seem quite clear to you. But an outsider, or someone seeing the map for the first time, won’t be as familiar with the material, and will be excellently positioned to see issues you hadn’t considered. Practical criticisms, even when they are blunt, are important to your work – which is why you should always find outsiders to critique your work.

Constructive
Example: “The legend looks a little crowded. Perhaps lowering the font size will help.”
A constructive criticism is a negative or practical criticism coupled with a suggestion for improvement. This format is how people are taught to discuss their work in school, because a learning environment is not the place to develop a thick skin – it’s a place to learn from mistakes. Therefore, criticism is cooled with helpful advice. The key for the professional cartographer is to separate the advice out, and focus on the criticism. In the example above, it’s possible that lowering the font size of the legend will help. But there may be other ways to uncrowd the legend, like streamlining the information presented, or repositioning text. Don’t think you are somehow bound to decrease the font size just because it was suggested constructively.

The opposite of criticism is praise. Obviously, you can never have too much of that, right? Actually, praise can be dangerous if it isn’t considered critically. If five people look at the map, and four people have a problem with the background color, are you going to trust the fifth just because her praise makes you feel better? Sometimes praise is just praise, but too much praise can mean you’re not asking the right people for feedback.

Ultimately, the goal is to make great maps, and even the best can get better. Any kind of feedback, whether, positive, negative, or neutral, can be useful to that end. So don’t be scared of criticism – seek it out! Find some friends or colleagues that you can trust to be honest, and ask for their help. And when other cartographers ask for your help, be honest but kind: give them the kind of feedback you’d like to receive.