Using OpenStreetMap Geo-Data with PicoLisp

Since quite some time the PicoLisp database supports multi-dimensional indexes via the '+UB' prefix class.

This lends itself to experimenting with geographic information systems (GIS), especially as vast amounts of free data are available in the 'OpenStreetMap' project, and we have a convenient JavaScript Canvas library in PicoLisp (see Simple Canvas Drawing).

The following example shows a complete OSM application. It is not really useful by itself - you can easily find more complete and useful things in OSM and other Geo systems. But it is a system which is completely under your control (100% PicoLisp), not so big that it can't be understood (just 448 lines of code), and it demonstrates some advantages of PicoLisp programming like DB/GUI integration and server-side JavaScript/GUI programming (no need to touch (and/or debug!) any app-specific JavaScript code).

Last but not least it can be used as a starting point for a production application.

Runtime Environment Setup

PicoLisp Runtime

Without going into the details (see https://software-lab.de/INSTALL), I assume you have a global PicoLisp installation running on your system. The following application requires an up-to-date 64-bit version of PicoLisp, so for the sake of clarity let's start from scratch:

   $ cd /tmp                              # Go to the /tmp directory
   $ wget https://software-lab.de/picoLisp.tgz    # Fetch the latest tarball
   $ tar xfz picoLisp.tgz                 # Extract it
   $ cd picoLisp                          # Go to the extracted directory
   $ (cd src64; make)                     # Build the 64-bit version

Here is a short video of the process:



Instead of /tmp you can use any directory you have write-access to.

If you have only a 32-bit machine, use (cd src64; make emu) to build the emulator version of 64-bit PicoLisp. This will work just as fine, but is a bit slower.

OpenStreetMap Sample Data

Visit http://www.openstreetmap.org to obtain map data. Pressing the "Export" button will download a file named "map.osm".

Fetching a partial map of Berlin:



The contents of "map.osm"

   $ view ~/Downloads/map.osm

are XML data of the form

   <?xml version="1.0" encoding="UTF-8"?>
   <osm version="0.6" generator="CGImap 0.3.3 ...
    <bounds minlat="52.5570000" minlon="13.3732000" maxlat="52.5854000" maxlon="13.4219000"/>
    <node id="21537085" visible="true" ... lat="52.5748141" lon="13.3746242"/>
    <node id="21537796" visible="true" ... lat="52.5747614" lon="13.3730684"/>
    <node id="21537822" visible="true" ... lat="52.5729236" lon="13.3671350"/>
    <node id="21538153" visible="true" ... lat="52.5743796" lon="13.3686259"/>
    <node id="21540762" visible="true" ... lat="52.5749761" lon="13.3802028">
     <tag k="highway" v="traffic_signals"/>
    </node>
   ...
    <way id="4071972" visible="true" version="20" changeset="17953455" timestamp="...
     <nd ref="21538153"/>
     <nd ref="92893493"/>
     <nd ref="92893494"/>
     <nd ref="751779784"/>
     <nd ref="208131832"/>
     <nd ref="751778357"/>
     <nd ref="21537796"/>
     <nd ref="1936531000"/>
     <tag k="highway" v="tertiary"/>
     <tag k="lit" v="yes"/>
     <tag k="maxspeed" v="50"/>
     <tag k="name" v="Klemkestra├če"/>
     <tag k="postal_code" v="13409"/>
    </way>
   ...
   </osm>

consisting - among other data - of "nodes" and "ways". Roughly speaking, a node is a point, and a way is a number of interconnected points.

The OpenStreetMap Application

Now proceed to install the application (continuing in the same directory above):

   $ wget https://software-lab.de/osm.tgz
   $ tar xfz osm.tgz

Look at the model representing a subset of the OSM data, by editing "osm/er.l":

   $ view osm/er.l

OSM nodes are represented in a straightforward way:

   (class +Nd +Entity)
   (rel id (+Key +Number))                         # Node-ID
   (rel nm (+Fold +Ref +String))                   # Name
   (rel lt (+UB +Aux +Ref +Number) (ln) NIL 7)     # Latitude + 90.0
   (rel ln (+Number) 7)                            # Longitude + 180.0
   (rel nb (+List +Joint) nb (+Nd))                # Neighbours
   (rel w (+List +Joint) nd (+Way))                # Ways

The id is taken directly from the XML attribute, and the name nm may appear (optionally) in a tag attribute.

The latitude lt (from the lat attribute) and the longitude ln (from the lon attribute) are combined into a single two-dimensional index with the '+UB' and '+Aux' prefix classes.

The list of neighbours nb of this node, and the list of ways w where this node is member of, is created during the import of the way data.

OSM ways are represented by:

   (class +Way +Entity)
   (rel id (+Key +Number))                         # Way-ID
   (rel nm (+Fold +Ref +String))                   # Name
   (rel nd (+List +Joint) w (+Nd))                 # Nodes

Here, too, we have a unique id and an optional name nm. The list nd holds the member nodes of this +Way.

Now let's go ahead and start the application.

   $ ./pil osm/main.l -main -go +

It will listen on port 8080 by default.

We open localhost:8080 with the browser, and log in with the user name "osm" and the password "osm".

Inspecting the nodes and ways, we see that they are still empty. So we click on "Import", upload our previously downloaded "map.osm", and click on "Start". After a few seconds all nodes and ways from the OSM file are imported.



OSM GUI

The application comes with a simple GUI to visualize the data. As example I pick a way ("Heinrich-Mann-Stra├če" in Berlin).

Ways are presented (from "osm/way.l") with the standard PicoLisp GUI components. On top, below the navigation panel, is a numeric field for the ID and a text field for the name.

The canvas area shows all nodes and ways in the selected range. The way itself is marked as a red line.

Clicking with the mouse in the canvas area will move the center (the faint cross in the middle) to that point. Dragging the mouse allows continuous scrolling. The four buttons <, >, v and ^ below the canvas component shift the view area left, right, down and up, respectively. In all cases the two numeric fields (latitude and longitude of the center) are updated.

A double-click zooms into the picture, it is equivalent to a single-click followed by pressing the + button. The - button zooms out. Above a certain limit, the ways are not drawn any more, and only the nodes are plotted.



GUI Code

The GUI forms for nodes ("osm/nd.l") and ways are similar, so I focus on ways ("osm/way.l") here.

   # 06dec13abu
   # (c) Software Lab. Alexander Burger

   (must "Way" GeoData)

   (menu ,"Way"
      (idForm ,"Way" '(choWay) 'id '+Way T '(may Delete) '((: nm))
         (<grid> 2
            "ID" (gui '(+E/R +NumField) '(id : home obj) 8)
            ,"Name" (gui '(+E/R +Cue +TextField) '(nm : home obj) ,"Way" 40) )
         (osmCanvas
            (let L (mapcar get (: obj nd) '(lt .))
               (/ (+ (apply min L) (apply max L)) 2) )
            (let L (mapcar get (: obj nd) '(ln .))
               (/ (+ (apply min L) (apply max L)) 2) )
            (: obj nd) )
         (gui '(+E/R +Chart) '(nd : home obj) 6
            '((This) (list NIL This (: lt) (: ln)))
            cadr )
         (<table> NIL NIL
            '((btn) (NIL ,"Nodes") (align ,"Latitude") (align ,"Longitude"))
            (do 8
               (<row> NIL
                  (choNd 1)
                  (gui 2 '(+Obj +TextField) '(nm +Nd) 30)
                  (gui 3 '(+LatField) 11)
                  (gui 4 '(+LonField) 11)
                  (gui 5 '(+DelRowButton))
                  (gui 6 '(+BubbleButton)) ) ) )
         (<spread> (scroll 8 T) (editButton T)) ) )

   # vi:et:ts=3:sw=3

must is the obligatory permission check, and menu in combination with idForm sets up the standard form infrastructure.

Then follow the "ID" and "Name" fields mentioned above, the osmCanvas for the two-dimensional drawing, and finally a chart with the nodes of the current +Way.

osmCanvas is the interesting part. It calls the HTML <canvas> function, and creates the shift- and zoom-buttons, and the latitude/longitude fields described above. It is defined in "osm/draw.l":

   (de osmCanvas (Lat Lon Nd)
      (--)
      (<canvas> "$osmView" (osmView (: dx)) (osmView (: dy)))
      (javascript NIL "onload=drawCanvas('$osmView', -1)")
      (unless *PRG
         (osmView
            (=: home *Top)
            (=: scl 128)
            (=: lt Lat)
            (=: ln Lon)
            (=: nd Nd) ) )
      (--)
      (<spread>
         (prog
            (gui '(+Rid +Button) "<"
               '(osmView (dec (:: ln) (>> -7 (: scl)))) )
            (gui '(+Rid +Button) ">"
               '(osmView (inc (:: ln) (>> -7 (: scl)))) )
            (gui '(+Rid +Button) "v"
               '(osmView (dec (:: lt) (>> -7 (: scl)))) )
            (gui '(+Rid +Button) "^"
               '(osmView (inc (:: lt) (>> -7 (: scl)))) )
            (<nbsp> 3)
            (gui '(+Rid +Able +Button) '(osmView (> (: scl) 1)) "+"
               '(osmView (=: scl (>> 1 (: scl)))) )
            (gui '(+Rid +Button) `(char (hex "2014"))
               '(osmView (=: scl (>> -1 (: scl)))) ) )
         (gui 'lt '(+Rid +Upd +LatField) '(osmView (: lt)) 11)
         (gui 'ln '(+Rid +Upd +LonField) '(osmView (: ln)) 11)
         (gui '(+Rid +Button) "Go"
            '(osmView
               (=: lt (val> (: home lt)))
               (=: ln (val> (: home ln))) ) ) )
      (----) )

osmCanvas takes three arguments: The latitude and longitude of the initial center of view, and an optional list of nodes for the red path of the +Way.

Recall from the +Way GUI above

   (osmCanvas
      (let L (mapcar get (: obj nd) '(lt .))
         (/ (+ (apply min L) (apply max L)) 2) )
      (let L (mapcar get (: obj nd) '(ln .))
         (/ (+ (apply min L) (apply max L)) 2) )
      (: obj nd) )

This calculates the center of the way (the midpoint of the minimal and maximal latitude, and the midpoint of the minimal and maximal longitude), and then passes the nd list of the +Way object.

Canvas Drawing

At the heart of the PicoLisp canvas library (in "@lib/canvas.l" and "@lib/canvas.js") is the function drawCanvas. It is implicit in all drawing, both in the Lisp- and in the JavaScript-part. There is no other way. If you want to draw on a canvas with these libraries, you need to define a Lisp function with the name drawCanvas.

A minimal example, drawing just a single rectangle, could be

   (de drawCanvas ()
      (make
         (csStrokeRect 100 100 400 300) ) )



drawCanvas takes up to seven optional arguments:
  1. The HTML-ID of the canvas component. This can be used to differentiate between several canvasses on the page
  2. The delay (in milliseconds) which was passed to JavaScript for auto-refresh (animation)
  3. A numeric flag which specifies the type of click (see the comment in "@lib/canvas.l"):
    • 1 = Single click
    • 2 = Double click
    • 0 = Start (drag)
    • -1 = Move (drag)
  4. X coordinate of the click
  5. Y coordinate of the click
  6. X coordinate of the move
  7. Y coordinate of the move


For our OSM application, drawCanvas is defined in "osm/draw.l".

   (de drawCanvas (This Dly F X Y X2 Y2)
      (make

Check for single click, and set the center in ln and lt,

         (cond
            ((gt0 F)  # Click
               (=: ln (lon/x X))
               (=: lt (lat/y Y))

then also check for double-click (F is 2)

               (and
                  (= 2 F)
                  (> (: scl) 1)
                  (=: scl (>> 1 (: scl))) )
               (csPost) )

and set the scale scl to half its value unless it is 1 already.

Check for move (drag) and adjust the coordinates accordingly:

            ((le0 F)  # Drag
               (when (=0 F)
                  (setq *CsMvX X  *CsMvY Y) )
               (dec (:: ln) (* (: scl) (- X2 *CsMvX)))
               (inc (:: lt) (* (: scl) (- Y2 *CsMvY)))
               (setq *CsMvX X2  *CsMvY Y2)
               (csPost) ) )



Now the actual drawing takes place. First the area is cleared, and the center cross is drawn,

         (csClearRect 0 0 (: dx) (: dy))
         (csStrokeStyle "black")
         (csLineWidth "0.25")
         (csStrokeLine (- (: dx/2) 12) (: dy/2) (+ (: dx/2) 12) (: dy/2))
         (csStrokeLine (: dx/2) (- (: dy/2) 12) (: dx/2) (+ (: dy/2) 12))
         (csLineWidth 1)

then all nodes in the current view are collected from the database:

         (let Lst
            (collect 'lt '+Nd
               (list (lat/y (: dx)) (lon/x 0))
               (list (lat/y 0) (lon/x (: dx))) )



If the scale is too big, only nodes are plotted,

            (if (>= (: scl) 1024)
               (csDrawDots 1 1
                  (make
                     (for Nd Lst
                        (link (x/lon (; Nd ln)) (y/lat (; Nd lt))) ) ) )

otherwise all neighbouring nodes are interconnected by lines

               (csBeginPath)
               (for Nd Lst
                  (for Nb (; Nd nb)
                     (unless (memq Nd (get Nb NIL))
                        (push (prop Nd NIL) Nb)
                        (csLine
                           (x/lon (; Nd ln))
                           (y/lat (; Nd lt))
                           (x/lon (; Nb ln))
                           (y/lat (; Nb lt)) ) ) ) )
               (csStroke)
               (for Nd Lst
                  (put Nd NIL NIL) )

and then, if the scale is not too small, we mark named nodes, and write their names in green color

               (when (>= 128 (: scl))
                  (csFillStyle "green")
                  (csDrawDots 2 2
                     (make
                        (for Nd Lst
                           (unless (; Nd nb)
                              (link (x/lon (; Nd ln)) (y/lat (; Nd lt))) ) ) ) )
                  (for Nd Lst
                     (when (; Nd nm)
                        (csFillText @ (x/lon (; Nd ln)) (y/lat (; Nd lt))) ) ) )



Finally - if we have a list of nodes (i.e. a way) - it is drawn in red:

               (when (: nd)
                  (csStrokeStyle "red")
                  (csLineWidth 3)
                  (csBeginPath)
                  (csMoveTo (x/lon (: nd 1 ln)) (y/lat (: nd 1 lt)))
                  (for Nd (cdr (: nd))
                     (csLineTo (x/lon (; Nd ln)) (y/lat (; Nd lt))) )
                  (csStroke) ) ) ) ) )



All this is just a quick and short overview, though there is not much more code involved. Try it yourself! For details, study the sources and ask in #picolisp (IRC) and in the mailing list.

https://picolisp.com/wiki/?osmgeodata

09apr17   erik