Server-Sent Events

Sending HTML page updates from server to client

The normal PicoLisp Web GUI, running over HTTP, is client-driven: A client (typically a browser) initiates the connection, sends GET or POST requests, and receives responses from the server. HTTP by itself provides no way for the server to send anything to the client on its own.

It is often desirable, however, to display something to the user without the need of manually reloading the page.

Web Sockets

One way to do this is Web Sockets, a full-duplex protocol. It is not trivial, and I cannot tell much about it as, I have never used it.

But some people did, for example Henrik Sarvell with Websockets with PicoLisp, based on Jose I. Romero's https://bitbucket.org/iromero91/web.l/src/default/web/sockets.l library, so I won't further elaborate on it.

+Auto Buttons

The traditional way in the PicoLisp GUI to get some automatism is the ^{https://software-lab.de/doc/form/refA.html#+Auto +Auto} button prefix class. It is used in PicoLisp applications since many years.

An +Auto +Button causes the current form to auto-update periodically - essentially polling the server at regular intervals. This is not very efficient, but quite smooth as it refreses the form contents at the JavaScript level.

The following clock function shows a page with a time field and an auto button which starts after a quarter second, and then updates the form roughly every second:
   (de clock ()
      (action
         (html 0 "Clock" "@lib.css" NIL
            (form NIL
               (gui '(+TimeField) 8)
               (gui '(+Click +Auto +Button) 250 'This 999 '(pop *Throbber)
                  '(set> (field -1) (time)) ) ) ) ) )
See below for a complete program listing.

+Auto is fine if an action needs to be repeated anyway, and it has the advantage of being fully integrated into the form GUI framework, updating the complete state on the server (variables, GUI components, database etc.).

And: It also works if JavaScript is not available (e.g. in text browsers or web scrape clients), if the user presses the button manually, or the script triggers the button explicitly.

Server-Sent Events

Server-Sent Events are an elegant, efficient and asynchronous way to send data to clients.

They are available in PicoLisp since version 18.5.23.

They are a lot simpler than Web Sockets - however only one-way, while Web Sockets are bi-directional. The advantage of the latter does not outweigh the first in my opinion, as the opposite direction (from client to server) is available in HTTP anyway.

The API consists of just two functions: 'serverSentEvent' and 'serverSend', usable inside the standard PicoLisp GUI. They don't need a form environment, a plain HTML session is enough.

The serverSentEvent Function

'serverSentEvent' is embedded at an arbitrary place in the page as:
   (serverSentEvent "id" 'var . prg)
The first argument should be the ID of a DOM component in the page. Later calls to serverSend will update this component. The ID should be unique within this program session for all serverSentEvent use cases, as it is also used as a key to an internal association table.

The second argument var is a variable which is automatically bound to a socket as soon as the client establishes the event stream connection, i.e. shortly after the page with the embedded call to serverSentEvent is displayed. In essence, this socket is then connected to the DOM component "id".

When that happened, the prg body is executed to set up everything needed for further event sending via the socket in var.

var is automatically set to NIL, and the socket closed, when the client closes the connection. Then the server should clean up whatever is necessary.

The serverSend Function

When the event stream connection is established from the client, and prg is executed, serverSend may be called anytime while the connection is open:
   (serverSend 'sock . prg)
It sends all output (HTML text) generated by prg to the inner HTML of the component connected to the socket sock.

An Interactive Test

Server-Sent Events can be tested interactively in the PicoLisp REPL.

Create an empty <div> and connect it to a global variable *Sse:
   (<div> '(id . "sseDiv"))
   (serverSentEvent "sseDiv" '*Sse)
The prg body is empty here, no initialization needed.

Then send some HTML code from the REPL:
   : (serverSend *Sse (<h1> NIL "Hello" (<nbsp>) (<em> "World") "!"))


sse1.png

The full source listing is:
   #!/data/data/com.termux/files/usr/bin/pil -load "@lib/http.l" "@lib/xhtml.l"
   # 03aug18abu

   (de sseTest ()
      (html 0 "SSE" "@lib.css" NIL
         (<div> '(id . "sseDiv"))
         (serverSentEvent "sseDiv" '*Sse) ) )

   (de work ()
      (app)
      (timeout)
      (redirect (baseHRef) *SesId "!sseTest") )

   (server 8080 "!work")
Adjust the first line for the path to your environment's pil interpreter. A typical value on Unix is #!/usr/bin/pil ... (the above is for Termux on Android).

sseTest generates a page with the above empty <div>.

work is a helper function starting a session with (app). It redirects immediately to sseTest, and is only needed for an application like this, which needs JavaScript already on the very first page (to satisfy the Same-Origin-Policy, because (app) allocates a new port for the session).

work calls (timeout) without arguments in the new session, effectively disabling timeouts. This is also normally not needed, but useful here so that the session does not close while idling in the REPL.

PilBox

In PilBox it is even easier. Assuming you have the Android App installed, you do in Termux:
   $ mkdir sse
   $ cat >sse/App.l
   "Server-Sent Events"

   (on *Repl)

   (menu "Server-Sent Events"
      (<h1> NIL "Server-Sent Events")
      (<div> '(id . "sseDiv"))
      (serverSentEvent "sseDiv" '*Sse) )
   <Ctrl-D>
   $ zip -rFS sse.zip sse/App.l
     adding: sse/App.l (deflated 37%)
   $ termux-share sse.zip
PilBox should start up.

The line (on *Repl) activates the REPL in this App, so that it is not necessary to navigate to the REPL page first to click the checkbox.

You just swipe to the REPL on the right side and call (serverSend *Sse ...).

Simple Clock Display

Revisiting the above clock example with +Auto +Button, we can do something similar with Server-Sent Events:
   (de clock ()
      (html 0 "Clock" "@lib.css" NIL
         (<div> '(id . "clockDiv"))
         (serverSentEvent "clockDiv" '*ClockSock
            (task -960 0  # More than once per second
               (if *ClockSock
                  (serverSend *ClockSock
                     (<h2> NIL (prin (tim$ (time) T))) )
                  (task -960) ) ) ) ) )
After *ClockSock is connected, we start a background task every 960 seconds which sends the current time as a string to the clockDiv element. After *ClockSock became NIL, the task stops.

Again, see below for the full source code.

SSH Terminal

As a more involved example, let's look at a "chat" with a remote shell via SSH. Not really useful in itself, but it shows how to handle asynchronous messages in a chat-like system.

We use a <pre> element to show the text in a global variable *ChatTxt in monospaced font:
   (<pre>
      '((id . "msg")
         (style . "height:48em; font-family: monospace") )
      (ht:Prin *ChatTxt) )
serverSentEvent connects it to a global variable *ChatSock and starts a task which opens a pipe to the ssh utility, and stores the file descriptor in a global variable *Ssh.

All incoming characters are appended to *ChatTxt, replacing newlines with <br> tags, and tabs with three spaces. The length of the text is limited to 40 lines, and sent to the chat socket with serverSend:
   (serverSentEvent "msg" '*ChatSock
      (off *ChatTxt)
      (zero *ChatLen)
      (task (setq *Ssh (pipe (exec "ssh" "-tt" *User@Host)))
         (in @
            (if (and *ChatSock (not (eof)))
               (let? C (char)
                  (unless (= C "r")
                     (queue '*ChatTxt
                        (case C
                           ("n"
                              (if (= *ChatLen 39)
                                 (until (= "<br>" (++ *ChatTxt)))
                                 (inc '*ChatLen) )
                              "<br>" )
                           ("t" "   ")
                           (T C) ) )
                     (serverSend *ChatSock
                        (ht:Prin *ChatTxt) ) ) )
               (task (close *Ssh))
               (off *Ssh) ) ) ) )
Commands to the remote shell can be entered into a +TextField, and sent by pressing Enter or the "Chat" button:
   (gui '(+Focus +TextField) 80)
   (gui '(+JS +Button) "Chat"
      '(when (and *Ssh (val> (field -1)))
         (out *Ssh (prinl @))
         (clr> (field -1)) ) )
Again, the complete source code is appended below.

The script expects a single argument of the form "user@host" for the SSH session. The login is best provided for with proper key files on the remote machine, otherwise you need to type the password in the terminal where the script was started.

sse2.png

Source listings

+Auto Button

   #!/data/data/com.termux/files/usr/bin/pil
   # 03aug18abu

   (allowed ()
      "!work" "!clock" "@lib.css" )

   (load "@lib/http.l" "@lib/xhtml.l" "@lib/form.l")

   (de clock ()
      (action
         (html 0 "Clock" "@lib.css" NIL
            (form NIL
               (gui '(+TimeField) 8)
               (gui '(+Click +Auto +Button) 250 'This 999 '(pop *Throbber)
                  '(set> (field -1) (time)) ) ) ) ) )

   (de work ()
      (app)
      (redirect (baseHRef) *SesId "!clock") )

   (server 8080 "!work")


Simple Clock Display

   #!/data/data/com.termux/files/usr/bin/pil
   # 03aug18abu

   (allowed ()
      "!work" "!clock" "@lib.css" )

   (load "@lib/http.l" "@lib/xhtml.l")

   (de clock ()
      (html 0 "Clock" "@lib.css" NIL
         (<div> '(id . "clockDiv"))
         (serverSentEvent "clockDiv" '*ClockSock
            (task -960 0  # More than once per second
               (if *ClockSock
                  (serverSend *ClockSock
                     (<h2> NIL (prin (tim$ (time) T))) )
                  (task -960) ) ) ) ) )

   (de work ()
      (app)
      (redirect (baseHRef) *SesId "!clock") )

   (server 8080 "!work")


SSH Terminal

   #!/data/data/com.termux/files/usr/bin/pil
   # 03aug18abu

   ## Usage:
   ## misc/sshChat user@host

   (setq *User@Host (opt))

   (allowed ()
      "!work" "!chat" "@lib.css" )

   (load "@lib/http.l" "@lib/xhtml.l" "@lib/form.l")

   (de chat ()
      (action
         (html 0 "Chat" "@lib.css" NIL
            (form NIL
               (<pre>
                  '((id . "msg")
                     (style . "height:48em; font-family: monospace") )
                  (ht:Prin *ChatTxt) )
               (serverSentEvent "msg" '*ChatSock
                  (off *ChatTxt)
                  (zero *ChatLen)
                  (task (setq *Ssh (pipe (exec "ssh" "-tt" *User@Host)))
                     (in @
                        (if (and *ChatSock (not (eof)))
                           (let? C (char)
                              (unless (= C "r")
                                 (queue '*ChatTxt
                                    (case C
                                       ("n"
                                          (if (= *ChatLen 39)
                                             (until (= "<br>" (++ *ChatTxt)))
                                             (inc '*ChatLen) )
                                          "<br>" )
                                       ("t" "   ")
                                       (T C) ) )
                                 (serverSend *ChatSock
                                    (ht:Prin *ChatTxt) ) ) )
                           (task (close *Ssh))
                           (off *Ssh) ) ) ) )
               (--)
               (gui '(+Focus +TextField) 80)
               (gui '(+JS +Button) "Chat"
                  '(when (and *Ssh (val> (field -1)))
                     (out *Ssh (prinl @))
                     (clr> (field -1)) ) ) ) ) ) )

   (de work ()
      (app)
      (redirect (baseHRef) *SesId "!chat") )

   (server 8080 "!work")

https://picolisp.com/wiki/?serversentevents

07aug18   abu