bin/ssl - client-side SSL tool in PicoLisp distribution

3 ways to use bin/ssl:

1. Standard HTTPS GET:
   bin/ssl host [-]port url [+]
2. One-shot file transfer over HTTPS, assumes httpGate and PicoLisp process on server side:
   bin/ssl host [-]port url [[[-]key] [-]file] [+]
3. Daemon process to do asynchronous DB replication or asynchronous transfer of arbitrary data:
   bin/ssl host [-]port url [-]key journal blobdir interval [timeout] [+]

Also note:

• "-" before port number argument disables server certificate checking e.g. if you have self-signed certificate
• "+" as last parameter switches on debug mode (i.e. prints errors and doesn't fork)
• "-" before file and key causes the parameter to be treated as a plain string instead of a file name. That string (with the leading hyphen stripped) will be sent instead of file content. No escaping is supported by the program itself.
• key is a file containing a long random string (shared secret) followed by newline character. This is for client authentication.
• interval sets the time interval in seconds between checks of journal file and also time interval between SSL connections retries
• timeout in minutes. If this parameter is set, bin/ssl exits after specified time of inactivity; otherwise it will run infinitely.

However, main purpose of bin/ssl is asynchronous database replication over SSL, which means working in concert with bin/httpGate and bin/replica.

Example of startup script for bin/ssl

#!/bin/bash

echo "Starting ssl"

#Here we assume several DB-applications on the same host

test -z $(pgrep -u app -f "bin/ssl .* foo.r") && bin/ssl <host> 443 'foo.r/!replica' key/<xxx> fifo/foo blob/foo/ 20
test -z $(pgrep -u app -f "bin/ssl .* bar.r") && bin/ssl <host> 443 'bar.r/!replica' key/<xxx> fifo/bar blob/bar/ 20

Furthermore you can use inotifywait to automatically (re)start bin/ssl processes (see inotify-tools package). Then you can use timeout parameter to reduce the number of idle bin/ssl processes, e.g. if you have a large number of databases on the host:

#!/bin/bash

if test -z $(pgrep -u app -x inotifywait)
then
   echo inotifywait
   inotifywait -mq  -e modify  --format '%f' --exclude beat  fifo |
   while read F
   do
      if test -z $(pgrep -u app -f "bin/ssl .* $F.r")
      then
         bin/ssl <remoteserver> 443 $F.r/!replica key/thisserverkey fifo/$F db/$F/blob/ 20 60
      fi
   done &
fi

Whole picture

                     journal1                          app1
+-------------+     +--------+      +------------+    +-----------+   +---+
| inotifywait +-+-->+ bin/ssl+----->|bin/httpGate+-+->|bin/replica+-->|DB1|
+--+----------+ |   +--------+      +------------+ |  +-----------+   +---+
   |            |              INET                |
   v                 journal2                      |   app2
+--+--------+   |   +--------+                     |  +-----------+   +---+
|DB journals|   +-->+ bin/ssl+----->               +->|bin/replica+-->|DB2|
|directory  |   |   +--------+                        +-----------+   +---+
+-----------+   |
                +-->...

Asyncronous replication under the hood:

1. The script above should be started once at server startup.
2. It starts inotifywait, that monitors fifo/ folder, where journals for all databases reside.
3. When (commit) in PicoLisp application writes changes to the end of appropriate journal file, inotifywait starts dedicated bin/ssl process for that journal, if it's not yet started.
4. bin/ssl runs as daemon process and periodically checks if 'journal size > 0'.
5. If journal size is not zero, bin/ssl locks the journal, reads the whole file into memory, truncates it and with specified interval tries to send all data to the server (httpGate <--> bin/replica).
6. If the journal data has been successfully sent, bin/ssl tries to send every file from the directory specified in blobdir parameter (they are symlinks to real blob-files). If the file has been successfully sent, it will be deleted.

Data format used for journal sending (from 'ssl.c'):

if ((sd = sslConnect(ssl, av[1], av[2])) >= 0) {
                       if (SSL_write(ssl, get, getLen) == getLen  &&
                           (!*av[4] || sslFile(ssl,av[4]))  &&       // optional key
                           SSL_write(ssl, len, lenLen) == lenLen  && // data length
                           SSL_write(ssl, Data, Size) == Size  &&    // raw data
                           SSL_write(ssl, "T", 1) == 1  &&           // acknowledge ('T' character)
                           SSL_read(ssl, buf, 1) == 1  &&  buf[0] == 'T' ) { // we should receive 'T'

For blobs:

                  if ((sd = sslConnect(ssl, av[1], av[2])) >= 0 ) {
                     if (SSL_write(ssl, get, getLen) == getLen  &&
                           (!*av[4] || sslFile(ssl,av[4]))  &&       // key
                           SSL_write(ssl, buf, n) == n  &&           // path
                           SSL_write(ssl, "n", 1) == 1  &&          // nl
                           sslFile(ssl, nm) )                        // file

It's also possible to transfer arbitrary data in the same manner. E.g. you can write data with (pr) to file on client side and read data with (rd) on server side. If you don't need to send files, you can pass "" as blobdir parameter.

Example of server side code:

(when (fetch '(key . +Dev) (line T)) # find shared secret from DB
   (let? X (line T)
      (if (format X)
         (when (abort 420 (out (tmp 'rpc) (echo @)))  # Commands
            (prin (peek))
            (flush)
            (in (tmp 'rpc)
               (while (rd)
                  (eval @) ) ) )
         (let File (pack *Data X)                     # Data
            (call 'mkdir "-p" (dirname File))
            (out File (echo)) ) ) ) ) ) ) )

https://picolisp.com/wiki/?ssl

23nov15

rick