Multiple asynchronous HTTP POSTs and 'call' versus 'in' - a word of warning

If you've been using PicoLisp for some time doing nitty gritty stuff like interfacing with such horrible things as PHP and other abominations you can probably infer from the headline that the pitfall here is quite trivial. However, it's the journey that is the important part, not the somewhat anticlimactic end to this whole adventure. Having said that, let's get to it.

In PicoLisp there is a clever way of asynchronous evaluation, it can be useful if you:

1. Have a very computationally heavy problem, you fork the execution to utilize multiple cores and all that cheap RAM.
2. Need to query X services and don't care to wait for each call to finish before making the next one.

This "strategy" involves later and wait, see documents for later in particular.

My interest is web development so #2 is standard for me and up until now I've used sockets in combination with pr and rd. The "standard" PicoLisp way to do these things if you will.

However, at the moment I'm working on something new and I want to work solely with HTTP, not with PLIO, despite it being faster/more efficient than HTTP. Let's just say that HTTP is the only allowed way of communication in my "spec".

So I went happily on my way coding up all this communication with the help of CURL and the call function.

Let's have a code listing:

(de callAll (Func Data)
   (let Result
      (make
         (for S (collect 'id '+Server)
            (later (chain (cons "void"))
               (list (; S id) (callOne Func Data S)))))
      (wait 30000 (not (memq "void" Result)))
      Result))

Here you can see the core of the strategy, we loop through a list of nodes (+Server) to query each and every for some arbitrary data. What you don't see in the above listing is the contents of callOne but it doesn't matter, we'll get to that soon enough. In essence what we're doing here is getting all the nodes in my massive distributed database (there are at the time of this writing two of them running on my laptop but whatever).

These nodes might be more or less busy doing other stuff so response times might vary and can you imagine if the first one takes 10 seconds to reply and the second one takes 5 seconds? In a non-asynch world that would add up to a grand total of 15 seconds for the query.

Luckily we use later and wait to avoid that and query them in parallel for a grand total of 10 seconds. What's happening here is that we add the results to a list, each entry in the list will start with "void", then we fork. Each fork will return a result (or not but we'll get to that very very soon).

The second (list ... ) argument there is supposed to return a list with the id of the node in the car and the result of the query in the cdr. If a node times out one or more positions in the list will still be "void" instead of something useful.

Finally the wait call will wait for 30 seconds or until all nodes have returned something (i.e not timed out). After that wait we return the list which I expected would look something like this for my specific case (counting the objects in each node): ((1 "1") (2 "0")).

The problem was, the result didn't look like that, it looked more like (NIL NIL). I was using call like this: (call 'curl '-m 30 '-d "key1=value1&key2=value2" "http://localhost:8081/@exec"), the reason for that is ye old copy paste problem, not even that in this case actually. Just checking code from an old project and seeing call in action without realizing that in that case it was actually used correctly (I wasn't interested in the returned content).

Yep, call returns T, not the actual result which makes it useless in this case. I now use in, like this: (in (list "curl" "-m" Timeout "-d" "key1=value1&key2=value2" "http://localhost:8081/@exec") (till NIL T) ) and all is well.

So now you don't have make the same mistake and spend two hours of your life before you actually RTFM. However, by now you should've realized that asynchronous logic can be indispensible when you don't want to wait for too long for something good.

http://picolisp.com/wiki/?asynchronousstory

29jul11

hsarvell