pup_event InterProcess CommunicationPage updated February 23, 2018, partially updated October 25, 2019pup_event is the generic name that I have given to the underlying ...how to describe it... code that does everything "under the bonnet", in the background, that makes your pup run nice. This page describes an implementation of InterProcess Communication in Puppy and derivatives, that falls under the pup_event umbrella. So, this can be known as the "pup_event IPC" technique, to give it a name when referenced or discussed elsewhere. This was introduced to Puppy mid-2013. The method chosen is server-less, that is, works peer-to-peer. Applications send and receive messages between themselves directly, without an intermediate server or daemon. However, some applications can provide information that other applications request, and I use the word server in reference to them. I also refer to applications as clients of the IPC mechanism. The pup_event IPC mechanism offers various services to client applications:
pup_event_ipcThis is a CLI (commandline) application named 'pup_event_ipc'. It is designed to be used in shell scripts, acting as an intermediary to the pup_event IPC mechanism, enabling scripts to pass messages between each other. It is located at /sbin/pup_event_ipc.Usage of pup_event_ipc is shown by examples in the rest of this document. You can also run it like this to see some usage information: # pup_event_ipc --helpThe main pup_event daemon 'pup_event_frontend_d' is written in C. 'pup_event_ipc' is written in BaCon (BASIC to C translator), and the source code can be found in the 'pup-tools' tarball. Look here: http://distro.ibiblio.org/easyos/source/alphabetical/p/ Note, the source is also available in woofQ, the "Woof" builder for EasyOS and Quirky Linux. The code can be examined to see how any other application can access the IPC mechanism. The commandline format of pup_event_ipc is this: # pup_event_ipc "request:client[:message]" [-t <n>] -t <n>: Specifies timeout in milliseconds. Timeout will return with value=1.*The names for clients need to be unique. Choose anything you want, perhaps prefixing all names with your initials, for example John Smith could prefix with "js_". Message formatsThis section describes message formats. This is already introduced above, as:"request:client[:message]"The various permutations of this are described below. Synchronous message-swappingDemonstrating synchronous message-swap between two different applications.In one terminal window I type this: # pup_event_ipc "js_app1:js_app2:some stuff from app2"In another terminal window I type this: # pup_event_ipc "js_app2:js_app1:data from app1"After launching both of them: First window: # pup_event_ipc "js_app1:js_app2:some stuff from app2"Second window: # pup_event_ipc "js_app2:js_app1:data from app1"This is a very simple mechanism that can be use by two applications to perform a synchronization, or to pass messages between each other. It is a simple swap, and both sides must post to the other. If only one side posts, the message will remain in the "mailbox" until the other side posts. The utility 'pup_event_ipc' blocks, that is, waits, until the swap is completed, before exiting. Note, the CLI utility 'pup_event_ipc' does have a timeout option (-t n), so either side can terminate before the swap has completed. Mailbox (asynchronous) message passingWhen a client posts a message to another client, it goes first into the IPC "mailbox", later it gets delivered to the other client.With asynchronous message passing, you don't send the message to the other client, you send it to the mailbox only, then the sending-client is free to do other stuff. The other client, to which you would like your message to go, can at any time query "is there a message waiting for me?" and if so, fetch it. The format of messages is: "mailbox:client[:message]"If a client posts a message with this format, with the third-field message-string, it gets posted, and the server acknowledges. For example: # pup_event_ipc "mailbox:js_app1:message string posted to app1"A client, any client, can ask if there is mail waiting for "js_app1". This is done by having no third field. For example: # pup_event_ipc "mailbox:js_app1"However, if this is done again: # pup_event_ipc "mailbox:js_app1"That last example returns immediately, but if you want to wait until a message arrives at the mailbox, use this variant: # pup_event_ipc "waitmail:js_app1"The utility will not exit until there is a message delivered. But of course, the pup_event_ipc utility does support a timeout: # pup_event_ipc "waitmail:js_app1:" -t 4000Synchronous message-swapping and asynchronous mailbox message-passing, are the two peer-to-peer messaging and synchronization mechanisms. There is another group, that is a bit more like a server-client arrangement, in which the 'request' field of the message requests something from another application. That other application will be serving a certain kind of information. Although I don't really want to consider these as servers, as they are just equal clients in the IPC mailbox system, for the sake of putting a name on them, I am referring to this group of applications as "server applications". Server applications are documented below. Currently, the 'pup_event-frontend_d' server offers block, network, x, timeout1, timeout4 and timeout60. Block hotplug eventsA client application may query the "kernel-event server application" (pup_event_frontend_d) for various information, such as detecting plugin/removal of USB or card drives.Consider the block event. For example: # pup_event_ipc "block:js_app1"The utility waits until a drive hotplug event occurs, such as a USB Flash stick plugged in, then returns with the information. The 'pup-event_ipc' utility is only designed as a one-shot query, so exits after detecting the events. However, hotplug events are not lost. The first "block" request has registered "js_app1" (from above example) and hotplug events will accumulate. So, by having a loop, to receive and process the hotplug events, no events will be missed. Ex shell script: while true; doWhat needs to be understood about the above, is how simple the implementation is. The client application just creates a file (if it does not exist) named "/tmp/pup_event_ipc/block_js_app1" and waits for the server-application to put something into it. This inotify IPC technique is described in the next section. The inotify backendThe IPC technique used is more than a little unusual. The established techniques such as D-Bus, named pipes, message queues and Unix Domain Sockets, were investigated and rejected for one reason or another.The inotify mechanism supported by the Linux kernel is not intended to be used for IPC, at least not to the extent that I have done. As it has turned out, however, it "ticks all the boxes" for what I want for Puppy -- it is very fast, low resources, extremely simple, and remarkably amenable to the particular uses that I wanted to put it to. The References Section below has links that introduce inotify, read those if you need to understand the fundamentals. There is a directory /tmp/pup_event_ipc, the "mailbox", that contains the files that have the messages to be transferred. When a file pertaining to a particular client changes, the client "wakes up" and handles it. Basically the steps are:
That is really all it is. From those three steps, a complete IPC system can be devised. Of course, there all kinds of caveats and gotchas in this mechanism, such as guaranteeing a complete message is written to a file before it is read, and that files do not contain garbled messages if two or more clients write to them simultaneously. The way that I have implemented it, does, I think, avoid these pitfalls. Atomicity of read and write is important for this to work, and from reading online I got two different figures for this for Linux: 1KB or 4KB (kilobytes). I was optimistic and used the higher figure. A communication is between two clients, using two files, for each direction of transfer. As long as they choose unique names, there is not going to be a clash with other transfers. There will not occur the problem of two different clients simultaneously writing to the same file. Anyway, even if two or more clients were allowed to write to the same file "simultaneously", atomicity and the O_APPEND write mode (see below), will ensure that individual messages remain intact. The CLI utility is the first implementation of inotify-based IPC for Puppy Linux, though, it is generic and will work for all Linux distributions. The easiest way I think to explain how it all "hangs together" is to show the code for this utility. The utility 'pup_event_ipc' is written in BaCon (see: https://bkhome.org/archive/bacon) with embedded C, so is a binary executable, small and fast. This is the code as at February 23, 2018:
This is a remarkably small program to implement a complete IPC system. Essentially, it does this:
Messages are line-based, that is, one message per line. This allows messages to accumulate, while waiting to be read -- hence, most important, no messages get lost. Take the message "block:myapp" as an example. The name "myapp" is an arbitrary choice, anything unique is suitable. If this is posted by the CLI utility: # pup_event_ipc "block:myapp"If you were to trace through the above code, you would see that if file "/tmp/pup_event_ipc/block_myapp" is opened (or created if it does not exist), then there is a block on waiting for something to be written to the file. When something is written to the file, it is read, and the file zeroised. The read contents are printed on stdout. The above scenario means that some other application must write to /tmp/pup_event_ipc/block_myapp. In the very latest builds of Puppy, we do have a an application that detects drive hotplug activity, a daemon named pup_event_frontend_d, that performs this information server function. The function that such a server would have to perform can be show by a shell script: for ONEFILE in `ls /tmp/pup_event_ipc/block_*`That's it, done. All clients that are waiting on block-drive hotplug notification, will "wake up" and read the data. As mentioned before, it does not matter if a client is in a loop, doing other stuff: while true; doWhile executing "other stuff", any new block-drive events will append to /tmp/pup_event_ipc/block_myapp (that is, queue ) until it is next read. There is a lot that can be said about the flexibility of this inotify-based IPC mechanism. Just one example: although the pup_event_ipc utility does have a timeout option, you can cause a timeout whenever you want -- say, for example that a shell script has executed "pup_event_ipc 'block:myapp'" and is blocked, waiting on a modification to file /tmp/pup_event_ipc/block_myapp. Well, if you just execute "touch /tmp/pup_event_ipc/block_myapp", the read() function that is waiting will return, with zero data of course -- but voila, a simple way to cause a timeout. Example code blocksI will add to this section. For now, I have included code to show how the "server" can be implemented to serve block-drive event notifications.Block-drive event notificationIt has been explained above, that a client application may request block-drive event notification by posting a message of format "block:myapp" to the IPC mailbox.Puppy Linux has a daemon, named pup_event_frontend_d, at /usr/local/pup_event (compiled from the BaCon source file, pup_event_frontend_d.bac, and there is a wrapper script at /sbin). This daemon is started from /root/.xinitrc when X starts up. It does a lot of background stuff, such as manage the desktop drive icons. For that latter reason, it does have information about drive hotplug events. It was an easy matter to add a bit of extra code to pup_even_frontend_d, to post this information to clients that want it, via the IPC mechanism. Example shell script code is shown in the above section, how this can be done. The actual code inserted into pup_event_frontend_d is shown below (as at Feb. 23 2018). Basically, it looks for any files named /tmp/pup_event_ipc/zipc_prefix$_* (ex: /tmp/pup_event_ipc/block_*), then writes the drive hotplug information to them (string zipc_event$):
Note, BaCon is just a translator to C, which is then compiled using the Gnu C compiler. It is really just a thin translation layer, so many C-isms can be put into the BaCon code, and pure C can be embedded. I use BaCon as I prefer the syntax, plus it has many higher-level features, such as better string handling, and associative arrays. It is easy to read the above and translate to all-C if required. A caveat though. My utilities must be compiled with BaCon version 3.0.2 or 3.7.2+, as in-between there is "string handling enhancements" that wreck the "thin translation layer". Feedback welcomeI will be posting about progress on pup_event to my blog, https://bkhome.org/news/Feedback will be welcome. If you can think of anything that might compromise the integrity of data transfer with this technique, or code modifications that will add improvements (while still keeping everything simple, including simple and small code). Regards, Barry Kauler ReferencesReference on libc functions:http://www.gnu.org/software/libc/manual/html_node/Function-Index.html Wikipedia: http://en.wikipedia.org/wiki/Inotify Intro to inotify: http://www.linuxjournal.com/article/8478 Monitor file system activity with inotify: http://www.ibm.com/developerworks/library/l-ubuntu-inotify/ Monitor Linux File System Events with Inotify: http://www.developertutorials.com/tutorials/linux/monitor-linux-inotify-050531-1133/ The inotify API: http://linux.die.net/man/7/inotify |
||||||||
| (c) Copyright Barry Kauler, 2013, 2018, 2019. All reproduction rights of this page are reserved. |