Table of Contents

As posted on the open-ils-dev mailing list:

Bill and I have been puzzling over how to give HTTP clients support for the more advanced features of OpenSRF, such as stateful, transactional sessions and streaming responses. The main problem is that HTTP is entirely stateless on its own, and OpenSRF is not. The second biggest problem is that most current HTTP clients do not lend themselves to the kind of connection control we'd like to have.

Well, with the addition of (what really should be) a fairly trivial translator service written as an Apache module, and a new server-side setting that allows OpenSRF Applications to loosen their restrictions on who can talk to them and when, we think we have a way to provide advanced, robust, degradable OpenSRF services to all modern HTTP clients.

Let us know what you think.

And so, herein follows …

A modest proposal for streaming, stateful OpenSRF-over-HTTP

Client Request

The method is POST, URL is that of the HTTP/XMPP translator. The body of the POST data should be a JSON encoded array of one or more osrfMessage objects. See the bottom of this document for both XMPP and HTTP examples of OpenSRF communication.

OpenSRF HTTP headers from client to server:

The X-OpenSRF-service and X-OpenSRF-to headers are mutually exclusive, and cannot be used together. There are exactly two instances where the X-OpenSRF-service header should be used:

- At the beginning of a stateful session, when sending the CONNECT message - For stateless requests

In all other instances, which is to say within any stateful session that is past the CONNECT phase, the client should use the X-OpenSRF-to header. The value of this header is set to the XMPP address returned in the by STATUS_OK message from the current session CONNECT request.

If multipart/x-mixed-replace mode is supported by the client, then it should send the following header with every HTTP request:

X-OpenSRF-multipart=true

Otherwise, the translator must assume multipart is not supported by the client.

Setting aside the underlying transport and supposing the complete equivalence of the HTTP headers to XMPP XML attributes as described above, all other data within the OpenSRF message remains entirely the same. In other words, The body of the HTTP request is exactly equivalent to the <body/> element with an XMPP <message/> element.

Treament of these connection headers within the HTTP/XMPP translator

configured router, as described in the core config file for OpenSRF clients. Typically this will look something like router@localhost/open-ils.search, where the service name provided in the header is used as the last component, the XMPP resource. This is used in the @to attr of the <message/> element on the XMPP network. This header is only used for the first message in a stateful connection or for stateless requests.

value provided in the first X-OpenSRF-from header from the STATUS_OK message provided by the server in response to the initial CONNECT request. This is used in the @to attr of the <message/> element on the XMPP network.

Used to trace the path of messages through the OpenSRF network. If not supplied by the client, this should be created and supplied by the translator. This datum is used in the @osrf_xid attr of the <message/> element on the XMPP network.

any string, though typically, a combination of the epoc time and originating process id are used. This is used by OpenSRF endpoints for identifying and continuing stateful sessions. If not supplied by the client, the translator should create and supply a value for this element. This datum is used as the text content of the <thread/> element on the XMPP network.

Response from the server

Assuming no error was encountered, the OpenSRF service will respond with among other things several bits of data that are critical to the ongoing communication between the client and server. The most important of these is the XMPP @from attr of the <message/> element. This must be used by the client for all future requests within a stateful session, assuming a stateful session was requested and created. This datum should be mapped by the translator to an HTTP header called X-OpenSRF-from in the response to the client. The client will then use this value in the X-OpenSRF-to header of all future in-session communication.

The body of the response, or responses if a multipart request is used and a streaming method called, will each be an array of one or more osrfMessage objects.

If the client does not request multipart/x-mixed-replace mode using the X-OpenSRF-multipart header, as described above, then the translator will collect all response osrfMessage objects, up to and including the first COMPLETE message, and packaged them into a single array object as a single request response to the client.

Changes to OpenSRF

Stateful communication with backend OpenSRF service instances will be made possible by giving each service the ability to accept "migratable" client remote IDs. If the configuration section for a given OpenSRF application has the setting 

  <migratable-clients>true</migratable-clients>

1)

then that service will ignore the requirement that the remote ID of a stateful client must match across all communication within a session, and instead rely only on the thread of the session for session tracking. It is normally, and currently always, an error condition for multiple client XMPP identifiers to be used with one thread and one session. This is to prevent session hijacking among other security concerns. However, loosening this restriction is required in order to support OpenSRF-over-HTTP, and concerns are largely mitigated by requiring the service to allow the feature explicitly in its configuration.

There is no restriction on the type of osrfMessage objects that can be sent, and as long as their order (and associated succesful responses) would constitute a normal stateful OpenSRF session on an XMPP network, the use of the <migratable-clients/> setting will allow this same functionality over HTTP.

Overview of the HTTP/XMPP translator

The translator will be written as an Apache module. Its entire purpose is to smooth out the differences between the HTTP and XMPP transports with respect to OpenSRF communication. It will do as little interpretation and manipulation of data as possible, modifying only that data which must absolutely be changed in order to support certain sub-optimal client software, and to map concepts between HTTP and XMPP.

Upstream HTTP-side handling

For upstream (client to server) communication, no data modification need occur other than the mapping request headers to XMPP attributes, and the adding of XMPP-specific router information to the service initially requested by the client. No modification or parsing of the message body will be performed on upstream communication.

Downstream HTTP-side handling

For downstream (server to client) communication, there are to possible logic paths. The first, being both simpler to implement and more generally useful on the client end, is the case where the client supports multipart/x-mixed-replace mode, as is the case with the Mozilla browser from version 1.7 on and all version of the Firefox browser. We will call this multipart mode. The second case contains all other browsers and clients, as no other browsers are known to support this mode at this time. We will call this collected mode.

  1. For multipart mode capable clients, no modification of any data will occur

other than the mapping of the XMPP attributes to their respective HTTP counterparts. All data contained in the body of the OpenSRF message will be passed unaltered as a multipart chunk followed by a boundary marker. The translator will use this response mode when it receives a request that is accompanied by a X-OpenSRF-multipart header set to true.

  1. For collected mode clients, all osrfMessage objects sent by the server will

be collected into a single JSON array, in the order they arrive, and delivered as a single HTTP response to the client. This will require parsing of the JSON in order to pull all osrfMessage objects out of each response packet and build the response array.

XMPP-side communication

The translator will use standard OpenSRF libraries to communicate with the XMPP network-based services, just as the existing gateway and srfsh applications do.

Example Data

XMPP message example

<message
       to='osrf@localhost/open-ils.search_listener_at_grendel.local_31319'
       from='gateway@localhost/1192540419__1192540420.166380_31367'
       router_command=''
       router_class=''
       osrf_xid='1192540419313673'>
 <thread>1192540427.567678.119254042731367</thread>
   <body>
[
   {
       "__c":"osrfMessage",
       "__p":{
           "threadTrace":"1",
           "locale":"en-us",
           "type":"REQUEST",
           "payload":{
               "__c":"osrfMethod",
               "__p":{
                   "method":"open-ils.search.biblio.record.mods_slim.retrieve",
                   "params":[1487101]
               }
           }
       }
   }
]
   </body>
</message>

HTTP message example

X-OpenSRF-to=open-ils.search
X-OpenSRF-xid=1192540419313673
X-OpenSRF-thread=1192540427.567678.119254042731367
X-OpenSRF-multipart=true

[
   {
       "__c":"osrfMessage",
       "__p":{
           "threadTrace":"1",
           "locale":"en-us",
           "type":"REQUEST",
           "payload":{
               "__c":"osrfMethod",
               "__p":{
                   "method":"open-ils.search.biblio.record.mods_slim.retrieve",
                   "params":[1487101]
               }
           }
       }
   }
]
1)
This name will probably change … it has met with resistance based on the fact that "migratable" is not a word. ;)