Table of Contents
As posted on the open-ils-dev mailing list:
Bill and I have been puzzling over how to give HTTP clients support for the more advanced features of OpenSRF, such as stateful, transactional sessions and streaming responses. The main problem is that HTTP is entirely stateless on its own, and OpenSRF is not. The second biggest problem is that most current HTTP clients do not lend themselves to the kind of connection control we'd like to have.
Well, with the addition of (what really should be) a fairly trivial translator service written as an Apache module, and a new server-side setting that allows OpenSRF Applications to loosen their restrictions on who can talk to them and when, we think we have a way to provide advanced, robust, degradable OpenSRF services to all modern HTTP clients.
Let us know what you think.
And so, herein follows …
A modest proposal for streaming, stateful OpenSRF-over-HTTP
Client Request
The method is POST, URL is that of the HTTP/XMPP translator. The body of the POST data should be a JSON encoded array of one or more osrfMessage objects. See the bottom of this document for both XMPP and HTTP examples of OpenSRF communication.
OpenSRF HTTP headers from client to server:
- X-OpenSRF-service=<service-name> for example, open-ils.search
- X-OpenSRF-to=<XMPP-address> value of the last X-OpenSRF-from delivered by the server
- X-OpenSRF-xid=<timestamp-ms> for tracing messages to the client
- X-OpenSRF-thread=<guid-string> uniquely identifies this conversation
The X-OpenSRF-service and X-OpenSRF-to headers are mutually exclusive, and cannot be used together. There are exactly two instances where the X-OpenSRF-service header should be used:
- At the beginning of a stateful session, when sending the CONNECT message - For stateless requests
In all other instances, which is to say within any stateful session that is past the CONNECT phase, the client should use the X-OpenSRF-to header. The value of this header is set to the XMPP address returned in the by STATUS_OK message from the current session CONNECT request.
If multipart/x-mixed-replace mode is supported by the client, then it should send the following header with every HTTP request:
X-OpenSRF-multipart=true
Otherwise, the translator must assume multipart is not supported by the client.
Setting aside the underlying transport and supposing the complete equivalence
of the HTTP headers to XMPP XML attributes as described above, all other data
within the OpenSRF message remains entirely the same. In other words, The body
of the HTTP request is exactly equivalent to the <body/>
element with an XMPP
<message/>
element.
Treament of these connection headers within the HTTP/XMPP translator
- X-OpenSRF-service – Should be mapped to the XMPP endpoint within the
configured router, as described in the core config file for OpenSRF clients.
Typically this will look something like router@localhost/open-ils.search
,
where the service name provided in the header is used as the last component,
the XMPP resource. This is used in the @to
attr of the <message/>
element on
the XMPP network. This header is only used for the first message in a stateful
connection or for stateless requests.
- X-OpenSRF-to – Within the client, he value for this header is taken by from
value provided in the first X-OpenSRF-from header from the STATUS_OK message
provided by the server in response to the initial CONNECT request. This is
used in the @to
attr of the <message/>
element on the XMPP network.
- X-OpenSRF-xid – Typically, the time since the Unix epoc in milliseconds.
Used to trace the path of messages through the OpenSRF network. If not
supplied by the client, this should be created and supplied by the translator.
This datum is used in the @osrf_xid
attr of the <message/>
element on the XMPP
network.
- X-OpenSRF-thread – A unique identifier for this request or session. Can be
any string, though typically, a combination of the epoc time and originating
process id are used. This is used by OpenSRF endpoints for identifying and
continuing stateful sessions. If not supplied by the client, the translator
should create and supply a value for this element. This datum is used as the
text content of the <thread/>
element on the XMPP network.
Response from the server
Assuming no error was encountered, the OpenSRF service will respond with among
other things several bits of data that are critical to the ongoing
communication between the client and server. The most important of these is
the XMPP @from
attr of the <message/>
element. This must be used by the client
for all future requests within a stateful session, assuming a stateful session
was requested and created. This datum should be mapped by the translator to an
HTTP header called X-OpenSRF-from in the response to the client.
The client will then use this value in the X-OpenSRF-to header of all
future in-session communication.
The body of the response, or responses if a multipart request is used and a
streaming method called, will each be an array of one or more osrfMessage
objects.
If the client does not request multipart/x-mixed-replace mode using the
X-OpenSRF-multipart header, as described above, then the translator will
collect all response osrfMessage
objects, up to and including the first
COMPLETE message, and packaged them into a single array object as a single
request response to the client.
Changes to OpenSRF
Stateful communication with backend OpenSRF service instances will be made possible by giving each service the ability to accept "migratable" client remote IDs. If the configuration section for a given OpenSRF application has the setting
<migratable-clients>true</migratable-clients>
then that service will ignore the requirement that the remote ID of a stateful client must match across all communication within a session, and instead rely only on the thread of the session for session tracking. It is normally, and currently always, an error condition for multiple client XMPP identifiers to be used with one thread and one session. This is to prevent session hijacking among other security concerns. However, loosening this restriction is required in order to support OpenSRF-over-HTTP, and concerns are largely mitigated by requiring the service to allow the feature explicitly in its configuration.
There is no restriction on the type of osrfMessage
objects that can be sent,
and as long as their order (and associated succesful responses) would
constitute a normal stateful OpenSRF session on an XMPP network, the use of
the <migratable-clients/>
setting will allow this same functionality over
HTTP.
Overview of the HTTP/XMPP translator
The translator will be written as an Apache module. Its entire purpose is to smooth out the differences between the HTTP and XMPP transports with respect to OpenSRF communication. It will do as little interpretation and manipulation of data as possible, modifying only that data which must absolutely be changed in order to support certain sub-optimal client software, and to map concepts between HTTP and XMPP.
Upstream HTTP-side handling
For upstream (client to server) communication, no data modification need occur other than the mapping request headers to XMPP attributes, and the adding of XMPP-specific router information to the service initially requested by the client. No modification or parsing of the message body will be performed on upstream communication.
Downstream HTTP-side handling
For downstream (server to client) communication, there are to possible logic paths. The first, being both simpler to implement and more generally useful on the client end, is the case where the client supports multipart/x-mixed-replace mode, as is the case with the Mozilla browser from version 1.7 on and all version of the Firefox browser. We will call this multipart mode. The second case contains all other browsers and clients, as no other browsers are known to support this mode at this time. We will call this collected mode.
- For multipart mode capable clients, no modification of any data will occur
other than the mapping of the XMPP attributes to their respective HTTP counterparts. All data contained in the body of the OpenSRF message will be passed unaltered as a multipart chunk followed by a boundary marker. The translator will use this response mode when it receives a request that is accompanied by a X-OpenSRF-multipart header set to true.
- For collected mode clients, all osrfMessage objects sent by the server will
be collected into a single JSON array, in the order they arrive, and delivered
as a single HTTP response to the client. This will require parsing of the JSON
in order to pull all osrfMessage
objects out of each response packet and build
the response array.
XMPP-side communication
The translator will use standard OpenSRF libraries to communicate with the XMPP network-based services, just as the existing gateway and srfsh applications do.
Example Data
XMPP message example
<message to='osrf@localhost/open-ils.search_listener_at_grendel.local_31319' from='gateway@localhost/1192540419__1192540420.166380_31367' router_command='' router_class='' osrf_xid='1192540419313673'> <thread>1192540427.567678.119254042731367</thread> <body> [ { "__c":"osrfMessage", "__p":{ "threadTrace":"1", "locale":"en-us", "type":"REQUEST", "payload":{ "__c":"osrfMethod", "__p":{ "method":"open-ils.search.biblio.record.mods_slim.retrieve", "params":[1487101] } } } } ] </body> </message>
HTTP message example
X-OpenSRF-to=open-ils.search X-OpenSRF-xid=1192540419313673 X-OpenSRF-thread=1192540427.567678.119254042731367 X-OpenSRF-multipart=true [ { "__c":"osrfMessage", "__p":{ "threadTrace":"1", "locale":"en-us", "type":"REQUEST", "payload":{ "__c":"osrfMethod", "__p":{ "method":"open-ils.search.biblio.record.mods_slim.retrieve", "params":[1487101] } } } } ]