As posted on the open-ils-dev mailing list:
----

Bill and I have been puzzling over how to give HTTP clients support
for the more advanced features of OpenSRF, such as stateful,
transactional sessions and streaming responses.  The main problem is
that HTTP is entirely stateless on its own, and OpenSRF is not.  The
second biggest problem is that most current HTTP clients do not lend
themselves to the kind of connection control we'd like to have.

Well, with the addition of (what really should be) a fairly trivial
translator service written as an Apache module, and a new server-side
setting that allows OpenSRF Applications to loosen their restrictions
on who can talk to them and when, we think we have a way to provide
advanced, robust, degradable OpenSRF services to all modern HTTP
clients.

Let us know what you think.

And so, herein follows ...


**A modest proposal for streaming, stateful OpenSRF-over-HTTP**
----



====Client Request====

The method is POST, URL is that of the HTTP/XMPP translator.  The body of the
POST data should be a JSON encoded array of one or more osrfMessage objects.
See the bottom of this document for both XMPP and HTTP examples of OpenSRF
communication.

OpenSRF HTTP headers from client to server:

  * X-OpenSRF-service=<service-name> //for example, open-ils.search//
  * X-OpenSRF-to=<XMPP-address>      //value of the last **X-OpenSRF-from** delivered by the server//
  * X-OpenSRF-xid=<timestamp-ms>     //for tracing messages to the client//
  * X-OpenSRF-thread=<guid-string>   //uniquely identifies this conversation//

The **X-OpenSRF-service** and **X-OpenSRF-to** headers are mutually exclusive, and
cannot be used together.  There are exactly two instances where the
**X-OpenSRF-service** header should be used:

 - At the beginning of a stateful session, when sending the CONNECT message
 - For stateless requests

In all other instances, which is to say within any stateful session that is
past the **CONNECT** phase, the client should use the **X-OpenSRF-to** header.  The
value of this header is set to the XMPP address returned in the by **STATUS_OK**
message from the current session **CONNECT** request.


If **multipart/x-mixed-replace** mode is supported by the client, then it should
send the following header with every HTTP request:

 X-OpenSRF-multipart=true

Otherwise, the translator must assume multipart is not supported by the client.

Setting aside the underlying transport and supposing the complete equivalence
of the HTTP headers to XMPP XML attributes as described above, all other data
within the OpenSRF message remains entirely the same.  In other words, The body
of the HTTP request is exactly equivalent to the ''<body/>'' element with an XMPP
''<message/>'' element.


===Treament of these connection headers within the HTTP/XMPP translator===

  * **X-OpenSRF-service** -- Should be mapped to the XMPP endpoint within the
configured router, as described in the core config file for OpenSRF clients.
Typically this will look something like ''router@localhost/open-ils.search'',
where the service name provided in the header is used as the last component,
the XMPP resource.  This is used in the ''@to'' attr of the ''<message/>'' element on
the XMPP network.  This header is only used for the first message in a stateful
connection or for stateless requests.

  * **X-OpenSRF-to** -- Within the client, he value for this header is taken by from
value provided in the first **X-OpenSRF-from** header from the **STATUS_OK** message
provided by the server in response to the initial **CONNECT** request.  This is
used in the ''@to'' attr of the ''<message/>'' element on the XMPP network.

  * **X-OpenSRF-xid** -- Typically, the time since the Unix epoc in milliseconds.
Used to trace the path of messages through the OpenSRF network.  If not
supplied by the client, this should be created and supplied by the translator.
This datum is used in the ''@osrf_xid'' attr of the ''<message/>'' element on the XMPP
network.

  * **X-OpenSRF-thread** -- A unique identifier for this request or session.  Can be
any string, though typically, a combination of the epoc time and originating
process id are used.  This is used by OpenSRF endpoints for identifying and
continuing stateful sessions.  If not supplied by the client, the translator
should create and supply a value for this element.  This datum is used as the
text content of the ''<thread/>'' element on the XMPP network.


====Response from the server====

Assuming no error was encountered, the OpenSRF service will respond with among
other things several bits of data that are critical to the ongoing
communication between the client and server.  The most important of these is
the XMPP ''@from'' attr of the ''<message/>'' element. This must be used by the client
for all future requests within a stateful session, assuming a stateful session
was requested and created.  This datum should be mapped by the translator to an
HTTP header called **X-OpenSRF-from** in the response to the client.
The client will then use this value in the **X-OpenSRF-to** header of all
future in-session communication.

The body of the response, or responses if a multipart request is used and a
streaming method called, will each be an array of one or more ''osrfMessage''
objects.

If the client does not request **multipart/x-mixed-replace** mode using the
**X-OpenSRF-multipart** header, as described above, then the translator will
collect all response ''osrfMessage'' objects, up to and including the first
**COMPLETE** message, and packaged them into a single array object as a single
request response to the client.


====Changes to OpenSRF====

Stateful communication with backend OpenSRF service instances will be made
possible by giving each service the ability to accept "migratable" client
remote IDs.  If the configuration section for a given OpenSRF application
has the setting

<code>
  <migratable-clients>true</migratable-clients>
</code>((This name will probably change ... it has met with resistance based on the fact that "migratable" is not a word. ;) ))

then that service will ignore the requirement that the remote ID of a stateful
client must match across all communication within a session, and instead rely
only on the thread of the session for session tracking.  It is normally, and
currently always, an error condition for multiple client XMPP identifiers to
be used with one thread and one session.  This is to prevent session hijacking
among other security concerns.  However, loosening this restriction is required
in order to support OpenSRF-over-HTTP, and concerns are largely mitigated by
requiring the service to allow the feature explicitly in its configuration.

There is no restriction on the type of ''osrfMessage'' objects that can be sent,
and as long as their order (and associated succesful responses) would
constitute a normal stateful OpenSRF session on an XMPP network, the use of
the ''<migratable-clients/>'' setting will allow this same functionality over
HTTP.


====Overview of the HTTP/XMPP translator====

The translator will be written as an Apache module.  Its entire purpose is to
smooth out the differences between the HTTP and XMPP transports with respect
to OpenSRF communication.  It will do as little interpretation and manipulation
of data as possible, modifying only that data which must absolutely be changed
in order to support certain sub-optimal client software, and to map concepts
between HTTP and XMPP.


===Upstream HTTP-side handling===

For upstream (client to server) communication, no data modification need occur
other than the mapping request headers to XMPP attributes, and the adding of
XMPP-specific router information to the service initially requested by the
client.  No modification or parsing of the message body will be performed on
upstream communication.

===Downstream HTTP-side handling===

For downstream (server to client) communication, there are to possible logic
paths.  The first, being both simpler to implement and more generally useful
on the client end, is the case where the client supports
**multipart/x-mixed-replace** mode, as is the case with the Mozilla browser from
version 1.7 on and all version of the Firefox browser.  We will call this
**multipart mode**.  The second case contains all other browsers and clients, as
no other browsers are known to support this mode at this time.  We will call
this **collected mode**.

  - For **multipart mode** capable clients, no modification of any data will occur
other than the mapping of the XMPP attributes to their respective HTTP
counterparts.  All data contained in the body of the OpenSRF message will be
passed unaltered as a multipart chunk followed by a boundary marker.  The
translator will use this response mode when it receives a request that is
accompanied by a **X-OpenSRF-multipart** header set to **true**.

  - For **collected mode** clients, all osrfMessage objects sent by the server will
be collected into a single JSON array, in the order they arrive, and delivered
as a single HTTP response to the client.  This will require parsing of the JSON
in order to pull all ''osrfMessage'' objects out of each response packet and build
the response array.


===XMPP-side communication===

The translator will use standard OpenSRF libraries to communicate with the XMPP
network-based services, just as the existing gateway and srfsh applications do.


====Example Data====

===XMPP message example===

<code>
<message
       to='osrf@localhost/open-ils.search_listener_at_grendel.local_31319'
       from='gateway@localhost/1192540419__1192540420.166380_31367'
       router_command=''
       router_class=''
       osrf_xid='1192540419313673'>
 <thread>1192540427.567678.119254042731367</thread>
   <body>
[
   {
       "__c":"osrfMessage",
       "__p":{
           "threadTrace":"1",
           "locale":"en-us",
           "type":"REQUEST",
           "payload":{
               "__c":"osrfMethod",
               "__p":{
                   "method":"open-ils.search.biblio.record.mods_slim.retrieve",
                   "params":[1487101]
               }
           }
       }
   }
]
   </body>
</message>
</code>

===HTTP message example===

<code>

X-OpenSRF-to=open-ils.search
X-OpenSRF-xid=1192540419313673
X-OpenSRF-thread=1192540427.567678.119254042731367
X-OpenSRF-multipart=true

[
   {
       "__c":"osrfMessage",
       "__p":{
           "threadTrace":"1",
           "locale":"en-us",
           "type":"REQUEST",
           "payload":{
               "__c":"osrfMethod",
               "__p":{
                   "method":"open-ils.search.biblio.record.mods_slim.retrieve",
                   "params":[1487101]
               }
           }
       }
   }
]

</code>