Evergreen DokuWiki

Explorations and milestones for the browser staff client development project.

I have a mixed bag of updates this week:

I put together a functioning infinite scroll grid using the angular-ui toolkit's "ng-scroll" directive. Based on my findings there and subsequent discussion with the community about the benefits of scrolled vs. paged grids, I rearranged much of the grid code (a lot of which I needed to do anyway) to operate as a paged grid by default, but one that can be easily turned into a scrolled grid. The transition would require a tiny JS translator object and hiding some of the paging controls. It occurred to me this is easier than going in the opposite direction. This provides some flexibility and may give users a chance to try either approach.

• row selection
  • multi-select via checkboxes and control-click
  • ranged selection via shift-click
  • select all / none
• Sorting
  • quick sort by clicking on column headers
  • primary, secondary, tertiary, etc. and ascending/descending sort priorities via numeric configuration, just like the Dojo Grid Configuration dialog.
• Pagination
  • home, next, prev navigation
  • go to page
  • page size selection
• CSV download and printing
• Column reordering via drag-n-drop
• Column resize via drag-n-drop and via the grid configuration row.
  • The drag-n-drop controls are buggy in FF at the moment
  • Column resize only operates in one direction per drag action.
    • I can expand on that for those interested.
  • There is a lot of room for improvement with the drag-n-drop UI bits in particular

I reached a milestone recently while printing CSV output from a grid through Hatch (the print/storage service) running on my desktop. It was the first practical application of printing in this manner. Yay! It was only text (CSV), but that's a start.

The Hatch-connecting code is designed to fail gracefully and use the browser printer if no connection can be made. Browser printing is done (for now, anyway) via print CSS media and standard $window.print() on the current page .. no window.open() (and the various browser oddities that ensue) required.

Based on discussions at the #egils14, I put together a small status bar which lives at the bottom of the page. For now, it shows connectivity information for the server websockets connection and the Hatch websockets connection. It also supports application-generated messages. It's cute, but I think it could also be very useful and it will hopefully provide a more consistent mechanism for alerting staff to important information. More on this to follow…

I've encountered a number of challenges migrating to SSL WebSockets with untrusted certificates.

• Talking to the Evergreen server, Firefox provides no mechanism for allowing a connection to an untrusted certificate.
  • https://bugzilla.mozilla.org/show_bug.cgi?id=594502
• Related, Chrome will allow temporary connections to untrusted certificates for WebSockets within the current page, but not if the WebSocket connection is executed from within a shared web worker.
• Firefox (and probably Chrome, eventually) prevent security downgrades for WebSockets connections.
  • https://bugzilla.mozilla.org/show_bug.cgi?id=917829
  • This means all WebSockets connections coming from the staff UI must use SSL, since the staff UI uses SSL.
  • Of particular note here is the connection to the local print/storage service.

The solution to all of these is to use trusted certificates. This can be done by using a certificate from a trusted authority on your server or by configuring your browser to trust a locally-generated certificate, the kind we typically generate by default for Apache in Evergreen. There are a variety of free and low-cost certificate providers. Two that came up in #evergreen recently were namecheap GeoTrust RapidSSL and startssl.

The bigger challenge here will be providing trusted certificates to the local print storage service. I'll go out on a limb and say that providing trusted certificates for every workstation running Hatch is not a realistic goal. The more likely solution will be to generate a local certificate during the installation process and configuring the browser to trust said certificate. Alternate suggestions appreciated.

There are two ways we can integrate the new browser interfaces into staff work flow. Staff using a certain module can either use the browser directly as their primary work environment or we can (theoretically) integrate browser-based interfaces piecemeal into the existing XUL app. The purpose of the mixed (browser + XUL) approach is the ease staff into the new interfaces, to encourage earlier adoption by replacing functionality directly in the client, and to avoid the case where staff may have to switch back and forth between two different environments.

The mixed approach sounds very appealing, but it does not come for free. To integrate browser apps into XUL, there are a number of technical issues we have to address. These are the ones I've encountered so far:

• Cookies, localStorage, and sessionStorage require a local XUL shim, since the browser runs under the oils: scheme and not https:
  • This is a problem we have partially solved before with the existing web apps
• SharedWebWorkers are not supported in XUL, so we would need to modify the WebSockets library to be a singleton connection which loads at login time and is accessible (via XUL pass-thru) to each browser interface
• Each Angular app requires a small bit of code to support the oils: scheme
• We would likely need to hide some elements within XUL, like the navbar, which means they would not see the light of day for some time.
• Occasionally HTML in XUL looks or behaves differently than it does in browsers
• XUL is a pain to debug, though I may just be missing something…

Individually, these are all relatively minor issues. Put them together, then toss in the undiscovered issues, and a complicated picture forms. If we go the mixed route, we would essentially be building the interfaces on two similar, but separate environments. Custom code (eventually discarded) would need to be written and each environment would require its own review and testing phases for each new interface. These add overhead and would extend the duration of the coding portion of the project.

Finally, I'm *mostly* convinced that the mixed approach is feasible. We won't know for certain until we dive in.

Here's where I need your input…

On the one hand the development phase will be shorter and on the other we have a longer project, but one where individual interfaces may be integrated earlier into the staff work flow. They are both appealing.

What are the other aspects of this decision? What is your preference?

I found some code which demonstrates printing of web pages in Java:

http://www.javacodegeeks.com/2013/07/introduction-by-example-javafx-8-printing.html

It's using features not available until Java 8, due out this week. This will, in theory, allow us to print CSS-driven HTML, graphics, etc.

Found this simple chunk of code for automatically stretching Bootstrap grid columns to fill the available space. It would need to be Angular-ized.

This is similar to the "auto" width column settings in the Dojo grid column configuration UI, which cause a column to expand to fill the empty space. With this, a 12-column grid could have 8 columns of data without leaving 4 columns unused.

If we used something like this in combination with increasing the default number of grid columns (via Bootstrap Customize), we could have a nice high column count (e.g. 16 or 24) that flows nicely with only, for example, 8 columns of data.

This (again, tiny) chunk of code does not work with mixed widths, since all columns have to have -auto for correct layout, but that could probably be added without too much trouble.

This doesn't give us magically ideal widths, like a table would provide, but it's a step closer.

The final (known) big item which requires resolution before coding can begin in earnest is the display grid. This will be the workhorse UI component, used in any interface that provides a list of data (list of checkouts, list of holds, list of holdings, list of records, every conify interface, etc.), which is many if not most UIs.

Grid requirements:

• Can work as a flattener-based AutoGrid
• Can work as a manually driven grid, where the data is managed externally
• Columns can be auto generated from IDL classes (like conify interfaces) or manually set.
• Column labels can be applied in the markup or derived from the IDL (both i18n friendly).
• At least 2 levels of column sorting, preferably unlimited.
• Paging and/or scrolling
• Column picker
• What else?

I put together a proof of concept grid using Bootstrap Grids for the template.

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/collab/berick/web-staff-proto-ag-exp

It worked well, but has some obvious pluses and minuses, so I could really use some additional input on what we as a community see as the most important aspects of this oh-so-important bit of UI. What should we use as the underlying markup to drive the grid? I’ve listed 4 options below, there may be more:

1. Bootstrap Grids

• PRO: CSS is consistent with the rest of the UI
• PRO: It breaks and flows nicely on mobile devices, as expected
• PRO: div-based, so creating a scroll-able body (with static header) is trivial
• CON: Column widths are hard-coded percentages. Supporting draggable column resizing would be challenging. (Not sure how challenging, I haven’t tried).

2. Tables

• PRO: Column widths respond to the data to provide a more reasonable default width on initial display.
• CON: Supporting draggable column resizing would be challenging.
• CON: Table bodies do not naturally support scrolling. You basically have to coerce it into being a div, at which point you lose all of the table niceness.
• CON: Tables do not break/flow for mobile devices. They still work, they just stretch out horizontally.

3. We build our own non-Bootstrap grid-based tool.

• PROS: we can almost certainly make it do everything we need it to do.
• CON: it means more local code.

4. 3rd-Party Grid, e.g. http://angular-ui.github.io/ng-grid/

The pros and cons for this will change depending on the tool. Presently, this is my least favorite option. Since this UI will be heavily modified and customized to suit our needs, I think we need to use something we build. An good example of how 3rd-party code can create more work than it saves are the Dojo grids. In my estimation, we spent more time twiddling with the grids to get them to do what we want that we would have spent building a grid system that did exactly what we needed. Just my $0.02 on that. Other tools may be easier to work with.

Did I forget anything?

The great thing about my experiments so far is that the back-end code is markup-agnostic, since it’s all Angular. We could drop any markup (div, table, ol, etc.) into place and with the correct Angular loop / variable references, it will still work fine with the existing JS.

It occurred to me that the print server will need to operate over WebSockets instead of XMLHttpRequest, so that the browser can respond to asynchronous, external events. For example, integrating an RFID pad which sends checkout commands to the browser as items are laid on the pad. Since the conversation is instigated by the RFID pad and not the browser, a simple XMLHttpRequest call/response setup will not suffice.

Thanks to Warren A. Layton for testing the WebSockets install / code!

I’ve made number of improvements to the WebSockets gateway, including bug fixes and a new configuration option OSRF_WEBSOCKET_MAX_REQUEST_WAIT_TIME. This setting lets us tell the gateway to give up on a lingering outstanding request if it’s taking way too long and it’s the only thing preventing the connection from being marked as idle. It’s mostly a security improvement to prevent otherwise idle gateway processes from staying alive even after a proxied request died on the back-end with no response.

Author: Bill Erickson

• We're ready for some brave souls to test the WebSockets gateway. See recap in Bug 1268619
• The Evergreen web-staff-proto branch uses Websockets now by default. The patron search UI uses the new streaming version.

Author: Bill Erickson

It appears the version of libapr1 on Debian squeeze (found while testing on old dev server) has a bug which causes a segfault in the websockets Apache gateway.

http://svn.apache.org/viewvc/apr/apr/branches/1.4.x/CHANGES?view=markup

I'm pretty sure it's the item listed under APR 1.4.2. Squeeze reports it has 1.4.2 installed, but the bug described fits the scenario. Suffice to say, it's not a problem on Debian Wheezy.

Author: Bill Erickson

Some of the stuff I've been working on lately…

Mentioned in IRC, I posted a proof-of-concept Java print/storage service at http://git.evergreen-ils.org/?p=working/random.git;a=shortlog;h=refs/heads/collab/berick/hatch

It's built as a Jetty module, which allows us to publish a small API over HTTP directly on the workstation. To allow access from the browser, the module just has to return an HTTP "Access-Control-Allow-Origin" header whose value matches the Evergreen server (or "*" for testing).

The module is designed as a standard HTTP GET/POST handler, meaning the browser communicates via XMLHttpRequest. However, Jetty also supports WebSockets, so the module could also work as a WebSocket handler if the need arises. Since it's not talking over the network, the value of WebSockets is not as high, so I took the easier route for now.

My impression thus far is that Java has a powerful and fairly easy to use print API. I had no trouble finding / selecting printers, setting margin sizes, and flowing long paragraphs of text (instead of chopping them off). I'm not sure yet how to go about printing more complicated elements, like images, nor am I clear on how important that is, but presumably it's doable.

The file storage components were trivial.

Also, Java is portable. The service runs fine in Windows, Mac, and Linux.

So far, I have high hopes.

Testing, bug fixes, documentation, and cleanup continue on https://bugs.launchpad.net/opensrf/+bug/1268619

Beware, I may soon be turning WebSockets on by default in the web staff proto branch so I can test it more thoroughly.

As an experiment, I pushed an alternate version of the patron search API to the web-staff-proto branch to further test the value of WebSockets (and general API improvements). The new API returns a stream of patron objects instead of patron IDs. (We don't do this with XMLHttpRequest, because it requires the caller to wait for all responses to arrive before any may be rendered and the response messages may become very large). This speeds things up and requires the client to make considerably fewer network calls.

A traditional patron search of 50 results requires 102 OpenSRF messages and 102 individual network messages to complete. With WebSockets it takes 51 OpenSRF messages and about 12 (configurable) individual network messages. (When streaming, responses are "bundled" into small collections of responses as a form of I/O buffering, hence the 12 instead of 51 network messages).

In this specific example, patron results render about twice as fast. With properly built APIs, we should see comparable speedups on similarly shaped data sets (e.g. long lists of results).