Proposal: Use Websocketd for OpenSRF Websocket Gateway

This started as a research project resulting primarily from trying to address LP bug #1774703.

I propose we replace our Apache-driven OpenSRF Websocket handler with a websocketd implementation.

• Code: https://github.com/joewalnes/websocketd
• Binary builds: http://websocketd.com/

Websocketd is a purpose-built Websockets relay. It handles the Websocket protocol, communicating with back-end handlers via STDIO.

• The code is considerably less complicated
• No threads – we use a select() event loop.
• Should be easier to test and maintain
• Using the load test script (below) I am consistently able to cause failures in the apache-websocket handler, but not the websocketd handler.
• See also LP bug noted above.
• Uses about 15% less RAM (on my test VM)
• Speed is consistent with Apache (within fractions of milliseconds)
• Websocketd project has a lot more activity and contributors than the apache-websocket code we're currently using.
• fewer steps to set up

Only concern so far is lack of support for pre-forking backends. Each new WS connection requires an OpenSRF connect on startup. On my test VM this adds roughly 5ms of overhead to the first request.

Note that a proxy is not strictly required, because websocketd supports SSL connections, but I prefer to run it behind a proxy.

Check out and install OpenSRF branch user/berick/websocketd-backend @ working normally.

http://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/berick/websocketd-backend

cd /tmp
wget 'https://github.com/joewalnes/websocketd/releases/download/v0.3.0/websocketd-0.3.0-linux_amd64.zip'
unzip websocketd-0.3.0-linux_amd64.zip
sudo cp websocketd /usr/local/bin/

# Stop apache-websockets first since we're reusing the port.
sudo systemctl stop apache2-websockets # or similar

# AS USER opensrf
/usr/local/bin/websocketd --loglevel info --maxforks 250 --port 7682 /openils/bin/osrf-websocket-stdio

# Or optionally override the config file path
/usr/local/bin/websocketd --loglevel info --maxforks 250 --port 7682 /openils/bin/osrf-websocket-stdio /path/to/opensrf_core.xml

# Optionally background the process with a trailing '&'

Full set of command line args at https://github.com/joewalnes/websocketd/blob/master/help.go

Modify /etc/nginx/sites-enabled/osrf-ws-http-proxy.

# Websocketd supports SSL, but I run it in non-SSL mode since it's on the same machine as NGINX.
# Replace https:// with http://
proxy_pass http://localhost:7682;

# Update timeout values within the /osrf-websocket-translator section 
# to disconnect idle clients.
# The osrf-websocket-stdio handler has no idle timeout threads.
# Change to suit
proxy_send_timeout 5m;
proxy_read_timeout 5m;

Restart NGINX:

sudo systemctl restart nginx

Reminder:

Confirm WEBSOCKET_PORT_SSL has the expected value (usually 443) near the top of /openils/var/web/js/dojo/opensrf/opensrf_ws.js

The Perl test script runs a few calls / response loops resulting in several hundred websocket calls.

# beware this installs quite a few modules
sudo cpan Net::Async::WebSocket::Client;
sudo cpan IO::Async::SSL; # for testing full path

cd OpenSRF/src/websocket-stdio
perl tester.pl wss://localhost:443/osrf-websocket-translator

Log in to the browser client as usual.

Websocketd is a standalone program with no daemon mode (not yet anyway). Systemd to the rescue.

Put this content into file /lib/systemd/system/websocketd-osrf.service

[Unit]
Description=Websocketd OpenSRF Gateway

[Service]
Type=simple
User=opensrf
Group=opensrf
Environment=PATH=/openils/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Environment=LD_LIBRARY_PATH=/openils/lib:/usr/local/lib:/usr/local/lib/dbd:$LD_LIBRARY_PATH
ExecStart=/usr/local/bin/websocketd --loglevel error --maxforks 250 --port 7682 /openils/bin/osrf-websocket-stdio

# modify websocketd command line options to taste
# --sameorigin and --origin=domain1,domain2 flags are also supported for security.

# On Ubuntu 18.04, you may also need to include something like this:
[Install]
WantedBy=multi-user.target

Then add & start the service.

# disable apache2-websockets to avoid unintended starts & port conflicts
sudo systemctl disable apache2-websockets

# enable and start websocketd-osrf
sudo systemctl daemon-reload
sudo systemctl enable websocketd-osrf
sudo systemctl start websocketd-osrf