Evergreen DokuWiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Troubleshooting ======

If you have Evergreen installed and are encountering systematic errors, here is how you might go about troubleshooting and gathering logs for others to troubleshoot.  These instructions assume standard locations and filenames for Evergreen installations, and may also include commands that assume a Debian or Ubuntu Linux operating system.

**NOTE:** This content is also available in the Evergreen documentation at http://docs.evergreen-ils.org/1.6/draft/html/. In github: http://github.com/rsoulliere/Evergreen-DocBook/blob/master/1.6/admin/troubleshooting.xml. 

===== Step 1: Stop all the OpenSRF services, as well as Apache =====

==== Stopping Apache ====
Typically, you would use a command like this as user root:
<code>
/etc/init.d/apache2 stop
</code>
Another command might look like this:
<code>
apache2ctl stop
</code>
To double-check and see if apache has actually been terminated, run a command like so:
<code>
ps -aef | grep apache2
</code>
Apache has not been stopped if you see any entries like this:
<code>
root      4609     1  2 13:05 ?        00:00:00 /usr/sbin/apache2 -k start
opensrf   4612  4609  3 13:05 ?        00:00:00 /usr/sbin/apache2 -k start
</code>

==== Stopping OpenSRF ====
Typically, you would use a command like this as user opensrf:
<code>
/openils/bin/osrf_ctl.sh -c /openils/conf/opensrf_core.xml -l -a stop_all
</code>

**Note:** You may need to invoke the osrf_ctl.sh command without the -l option if the installation instructions you used did not have you use localhost in leu of your fully qualified domain name (FQDN) in your configuration file.  If this is the case, omit -l from all invocations of osrf_ctl.sh asked for in this document.

You should get either output resembling this:
<code>
Stopping OpenSRF C process 12515...
Stopping OpenSRF C process 12520...
Stopping OpenSRF C process 12526...
Stopping OpenSRF Perl process 12471...
Stopping OpenSRF Router process 12466...
</code>

Or, if services have already been stopped, like this:
<code>
OpenSRF C not running
OpenSRF Perl not running
OpenSRF Router not running
</code>

If you run into any troubles with stopping services, stop here and ask for help on either [[http://open-ils.org/listserv.php|OPEN-ILS-DEV]] or [[http://open-ils.org/irc.php|#evergreen]].  Other tips may be available on this [[troubleshooting:checking_for_errors_step1|page]].

Occasionally, in the trunk/ version of Evergreen, I have seen osrf_ctl.sh fail to kill OpenSRF processes, so we should check to make sure that none are still running with a command like this: 
<code>
ps -aef | grep OpenSRF
</code>

OpenSRF has not completely shutdown if you see any entries like this:
<code>
opensrf   4400     1  0 12:50 ?        00:00:00 OpenSRF System
opensrf   4425     1  0 12:50 ?        00:00:00 OpenSRF System-C
opensrf   4401  4400  0 12:50 ?        00:00:00 OpenSRF master [opensrf.settings]
opensrf   4408  4400  0 12:50 ?        00:00:00 OpenSRF master [open-ils.cat]
opensrf   4409  4400  0 12:50 ?        00:00:00 OpenSRF master [open-ils.supercat]
opensrf   4410  4400  0 12:50 ?        00:00:01 OpenSRF master [open-ils.search]
opensrf   4411  4400  0 12:50 ?        00:00:01 OpenSRF master [open-ils.circ]
opensrf   4412  4400  0 12:50 ?        00:00:00 OpenSRF master [open-ils.actor]
opensrf   4413  4400  1 12:50 ?        00:00:02 OpenSRF master [open-ils.storage]
opensrf   4414  4400  0 12:50 ?        00:00:01 OpenSRF master [open-ils.penalty]
opensrf   4415  4400  0 12:50 ?        00:00:00 OpenSRF master [open-ils.collections]
opensrf   4416  4400  0 12:50 ?        00:00:00 OpenSRF master [open-ils.ingest]
opensrf   4417  4400  0 12:50 ?        00:00:00 OpenSRF master [open-ils.reporter]
opensrf   4407  4400  0 12:50 ?        00:00:00 OpenSRF listener [opensrf.settings]
opensrf   4426  4425  0 12:50 ?        00:00:00 OpenSRF Listener [opensrf.math]
opensrf   4428  4425  0 12:50 ?        00:00:00 OpenSRF Listener [opensrf.dbmath]
opensrf   4430  4425  0 12:50 ?        00:00:00 OpenSRF Listener [open-ils.auth]
opensrf   4431  4425  0 12:50 ?        00:00:00 OpenSRF Listener [open-ils.cstore]
opensrf   4437  4425  0 12:50 ?        00:00:00 OpenSRF Listener [open-ils.reporter-store]
opensrf   4441  4400  0 12:50 ?        00:00:00 OpenSRF listener [open-ils.cat]
opensrf   4442  4400  0 12:50 ?        00:00:00 OpenSRF listener [open-ils.supercat]
opensrf   4443  4400  0 12:50 ?        00:00:00 OpenSRF listener [open-ils.search]
opensrf   4444  4400  0 12:50 ?        00:00:00 OpenSRF listener [open-ils.circ]
opensrf   4445  4400  0 12:50 ?        00:00:00 OpenSRF listener [open-ils.actor]
opensrf   4446  4400  0 12:50 ?        00:00:00 OpenSRF listener [open-ils.storage]
opensrf   4448  4400  0 12:50 ?        00:00:00 OpenSRF listener [open-ils.penalty]
opensrf   4449  4400  0 12:50 ?        00:00:00 OpenSRF listener [open-ils.collections]
opensrf   4450  4400  0 12:50 ?        00:00:00 OpenSRF listener [open-ils.ingest]
opensrf   4451  4400  0 12:50 ?        00:00:00 OpenSRF listener [open-ils.reporter]
opensrf   4402  4401  0 12:50 ?        00:00:00 OpenSRF drone [opensrf.settings]
opensrf   4403  4401  0 12:50 ?        00:00:00 OpenSRF drone [opensrf.settings]
opensrf   4404  4401  0 12:50 ?        00:00:00 OpenSRF drone [opensrf.settings]
opensrf   4405  4401  0 12:50 ?        00:00:00 OpenSRF drone [opensrf.settings]
opensrf   4406  4401  0 12:50 ?        00:00:00 OpenSRF drone [opensrf.settings]
opensrf   4427  4426  0 12:50 ?        00:00:00 OpenSRF Drone [opensrf.math]
opensrf   4429  4428  0 12:50 ?        00:00:00 OpenSRF Drone [opensrf.dbmath]
opensrf   4438  4414  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.penalty]
opensrf   4439  4437  0 12:50 ?        00:00:00 OpenSRF Drone [open-ils.reporter-store]
opensrf   4447  4430  0 12:50 ?        00:00:00 OpenSRF Drone [open-ils.auth]
opensrf   4452  4431  0 12:50 ?        00:00:00 OpenSRF Drone [open-ils.cstore]
opensrf   4454  4409  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.supercat]
opensrf   4455  4416  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.ingest]
opensrf   4456  4416  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.ingest]
opensrf   4457  4416  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.ingest]
opensrf   4458  4416  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.ingest]
opensrf   4459  4416  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.ingest]
opensrf   4460  4412  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.actor]
opensrf   4461  4417  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.reporter]
opensrf   4462  4408  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.cat]
opensrf   4463  4415  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.collections]
opensrf   4464  4410  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.search]
opensrf   4465  4411  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.circ]
opensrf   4466  4413  0 12:50 ?        00:00:00 OpenSRF drone [open-ils.storage]
</code>

You should manually kill these processes at this point.

If you were uable to stop OpenSRF with the above methods, you can also try this command:
<code>
rm –R /openils/var/run/*.pid
</code>
This will remove the temporary OpenSRF process files from the run directory which may have been left over from a previous shutdown of the system.


===== Step 2: Delete or archive the existing log files =====

By default, Evergreen log files are stored in
<code>
/openils/var/log/
</code>

To make it easier to scan the logs for errors, we should either erase or move the old logs out of the way while OpenSRF is not running.  As user opensrf, you might use a sequence of commands like so:
<code>
cd /openils/var/log/
mkdir old
mv *.* old/
</code>



===== Step 3: Check that jabber and memcached are running =====
Usually, you shouldn't have to restart your jabber and memcache servers when restarting OpenSRF.  But we do want to make sure they're at least running.

For ejabberd, you might try a command like so:
<code>
ps -aef | grep ejabberd
</code>
and expect output like this:
<code>
ejabberd  5508     1  0 Feb26 ?        00:00:00 /usr/lib/erlang/erts-5.5.5/bin/epmd -daemon
ejabberd  5525     1  0 Feb26 ?        00:01:47 /usr/lib/erlang/erts-5.5.5/bin/beam -- -root /usr/lib/erlang -progname erl -- -home /var/lib/ejabberd -sname ejabberd -s ejabberd -kernel inetrc "/etc/ejabberd/inetrc" -ejabberd config "/etc/ejabberd/ejabberd.cfg" log_path "/var/log/ejabberd/ejabberd.log" -sasl sasl_error_logger {file,"/var/log/ejabberd/sasl.log"} -mnesia dir "/var/lib/ejabberd" -noshell -noshell -noinput
ejabberd 13640  5525  0 Mar14 ?        00:00:00 /usr/lib/erlang/lib/ssl-3.1.1.1/priv/bin/ssl_esock
</code>

For memcached, you might try a command like so:
<code>
ps -aef | grep memcached
</code>

and expect output like this:
<code>
nobody   13646     1  0 Mar14 pts/6    00:00:00 /usr/bin/memcached -m 64 -p 11211 -u nobody -l 127.0.0.1
</code>

If you have trouble getting memcached and apache running, stop here and ask for help on either [[http://open-ils.org/listserv.php|OPEN-ILS-DEV]] or [[http://open-ils.org/irc.php|#evergreen]].


===== Step 4: Change the Log Levels in your config files =====
We're going to change the level of detail we log to help us narrow down any errors.  This is something you might not want to do in a production environment, because it will produce vastly larger log files that will take up more space.

To do this, we edit the OpenSRF bootstrap config file with a command like so as user opensrf:
<code>
editor /openils/conf/opensrf_core.xml
</code>

The editor involved will probably default to a Vi-clone like [[http://vim.org/|Vim]], which won't be immediately intuitive to a new user to Unix-type environments.  You may want to use an editor such as [[http://www.nano-editor.org/|nano]].

In any case, regardless of your text editor, you will want to search for every line containing <loglevel>.

By default, you will see entries like this:
<code>
<!-- 0 None, 1 Error, 2 Warning, 3 Info, 4 debug, 5 Internal (Nasty) -->
<loglevel>3</loglevel>
</code>

You will want to change every instance of loglevel 3 to loglevel 4, like so:
<code>
<loglevel>4</loglevel>
</code>

If we get really desperate with troubleshooting, we may want to use a loglevel of 5, instead.  And if we narrow things down to the database, we might want to change the logging options for Postgres as well, but I'll defer such tinkering to this [[troubleshooting:checking_for_errors_step4|page]].  If you run into trouble changing these settings, please stop here and ask for help on either [[http://open-ils.org/listserv.php|OPEN-ILS-DEV]] or [[http://open-ils.org/irc.php|#evergreen]].

===== Step 5: Start the OpenSRF router and check for errors =====
Typically, you would use a command like this as user opensrf:
<code>
/openils/bin/osrf_ctl.sh -c /openils/conf/opensrf_core.xml -l -a start_router
</code>

You should get output resembling this:
<code>
Starting OpenSRF Router
</code>

If you do not, then please stop here and ask for help on either [[http://open-ils.org/listserv.php|OPEN-ILS-DEV]] or [[http://open-ils.org/irc.php|#evergreen]].  We will want to see a copy of your router log and configuration files here:
<code>
/openils/var/log/router.log
/openils/conf/opensrf.xml
/openils/conf/opensrf_core.xml
</code>

And of course, any on-screen errors.  You should not send us files if they contain passwords that you may use in production environments.

**Note:** Evergreen 1.4 uses two routers by default, a public one and a private one, with two different logfiles:
<code>
/openils/var/log/private.router.log
/openils/var/log/public.router.log
</code>

If the router does appear to have started, then let's double check by grepping the router log file (or files) for errors like so:
<code>
grep ERR /openils/var/log/*router.log
</code>

If you encounter any output from that command, please stop here and contact the development community as stated above.

As a final sanity check, look for the router process with a command like so:
<code>
ps -aef | grep Router
</code>

You should see output like this if you're running EG1.2:
<code>
opensrf   5108     1  0 17:38 ?        00:00:00 OpenSRF Router
</code>
Or like this with EG1.4:
<code>
opensrf  22530     1  0 Mar09 ?        00:00:00 OpenSRF Router
opensrf  22531     1  0 Mar09 ?        00:00:00 OpenSRF Router
</code>

If not, please contact us.

===== Step 6: Start the OpenSRF perl services and check for errors =====

Typically, you would use a command like this as user opensrf:
<code>
/openils/bin/osrf_ctl.sh -c /openils/conf/opensrf_core.xml -l -a start_perl
</code>

With Evergreen 1.4 and later, you should get output resembling this:
<code>
Starting OpenSRF Perl
* starting all services for norcrossfx.norcross.esi
* starting servivce pid=7484 opensrf.settings
* starting servivce pid=7493 open-ils.cat
* starting servivce pid=7495 open-ils.supercat
* starting servivce pid=7497 open-ils.search
* starting servivce pid=7499 open-ils.circ
* starting servivce pid=7501 open-ils.actor
* starting servivce pid=7502 open-ils.storage
* starting servivce pid=7509 open-ils.penalty
* starting servivce pid=7512 open-ils.collections
* starting servivce pid=7514 open-ils.ingest
* starting servivce pid=7517 open-ils.permacrud
* starting servivce pid=7522 open-ils.fielder
* starting servivce pid=7527 open-ils.vandelay
* starting servivce pid=7516 open-ils.reporter
</code>
With earlier versions of Evergreen, you may simply get:
<code>
Starting OpenSRF Perl
</code>

If you do not, then please stop here and ask for help on either [[http://open-ils.org/listserv.php|OPEN-ILS-DEV]] or [[http://open-ils.org/irc.php|#evergreen]].  At a minimum, we will want to see a copy of your router log, osrfsys log and configuration files here:
<code>
/openils/var/log/router.log
/openils/var/log/osrfsys.log
/openils/conf/opensrf.xml
/openils/conf/opensrf_core.xml
</code>
And of course, any on-screen errors.  Though at this point, a copy of all files in /openils/var/log may be useful.  You should not send us files if they contain passwords that you may use in production environments.


If the Perl services do appear to have started, then let's double check by grepping the log files for errors like so:
<code>
grep ERR /openils/var/log/*.log
</code>

If you encounter any output from that command, please stop here and contact the development community as stated above.

As a final sanity check, look for OpenSRF processes with a command like so:
<code>
ps -aef | grep -i opensrf
</code>

You should see output like this:
<code>
opensrf   7453     1  0 16:04 ?        00:00:00 OpenSRF Router
opensrf   7484     1  0 16:09 ?        00:00:00 OpenSRF controller [opensrf.settings]
opensrf   7486  7484  0 16:09 ?        00:00:00 OpenSRF master [opensrf.settings]
opensrf   7487  7484  0 16:09 ?        00:00:00 OpenSRF listener [opensrf.settings]
opensrf   7488  7486  0 16:09 ?        00:00:00 OpenSRF drone [opensrf.settings]
opensrf   7489  7486  0 16:09 ?        00:00:00 OpenSRF drone [opensrf.settings]
opensrf   7490  7486  0 16:09 ?        00:00:00 OpenSRF drone [opensrf.settings]
opensrf   7491  7486  0 16:09 ?        00:00:00 OpenSRF drone [opensrf.settings]
opensrf   7492  7486  0 16:09 ?        00:00:00 OpenSRF drone [opensrf.settings]
opensrf   7493     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.cat]
opensrf   7495     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.supercat]
opensrf   7497     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.search]
opensrf   7499     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.circ]
opensrf   7501     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.actor]
opensrf   7502     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.storage]
opensrf   7503  7495  0 16:09 ?        00:00:01 OpenSRF master [open-ils.supercat]
opensrf   7504  7495  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.supercat]
opensrf   7505  7493  0 16:09 ?        00:00:01 OpenSRF master [open-ils.cat]
opensrf   7506  7497  0 16:09 ?        00:00:01 OpenSRF master [open-ils.search]
opensrf   7507  7497  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.search]
opensrf   7509     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.penalty]
opensrf   7510  7493  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.cat]
opensrf   7512     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.collections]
opensrf   7514     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.ingest]
opensrf   7516     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.reporter]
opensrf   7517     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.permacrud]
opensrf   7520  7517  0 16:09 ?        00:00:01 OpenSRF master [open-ils.permacrud]
opensrf   7521  7517  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.permacrud]
opensrf   7522     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.fielder]
opensrf   7524  7499  0 16:09 ?        00:00:01 OpenSRF master [open-ils.circ]
opensrf   7525  7522  0 16:09 ?        00:00:01 OpenSRF master [open-ils.fielder]
opensrf   7526  7522  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.fielder]
opensrf   7527     1  0 16:09 ?        00:00:00 OpenSRF controller [open-ils.vandelay]
opensrf   7529  7509  0 16:09 ?        00:00:01 OpenSRF master [open-ils.penalty]
opensrf   7531  7516  0 16:09 ?        00:00:01 OpenSRF master [open-ils.reporter]
opensrf   7532  7509  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.penalty]
opensrf   7533  7516  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.reporter]
opensrf   7534  7501  0 16:09 ?        00:00:01 OpenSRF master [open-ils.actor]
opensrf   7535  7501  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.actor]
opensrf   7536  7502  1 16:09 ?        00:00:02 OpenSRF master [open-ils.storage]
opensrf   7537  7499  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.circ]
opensrf   7538  7502  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.storage]
opensrf   7539  7512  0 16:09 ?        00:00:01 OpenSRF master [open-ils.collections]
opensrf   7540  7512  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.collections]
opensrf   7541  7514  0 16:09 ?        00:00:01 OpenSRF master [open-ils.ingest]
opensrf   7542  7527  0 16:09 ?        00:00:01 OpenSRF master [open-ils.vandelay]
opensrf   7543  7527  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.vandelay]
opensrf   7544  7514  0 16:09 ?        00:00:00 OpenSRF listener [open-ils.ingest]
opensrf   7545  7503  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.supercat]
opensrf   7546  7505  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.cat]
opensrf   7547  7524  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.circ]
opensrf   7548  7520  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.permacrud]
opensrf   7549  7520  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.permacrud]
opensrf   7550  7520  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.permacrud]
opensrf   7551  7520  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.permacrud]
opensrf   7552  7520  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.permacrud]
opensrf   7553  7525  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.fielder]
opensrf   7554  7525  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.fielder]
opensrf   7555  7525  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.fielder]
opensrf   7556  7525  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.fielder]
opensrf   7557  7529  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.penalty]
opensrf   7558  7525  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.fielder]                        
opensrf   7559  7531  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.reporter]
opensrf   7560  7534  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.actor]
opensrf   7561  7539  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.collections]
opensrf   7562  7541  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.ingest]
opensrf   7563  7541  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.ingest]
opensrf   7564  7541  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.ingest]
opensrf   7565  7542  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.vandelay]
opensrf   7566  7541  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.ingest]
opensrf   7567  7541  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.ingest]
opensrf   7568  7536  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.storage]
opensrf   7569  7506  0 16:09 ?        00:00:00 OpenSRF drone [open-ils.search]
</code>

If not, please contact us.

===== Step 7: Start the OpenSRF C services and check for errors =====

Typically, you would use a command like this as user opensrf:
<code>
/openils/bin/osrf_ctl.sh -c /openils/conf/opensrf_core.xml -l -a start_c
</code>

You should get output resembling this:
<code>
Starting OpenSRF C (host=localhost)
</code>

If you do not, then please stop here and ask for help on either [[http://open-ils.org/listserv.php|OPEN-ILS-DEV]] or [[http://open-ils.org/irc.php|#evergreen]]. At a minimum, we will want to see a copy of your router log, osrfsys log and configuration files here:
<code>
/openils/var/log/*router.log
/openils/var/log/osrfsys.log
/openils/conf/opensrf.xml
/openils/conf/opensrf_core.xml
</code>
And of course, any on-screen errors.  Though at this point, a copy of all files in /openils/var/log may be useful.  You should not send us files if they contain passwords that you may use in production environments.


If the C services do appear to have started, then let's double check by grepping the log files for errors like so:
<code>
grep ERR /openils/var/log/*.log
</code>

If you encounter any output from that command, please stop here and contact the development community as stated above.

As a final sanity check, look for OpenSRF processes with a command like so:
<code>
ps -aef | grep -i opensrf
</code>

In addition to what you saw before from the Perl services, you should see output like this:
<code>
opensrf   7780     1  0 16:39 ?        00:00:00 OpenSRF System-C
opensrf   7781  7780  0 16:39 ?        00:00:00 OpenSRF Listener [opensrf.math]
opensrf   7784  7780  0 16:39 ?        00:00:00 OpenSRF Listener [opensrf.dbmath]
opensrf   7785  7780  0 16:39 ?        00:00:00 OpenSRF Listener [open-ils.auth]
opensrf   7786  7780  0 16:39 ?        00:00:00 OpenSRF Listener [open-ils.cstore]
opensrf   7787  7780  0 16:39 ?        00:00:00 OpenSRF Listener [open-ils.reporter-store]
opensrf   7789  7781  0 16:39 ?        00:00:00 OpenSRF Drone [opensrf.math]
opensrf   7792  7784  0 16:39 ?        00:00:00 OpenSRF Drone [opensrf.dbmath]
opensrf   7793  7785  0 16:40 ?        00:00:00 OpenSRF Drone [open-ils.auth]
opensrf   7794  7786  0 16:40 ?        00:00:00 OpenSRF Drone [open-ils.cstore]
opensrf   7796  7787  0 16:40 ?        00:00:00 OpenSRF Drone [open-ils.reporter-store]
</code>

If not, please contact us.

===== Step 8: Smoke test with autogen.sh =====
The Autogen tool will take some dynamic information from the database and generate static Javascript files for use by the OPAC and staff client.  It is also able to refresh the proximity map between libraries for the purpose of efficiently routing hold requests.

As user opensrf, you invoke Autogen with a command like this:
<code>
/openils/bin/autogen.sh -c /openils/conf/opensrf_core.xml -u
</code>

You can expect output like this:
<code>
Updating fieldmapper
Updating web_fieldmapper
Updating OrgTree
removing OrgTree from the cache...
Updating OrgTree HTML
Updating locales selection HTML
Updating Search Groups
Refreshing proximity of org units
Successfully updated the organization proximity
Done
</code>

If you encounter any issues with Autogen, then please stop here and ask for help on either [[http://open-ils.org/listserv.php|OPEN-ILS-DEV]] or [[http://open-ils.org/irc.php|#evergreen]].   We will be interested in the usual log and configuration files:
<code>
/openils/var/log/*.log
/openils/conf/opensrf.xml
/openils/conf/opensrf_core.xml
</code>
And of course, please include any on-screen errors.


===== Step 9: Try to authenticate with the srfsh command-line OpenSRF client =====
As user opensrf, you should be able to invoke srfsh very simply like so:
<code>
srfsh
</code>
That command should give you a prompt which looks like this:
<code>
srfsh#
</code>
If it does not, then please stop here and ask for help on either [[http://open-ils.org/listserv.php|OPEN-ILS-DEV]] or [[http://open-ils.org/irc.php|#evergreen]].   At a minimum, we will want to see a copy of your router log, osrfsys log and configuration files here:
<code>
/openils/var/log/*router.log
/openils/var/log/osrfsys.log
/openils/conf/opensrf.xml
/openils/conf/opensrf_core.xml
/home/opensrf/.srfsh.xml
</code>
And of course, any on-screen errors.  Though at this point, a copy of all files in /openils/var/log may be useful.  You should not send us files if they contain passwords that you may use in production environments.  The .srfsh.xml file may be located elsewhere depending on your operating system, but it should be in the home directory for your opensrf user.
<code>
~/.srfsh.xml
</code>
Assuming you have a srfsh prompt, try logging in, **substituting the admin account credentials you created during the install process** into the following command:
<code>
login myAdminUsername myAdminPassword
</code>
You should see output similar to this:
<code>
Received Data: "6f63ff5542da1fead4431c6c280efc75"

------------------------------------
Request Completed Successfully
Request Time in seconds: 0.018414
------------------------------------

Received Data: {
  "ilsevent":0,
  "textcode":"SUCCESS",
  "desc":" ",
  "pid":7793,
  "stacktrace":"oils_auth.c:312",
  "payload":{
    "authtoken":"28804ebf99508496e2a4d2593aaa930e",
    "authtime":420.000000
  }
}

------------------------------------
Request Completed Successfully
Request Time in seconds: 0.552430
------------------------------------
Login Session: 28804ebf99508496e2a4d2593aaa930e.  Session timeout: 420.000000
srfsh#                                                                                        
</code>

If not, please collect the log and configuration files previously mentioned and contact us.  Also useful is the srfsh log, the location of which is defined in your ~/.srfsh.xml configuration file.  By default, it is found here:
<code>
/openils/var/log/srfsh.log
</code>

Pressing Control+D or entering "exit" will terminate srfsh.
<code>
exit
</code>

===== Step 10: Start Apache and check for errors =====

Evergreen has Apache modules that expect to find a running OpenSRF system.  Typically you could start Apache as user root with a command like
<code>
/etc/init.d/apache2 start
</code>
or
<code>
apache2ctl start
</code>

You may see output like this:
<code>
 * Starting web server apache2
   ...done.
</code>

You can check for Apache processes with a command like this:
<code>
ps -aef | grep -i apache
</code>

and can expect output like this:
<code>
root      8662     1  3 22:25 ?        00:00:01 /usr/sbin/apache2 -k start
opensrf   8667  8662  3 22:25 ?        00:00:01 /usr/sbin/apache2 -k start
opensrf   8668  8662  3 22:25 ?        00:00:01 /usr/sbin/apache2 -k start
opensrf   8669  8662  3 22:25 ?        00:00:01 /usr/sbin/apache2 -k start
opensrf   8670  8662  3 22:25 ?        00:00:01 /usr/sbin/apache2 -k start
opensrf   8671  8662  3 22:25 ?        00:00:01 /usr/sbin/apache2 -k start
</code>

Upon a successful start of the Apache OpenSRF modules, there will be a new log file:
<code>
/openils/var/log/gateway.log
</code>

However, you should check all of the log files for errors:
<code>
grep ERR /openils/var/log/*.log
</code>

If you encounter any issues with Apache, then please stop here and ask for help on either [[http://open-ils.org/listserv.php|OPEN-ILS-DEV]] or [[http://open-ils.org/irc.php|#evergreen]].   In addition to your normal log and configuration files, we'll also want to see some pertinent Apache configuration files:
<code>
/etc/apache2/eg_vhost.conf
/etc/apache2/sites-available/eg.conf
/openils/var/log/*.log
/openils/conf/opensrf.xml
/openils/conf/opensrf_core.xml
</code>
And of course, please include any on-screen errors.  The location of your Apache files may vary depending on your operating system.  Again, you should not send us files if they contain passwords that you may use in production environments.  It may also be useful to share your Apache logs:
<code>
/var/log/apache2/access.log
/var/log/apache2/error.log
</code>

===== Step 11: Try to authenticate with the staff client =====
===== Step 12: Try to authenticate with the OPAC =====
 
===== Bonus Step: Checking Ejabberd Account Credentials =====

You may have an ejabberd account problem, if you see an error like:

<code>
Checking Jabber connection for user router, domain public.localhost
Use of uninitialized value $@ in concatenation (.) or string at /usr/local/share/perl/5.10.1/OpenSRF/Transport/SlimJabber/Client.pm line 147, <DATA> line 35.
* Error connecting to jabber:
SCALAR(0x1244e20)
</code>

Verify that the username/password combinations are set correctly in opensrf_core.xml, but you can also check with ejabberd to make sure that they were setup correctly in the first place.

As root:

<code>
ejabberdctl check_password <user> <host> <password>; echo $?
</code>

If the result is 0, then things are fine with that username/password on the given host.  If the result is 1, then the credentials are not working.

You can change your ejabberd password with:

<code>
ejabberdctl change_password <user> <host> <new_password>
</code>