OTRS helpdesk upgrade
A few notes on upgrading the open source helpdesk OTRS (Open-source Ticket Resource System) Community Edition from version 5.016 (January 2017) to the latest, at the time of writing, version 6.0.40 of OTRS (April 2023). These might be helpful for others pursuing an upgrade of the open-source community variant of OTRS.
Background
We (Nikonians.org) have been using OTRS since probably 2003, on various linux distros, mainly Debian. Meanwhile we’ve processed over 200,000 tickets in the system and have been through quite a few releases of OTRS. Our appetite to switch to another system is low, especially since OTRS is solving all of our requirements as is. An upgrade to a newer version was though required since we needed to upgrade the underlying OS.
Status Quo
The current release of OTRS is 8.something and there is also a version 7 existing as well. Both of these are commercial and likely deviate a lot from the latest open source version. OTRS 6 is under GNU license.
Old style upgrades, e.g. Version 3 through 4, or 4 through 5 typically consisted of applying a monster amount of sequential patches – if you did not patch your system often. Going from 5 to 6 though is possible in a single swoosh with a single installation.
During upgrade from OTRS 5 to 6, the massive DB upgrade failed with
[Wed Oct 18 15:16:42 2023] DBUpdate-to-6.pl: DBD::mysql::db do failed: Got error 64 'Temp file write failure' from InnoDB at [removed]/Kernel/System/DB.pm line 471.
ERROR: OTRS-otrs.Console.pl-Dev::Code::CPANAudit-10 Perl: 5.24.1 OS: linux Time: Wed Oct 18 13:16:43 2023
Expanding the temp directory for MariaDB, which previously had resided, and sustained, in a partition of its own with just 650MB solved this. Reading up the docs, I gave MariaDB some 15GB temp file space.
After executing the upgrade on the Debian 8 server the OTRS system was not working at all with a segmentation fault being logged in Apache’s error log.
OTRS is written in Perl and even turning on debug messages seems to be quite a fruitless exercise and not much of any direct help was to be gained from logs.
Freexian LTS solves Debian upgrades
Upgrading the system from Debian 8 through 9 through 10 using Freexian LTS made a difference though, and was in the end the solution to get the system working well.
:/etc/apt# cat sources.list
# Debian 10 Buster - Full
deb http://deb.freexian.com/extended-lts buster main contrib non-free
# Debian 9 - Security updates
#deb http://deb.freexian.com/extended-lts stretch-lts main contrib non-free
# Debian 9 Stretch - Full
#deb http://deb.freexian.com/extended-lts stretch main contrib non-free
# Debian 8 - only security updates:
#deb http://deb.freexian.com/extended-lts jessie-lts main contrib non-free
# Debian 8 Jessie - Full
#deb http://deb.freexian.com/extended-lts jessie main contrib non-free
#deb http://archive.debian.org/debian jessie-updates main contrib non-free
#deb http://archive.debian.org jessie/updates main contrib non-free
The apt sources.list file modified thrice: Update of Jessie, then for the upgrades to Stretch and Buster, all via Freexian.
Moving from Debian 8 through 10 means going from MySQL to MariaDB and Debian 9 through 10 changes some vital DB core structure as well, so it is wise to have backups and snapshots of the system prior of doing this.
On Debian 10.13 (Buster) OTRS started to work, and kind of well. A DB script patch had for some reason failed and I needed to manually delete a column from multiple tables (some serious DB schema changes for Version 6 compared to Version 5) and the SetPermissions perl script was super generous, giving 775 access rights on all directories and 660 on all files. This is surely not required and one can, and probably should reduce these access rights further, which I did (directories on 750, scripts on 640, disabling all scripts that are not used, e.g. rpc.pl, public.pl, customer.pl, get-oauth2-token…, app.psgi, installer.pl)
Interestingly enough, the resulting behavior when accessing OTRS by GETting the index.pl (or any other OTRS script) in a browser was weird: With Apache 2.4.38 having the KeepAlive timeout set at our regular 5 seconds, the first request to OTRS went through fine, but all other requests died with an empty response body and a persistent 520 HTTP error. Reducing the KeepAlive to 1 second and I was able to get through 6 consecutive requests until the 520 error code. Turning KeepAlive off would yield the same 6 consecutive responses after fresh Apache restart until error.
FastCGI to CGI
We were using the FastCGI mod fcgid pointing the apache virtual conf script root to the fcgi dir of the OTRS installation. Switching to use cgi instead of fastcgi solved this crazy issue described above. I did not have the means to really investigate what was going on with fastcgi, but it smelled like it was cache related. e.g. as long as it had not filled up a cache with constantly fresh connections, all worked fine, but once data was being recycled, something silently crashed.
Conclusion
OTRS Community Edition 6.0.40 works like a charm and feels faster and has better functionality than 5, but it may need some serious invest of brains by the person executing the upgrade to reduce potential issues in production. I would’ve spent seriously less time if I would’ve skipped the fastcgi route and directly have gone with cgi. Some more modern routes than cgi has not been ventured in this project, even though OTRS can sit behind a proxy. We are using Apache (since OTRS seems so hard-wired with Apache code through its Perl scripts, it is probably impossible to get this working with nginx) with an SSL connection behind Cloudflare CDN sitting on a hard-locked-down Debian 10, with MariaDB 15.1 and Perl 5.28.1. I moved out all non-required perl scripts and killed the fcgi dir, doc dir and all readme files.
Side note
OTRS is picking up an avatar image from the profile site gravatar.com/avatar on a lot of page refreshes. This call can likely be removed once the system is working since it is leaking data to an external site.
Enjoy your efficient, open-source helpdesk!
Image notes
The image shows a telephone exchange and inboxes, Hotel Valhalla in northern Sweden 1955. Taken by photographer Sune Sundahl.