Critical software update (Daylight saving time)
Issue description
On a night, when clocks are turned forward 1 hour, the application scheduler incorrectly interprets system time and starts the scheduled tasks/jobs in a loop.
This causes excessive CPU usage and may cause "out-of-disk-space" issue due to grow of log file in size.
Affected versions
This bug existed in releases between:
- 6.0.0.1029 (released on August 20th, 2018) and
- 7.0.0.447 (released on March 27th, 2020).
To check a version of your MiaRec software, navigate in web portal to Administration -> Maintenance -> Version:
Important! The issue doesn't occur if operating system is configured with UTC timezone on system level.
To verify what timezone is configured on OS level, on Linux system, run the following command to verify system timezone:
date +"%Z %z"
If you see UTC +0000
in the output, then your system is not affected by this bug.
You do not need to do anything if your system is not affected!
Mitigation
If your system is affected, then there are two options to mitigate the issue:
- Short-term solution
- Long-term solution
Option1. Short term solution
If you do not have time to apply a long term solution, then we recommend to do the following:
Before a night when clocks are turned forward 1 hour, stop the MiaRec's task scheduler service.
On Linux, run the following command via SSH:
service celerybeat stop
On Windows, navigate to Control Panel -> Services panel, and stop the service MiaRec Celery Scheduler.
After clocks are turned forward 1 hour, start the previously stopped service by running command service celerybeat start
on Linux or by starting MiaRec Celery Scheduler service on Windows.
Explanation: The Celery Scheduler (celerybeat) service is responsible for triggering the scheduled tasks at the specified time (like every night, every hour, every 5 minutes etc). If this service is not running, then tasks are not run by schedule. It is safe to stop this service for a short period of time. Once it is started again, the scheduled tasks/jobs will catch-up and process the pending data. A "short period of time" could be hours or days depending on your system load. As a rule of thumb, if you record less than 500 users on the system, then you can safely stop the scheduler service for a few days, for example, before a weekend and then start on Monday.
Option 2. Long-term solution
For a long term solution, it is a necessary to update MiaRec software to the latest release. You can get the latest version by contacting your MiaRec representative.
Questions?
If you have any questions, contact our support team at support@miarec.com