celery service crashing on tor-weather after system reboot due to missing pid-file
Celery logs on the weather box suggest that post the system reboot on 3rd April, celery service had error starting due to a missing pid file.
Logs from journalctl --user -u tor-weather-celery.service
Apr 03 18:18:29 weather-01 systemd[599]: Stopping tor-weather-celery.service - Run the celery service for tor-weather...
Apr 03 18:18:30 weather-01 sh[1673238]: celery multi v5.3.6 (emerald-rush)
Apr 03 18:18:30 weather-01 sh[1673238]: > worker1@weather-01: DOWN
Apr 03 18:18:31 weather-01 systemd[599]: Stopped tor-weather-celery.service - Run the celery service for tor-weather.
Apr 03 18:18:31 weather-01 systemd[599]: tor-weather-celery.service: Consumed 4h 42min 52.923s CPU time.
-- Boot 2d0316f58fa84fb984ecce63cef42a1a --
Apr 03 18:21:07 weather-01 systemd[597]: Starting tor-weather-celery.service - Run the celery service for tor-weather...
Apr 03 18:21:07 weather-01 systemd[597]: tor-weather-celery.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Apr 03 18:21:07 weather-01 systemd[597]: tor-weather-celery.service: Failed with result 'exit-code'.
Apr 03 18:21:07 weather-01 systemd[597]: Failed to start tor-weather-celery.service - Run the celery service for tor-weather.
Apr 03 18:21:07 weather-01 systemd[597]: tor-weather-celery.service: Scheduled restart job, restart counter is at 1.
Apr 03 18:21:07 weather-01 systemd[597]: Stopped tor-weather-celery.service - Run the celery service for tor-weather.
Apr 03 18:21:07 weather-01 systemd[597]: Starting tor-weather-celery.service - Run the celery service for tor-weather...
Apr 03 18:21:07 weather-01 systemctl[700]: activating
Apr 03 18:21:07 weather-01 systemd[597]: tor-weather-celery.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Apr 03 18:21:07 weather-01 systemd[597]: tor-weather-celery.service: Failed with result 'exit-code'.
Apr 03 18:21:07 weather-01 systemd[597]: Failed to start tor-weather-celery.service - Run the celery service for tor-weather.
Apr 03 18:21:08 weather-01 systemd[597]: tor-weather-celery.service: Scheduled restart job, restart counter is at 2.
Apr 03 18:21:08 weather-01 systemd[597]: Stopped tor-weather-celery.service - Run the celery service for tor-weather.
Apr 03 18:21:08 weather-01 systemd[597]: Starting tor-weather-celery.service - Run the celery service for tor-weather...
Apr 03 18:21:08 weather-01 systemctl[890]: activating
Apr 03 18:21:08 weather-01 systemd[597]: tor-weather-celery.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Apr 03 18:21:08 weather-01 systemd[597]: tor-weather-celery.service: Failed with result 'exit-code'.
Apr 03 18:21:08 weather-01 systemd[597]: Failed to start tor-weather-celery.service - Run the celery service for tor-weather.
Apr 03 18:21:08 weather-01 systemd[597]: tor-weather-celery.service: Scheduled restart job, restart counter is at 3.
Apr 03 18:21:08 weather-01 systemd[597]: Stopped tor-weather-celery.service - Run the celery service for tor-weather.
Apr 03 18:21:08 weather-01 systemd[597]: Starting tor-weather-celery.service - Run the celery service for tor-weather...
Apr 03 18:21:08 weather-01 systemctl[1043]: activating
Apr 03 18:21:08 weather-01 systemd[597]: tor-weather-celery.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Apr 03 18:21:08 weather-01 systemd[597]: tor-weather-celery.service: Failed with result 'exit-code'.
Apr 03 18:21:08 weather-01 systemd[597]: Failed to start tor-weather-celery.service - Run the celery service for tor-weather.
Apr 03 18:21:08 weather-01 systemd[597]: tor-weather-celery.service: Scheduled restart job, restart counter is at 4.
Apr 03 18:21:08 weather-01 systemd[597]: Stopped tor-weather-celery.service - Run the celery service for tor-weather.
Apr 03 18:21:08 weather-01 systemd[597]: Starting tor-weather-celery.service - Run the celery service for tor-weather...
Apr 03 18:21:08 weather-01 systemctl[1113]: activating
Apr 03 18:21:08 weather-01 systemd[597]: tor-weather-celery.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Apr 03 18:21:08 weather-01 systemd[597]: tor-weather-celery.service: Failed with result 'exit-code'.
Apr 03 18:21:08 weather-01 systemd[597]: Failed to start tor-weather-celery.service - Run the celery service for tor-weather.
Apr 03 18:21:09 weather-01 systemd[597]: tor-weather-celery.service: Scheduled restart job, restart counter is at 5.
Apr 03 18:21:09 weather-01 systemd[597]: Stopped tor-weather-celery.service - Run the celery service for tor-weather.
Apr 03 18:21:09 weather-01 systemd[597]: tor-weather-celery.service: Start request repeated too quickly.
Apr 03 18:21:09 weather-01 systemd[597]: tor-weather-celery.service: Failed with result 'exit-code'.
Apr 03 18:21:09 weather-01 systemd[597]: Failed to start tor-weather-celery.service - Run the celery service for tor-weather.
Apr 13 16:21:39 weather-01 systemd[597]: Starting tor-weather-celery.service - Run the celery service for tor-weather...
Logs from journalctl --user -u tor-weather-celerybeat.service
-- Boot 2d0316f58fa84fb984ecce63cef42a1a --
Apr 03 18:21:07 weather-01 systemd[597]: Started tor-weather-celerybeat.service - Run the celerybeat service for tor-weather.
Apr 03 18:21:10 weather-01 sh[631]: celery beat v5.3.6 (emerald-rush) is starting.
Apr 03 18:21:10 weather-01 sh[631]: Traceback (most recent call last):
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/lib/python3.11/site-packages/celery/platforms.py", line 138, in acquire
Apr 03 18:21:10 weather-01 sh[631]: self.write_pid()
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/lib/python3.11/site-packages/celery/platforms.py", line 207, in write_pid
Apr 03 18:21:10 weather-01 sh[631]: pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Apr 03 18:21:10 weather-01 sh[631]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 03 18:21:10 weather-01 sh[631]: FileNotFoundError: [Errno 2] No such file or directory: '/run/user/1508/tor-weather/celery/beat.pid'
Apr 03 18:21:10 weather-01 sh[631]: During handling of the above exception, another exception occurred:
Apr 03 18:21:10 weather-01 sh[631]: Traceback (most recent call last):
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/bin/celery", line 8, in <module>
Apr 03 18:21:10 weather-01 sh[631]: sys.exit(main())
Apr 03 18:21:10 weather-01 sh[631]: ^^^^^^
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/lib/python3.11/site-packages/celery/__main__.py", line 15, in main
Apr 03 18:21:10 weather-01 sh[631]: sys.exit(_main())
Apr 03 18:21:10 weather-01 sh[631]: ^^^^^^^
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/lib/python3.11/site-packages/celery/bin/celery.py", line 236, in main
Apr 03 18:21:10 weather-01 sh[631]: return celery(auto_envvar_prefix="CELERY")
Apr 03 18:21:10 weather-01 sh[631]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
Apr 03 18:21:10 weather-01 sh[631]: return self.main(*args, **kwargs)
Apr 03 18:21:10 weather-01 sh[631]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
Apr 03 18:21:10 weather-01 sh[631]: rv = self.invoke(ctx)
Apr 03 18:21:10 weather-01 sh[631]: ^^^^^^^^^^^^^^^^
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
Apr 03 18:21:10 weather-01 sh[631]: return _process_result(sub_ctx.command.invoke(sub_ctx))
Apr 03 18:21:10 weather-01 sh[631]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
Apr 03 18:21:10 weather-01 sh[631]: return ctx.invoke(self.callback, **ctx.params)
Apr 03 18:21:10 weather-01 sh[631]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
Apr 03 18:21:10 weather-01 sh[631]: return __callback(*args, **kwargs)
Apr 03 18:21:10 weather-01 sh[631]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
Apr 03 18:21:10 weather-01 sh[631]: return f(get_current_context(), *args, **kwargs)
Apr 03 18:21:10 weather-01 sh[631]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 03 18:21:10 weather-01 sh[631]: File "/home/weather/tor-weather-venv/lib/python3.11/site-packages/celery/bin/base.py", line 134, in caller
Apr 03 18:21:10 weather-01 sh[631]: return f(ctx, *args, **kwargs)
Due to this, the service seems to be partially down since 3rd April. I have restarted both the tor-weather-celery.service
& tor-weather-celerybeat.service
, but doesn't seem like that has helped.