feat: add System Status page with Celery queue monitoring and purge#14349
feat: add System Status page with Celery queue monitoring and purge#14349valentijnscholten wants to merge 1 commit intoDefectDojo:devfrom
Conversation
valentijnscholten
requested review from
Maffooch and
mtesauro
as code owners
github-actions
bot
added
settings_changes
Maffooch
left a comment
Maffooch
left a comment
There was a problem hiding this comment.
This is a great start! Would it be possible to iterate over the task names, and get a count of those currently queued? That would allow for easier debugging into where bottlenecks are
There was a problem hiding this comment.
Please remove this, and add an entry to .gitignore to prevent it from coming in again
There was a problem hiding this comment.
Done, good find.
- Add dedicated System Status page (/system_status) with superuser-only
access, accessible from the navigation menu alongside System Settings
- Display Celery worker liveness, pending queue length with human-readable
duration formatting, and active task timeout/expiry configuration
- Add Purge queue button that POSTs to the new API endpoint and reloads
the page on success
- Fix get_celery_worker_status() to use app.control.ping() via the
pidbox control channel, which works correctly even when the task queue
is clogged (previously dispatched a task that would never be picked up)
- Add purge_celery_queue() utility using a direct broker connection
- Add two new superuser-only REST API endpoints:
GET /api/v2/celery/status/ - worker status, queue length, config
POST /api/v2/celery/queue/purge/ - purge all pending tasks
Both use the same permission guards as SystemSettingsViewSet
(IsSuperUser + DjangoModelPermissions against System_Settings)
- Add DD_CELERY_TASK_TIME_LIMIT (default 12h), DD_CELERY_TASK_SOFT_TIME_LIMIT
(default disabled), and DD_CELERY_TASK_DEFAULT_EXPIRES (default 12h)
environment variables to settings.dist.py with explanatory comments
- Move celery status rendering from server-side Django view to client-side
AJAX so dojo-pro can consume the same API endpoints
feat: add Refresh button next to Purge button on System Status page
remove plan
valentijnscholten
force-pushed
the
feature/celery-queue-status-ui
branch
from
c29ee67 to
73f690b
Compare
Redis doesn't provide a lightweight way to do this. The implementation would have to iterate over all entries in the queue and get the task name & count them. I didn't add that as it would be slow probably for clogged instances. But I could do a test to see if it's acceptable and put this behind a "Statistics" button to trigger it only when the user needs the extra level of detail. |
2 participants
Summary
/system_status) accessible to superusers from the navigation menu, alongside System Settings. Displays Celery worker liveness, pending queue length, and active task timeout/expiry settings.python manage.py dedupe). Reloads the page after a successful purge.SystemSettingsViewSet):GET /api/v2/celery/status/— worker status, queue length, and task configPOST /api/v2/celery/queue/purge/— purge all pending tasks from the broker queueget_celery_worker_status()to useapp.control.ping()via Celery's pidbox control channel instead of dispatching a task. The old approach would hang indefinitely when the task queue was clogged.settings.dist.py:DD_CELERY_TASK_TIME_LIMIT(default: 43200 = 12 h) — hard kill via SIGKILL after this many secondsDD_CELERY_TASK_SOFT_TIME_LIMIT(default: disabled) — raisesSoftTimeLimitExceededfor graceful cleanupDD_CELERY_TASK_DEFAULT_EXPIRES(default: 43200 = 12 h) — silently discard tasks that have been waiting in the queue longer than this before any worker picks them up