Workflow is hanging with "Job is about to start running on the runner"

[erichorwath]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.12.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Only intermittent...

Describe the bug

Sometimes we see workflows stuck with:

Waiting for a runner to pick up this job...
Job is waiting for a runner from 'linux-amd64-s-docker-s' to come online.
Job is about to start running on the runner: linux-amd64-s-docker-s-p45ch-runner-nvqfc (organization)

Describe the expected behavior

Workflow is assigned to a runner and workflow is processed.

Additional Context

K8s 1.32
GHES 3.16

Currently stuck with old ARC version due to https://github.com/actions/actions-runner-controller/issues/4370

Controller Logs

.

Runner Pod Logs

.

[nikola-jokic]

Hey @erichorwath,

Is this still happening or is it related to the incident we had?