Hourly crash - What services are running on a schedule

💻 Core Engineering
#1

What services, or goroutines, are running on a schedule for Corteza?

We have randomly started experiencing a crash of the Corteza application container every hour. We haven’t updated recently, started any new workflows that run on a schedule, or started any new cron jobs on the host but out of the blue we started getting a crash that happens every 60-minutes.

This started this morning.

We see a spike on memory and CPU usage within the container and then the container crashes and restarts.

Wondering what services/jobs Corteza may be running on an hourly basis.

This does not happen to any of the other containers on the host.

image
image1470×412 97.9 KB

Version: 2023.9.6

#2

Can you please check 2023.9.5 and 2023.9.7 as well?

Logs or any other context or debug output always helps.

1 Like
#3

2023.9.7 isn’t mentioned in the project github as a release. We usually wait to see it there so we can see the release notes before upgrading. I do see the image available on docker hub though. Looks like it has been available for a week. I’ll test it in dev.

Logs do not show anything remarkable.

That said, are you aware of any recurring activities performed by Corteza server? Is there anything it does automatically every hour or something like that?