Server-scripts execution delay after Corteza server start

#1

Hello,

After restarting the Corteza server, there is a delay of approximately 1 minute before server-scripts with triggers are recognized and executed correctly. During this delay, the Corteza web application is accessible, but server-scripts do not seem to be fully initialized or operational.

Immediately After Restart:

  • The Corteza web application becomes accessible.
  • Adding a record in a module does not trigger server-scripts as expected.
  • The “Admin / Corredor Scripts” section remains empty.

After About 1 Minute:

  • Adding a record in a module causes the server-scripts to execute as expected.
  • The “Admin / Corredor Scripts” section has the list of scripts

In my docker-compose.yaml file, the server service depends on the corredor service.

Relevant Logs:

From the Corteza server logs:

#First try
22:05:15.523    DEBUG   actionlog       looked-up user ...
22:05:18.523    DEBUG   actionlog       updated 123456789123456789 ...
22:05:19.333    ERROR   corredor        could not load corredor client scripts  {"error": "rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:53051: connect: connection refused\""}
.....
22:05:28.393    DEBUG   corredor        script registered       {"script": "/server-scripts/Sample.js:default", "explicit": 0, "triggers": 1}
....
#Second try
22:05:32.393    DEBUG   corredor         triggered       {"script": "/server-scripts/Sample.js:default" ....}
22:05:32.595    DEBUG   actionlog       updated 123456789123456789 ...

In this log, it took about 10 seconds for the script to be registered while the Corteza server was operational.

Could you please advise on how to ensure that the Corteza server or its web application does not become fully operational until the Corredor service has completely initialized and processed all scripts ?

Best regards,

1 Like
#2

As is, I’m not sure you can do anything about it, the two bits are more or less detached.

You could try setting depende_on in your docker-compose.

If you tell the corteza service it depends on corredor, it should not start up until Corredor is up and running.

Is this causing issues in production? I’m guessing the server doesn’t restart often; that plus scheduling restarts sound reasonable in my head.

How many Corredor scripts do you have?

#3

This configuration actually has only 1 server script. I guess with more scripts the delay can be bigger.

Corridor service is already added to depende_on docker-compose yaml.
Corridor server actually starts, but for some reasons, it takes some time to read the scripts.

I thought about two solutions :

1. Edit the initial healthcheck on corteza server:
→ Exemple in corteza/server/pkg/corredor/healthcheck.go
→ Add a condition that throws an error if corridor didn’t read all scripts via communication with corridor server

2. Edit the corteza-server-corredor to wait for the scripts to be read:
→ edit ENTRYPOINT / healthcheck to make it wait for a ready flag
→ edit server-script.ts to make the ready flag once the loader finishes its job.

I think solution number 1 is better but solution number 2 seems to be simpler.
I will try to make a pull request with the modifications for solution 2 but I might need some help.

In Docker Compose versions 3.x and above, the condition field in depends_on was removed. which makes waiting for another service to be healthy impossible.

I noticed the problem when a user created a record just after the server restarted. A server script was supposed to do the auto-increment but it did’t happened. After some debugging I noticed that corridor restart working after some delay.

#4

I was able to resolve the problem with :
→ for corteza : adding condition: service_healthy to depends_on option
(some stackoverflow posts said that condition where removed but after some digging it was not true).
→ for corridor : editing healthcheck : [“CMD”, “nc”, “-z”, “-v”, “corredor”, “53051”]

Corteza won’t start before corridor starts and with state healthy (instead of starting).

docker-compose.yaml

server:
    depends_on:
      db:
        condition: service_healthy
      corredor:
        condition: service_healthy

  corredor:
    env_file: [ .env ]
    image: cortezaproject/corteza-server-corredor:${VERSION}
    restart: on-failure
    networks: [ internal ]
    volumes:
      - "./extensions:/extensions"
    healthcheck: { test: ["CMD", "nc", "-z", "-v", "corredor", "53051"], interval: 30s, start_period: 1m, timeout: 30s, retries: 3}

Just for info, corridor logs :

{"level":30,"time":1723154801706,"name":"gRPC","msg":"server stopped","v":1}
{"level":30,"time":1723154825831,"name":"check","msg":"checking API endpoints","v":1}

It means that corredor takes around 24 seconds to start.

used version 2023.9.7

1 Like