Limitations with large number of executed workflows

[munawir]

Hi all,

I’ve been using Corteza for 3 months now and I’ve noticed some kind of critical limitation regarding executing workflows.

when we define a trigger we specify its constraints as much as possible.
this is suggested by the docs to reduce the amount of the executed workflows when they shouldn’t

Constraint your triggers as much as possible to prevent your workflows from executing when they shouldn’t.

the thing is we only can specify two fields the Module and the Namespace and those fields are Corteza specific. What if we include domain-specific fields from the module itself. for example,
if I have a module that has a field called status and I want a workflow to run only on the records which have Close as a status value.
if I have 20K of records in that module, the system will suffer from unwanted automation executions. what I currently do is that my workflow has Gateway(exclusive) after the trigger to check if the status is not Close then it terminates.

the following issues are critical and limit us in the company to depend 100% on Corteza

• we’ve noticed some of the triggers don’t even run > this is because our servers dropped them(I’m not sure if that is a valid reason). so we increased the number of our AWS servers to handle this fake load. which increased the bill
• when we monitor our sessions to see what workflows have run due to specific operations. we’ve found it so hard since they’re a lot of automation running at the nearly same time.
• if that trigger is set to be BeforeUpdate for example, and I update the record from the UI, it takes too long. so save that record. which damages the experience of the platform
• in some cases (and I’m not sure if that is the reason), the platform froze for any write operation. such as login or creating a new record. so the server will not respond to any write operation until we restart the server.
• lastly, it makes creating a workflow a bit difficult since we need to check every time after the trigger if that workflow should run or not.

I wonder if you guys are aware of such problems and if you’re working on a solution(if yes, please share your timeline if possible), and if not it would be my pleasure to clone my environment and help you to reproduce them one by one.
@tjerman @peter @darh @mia.arh I wrote this due to our previous communication in different topics, please do let me know if you need more clarifications on any of the points above.

I would very much appreciate it if you guys could technically explain the reasons behind my points above so I can pass it down to my manager to see if we can allocate some engineering resources to develop the feature or even a proper solution.

Thanks a lot , and sorry for the lengthy topic

[iit-dev]

How many records do you have in your database, I am considering change my project: lead management to Corteza with more than 500k records.

[munawir]

we’ve just started and the overall records would around a 100K

I don’t think the number of the records would be an issue unless they’re in one module and you have automations triggers on that module.
for example if you have a module that has 80K of records and you have triggers run on every record and that happen regularly, I think it would be a desaster

[tjerman]

@munawir

the thing is we only can specify two fields the Module and the Namespace and those fields are Corteza specific. What if we include domain-specific fields from the module itself. for example,
if I have a module that has a field called status and I want a workflow to run only on the records which have Close as a status value.

I’ll make a proposal to add field-based constraints as it does make sense to have.

As far as your concrete case goes; a bit back (before workflows but the concept is similar) we had to do a bunch of little automation scripts where tiny scripts would piece together the final outcome and it became a bit of a nightmare to maintain.

Perhaps you could consider joining smaller workflows (such as before update Closed leads, before update Open leads) into a single workflow to reduce the number of them.

@munawir

• we’ve noticed some of the triggers don’t even run > this is because our servers dropped them(I’m not sure if that is a valid reason). so we increased the number of our AWS servers to handle this fake load. which increased the bill

Corteza only drops workflows with invalid configurations so this would seem like a bug.
How many workflows are you running?

@munawir

• when we monitor our sessions to see what workflows have run due to specific operations. we’ve found it so hard since they’re a lot of automation running at the nearly same time.

Here, are you wishing for a nicer UI? In the admin panel there is a whole page dedicated to automation sessions

If you wish for a nicer UI write down some notes/suggestions and we’ll see what we can see.

@munawir

• if that trigger is set to be BeforeUpdate for example, and I update the record from the UI, it takes too long. so save that record. which damages the experience of the platform
• in some cases (and I’m not sure if that is the reason), the platform froze for any write operation. such as login or creating a new record. so the server will not respond to any write operation until we restart the server.

We’ve encountered some (general and workflow related) performance issues (workflows taking long to execute, the platform freezing, …) and we’re (by we I mean @darh and @peter ) working on resolving them.

@darh/@peter if you have something to add here please do

@munawir

and if not it would be my pleasure to clone my environment and help you to reproduce them one by one.

If you are able to get us some reproducible setup that would be amazing.

[munawir]

@tjerman

I’ll make a proposal to add field-based constraints as it does make sense to have.

looking forward

@tjerman

Corteza only drops workflows with invalid configurations so this would seem like a bug.
How many workflows are you running?

around 10K triggers at the same time

@tjerman

are you wishing for a nicer UI?

No, I know the Sessions page.
I meant when looking at the sessions time, I see all have the same time (even in seconds)
this is correct because the 10K triggers ran at the same time.
since you mentioned nicer UI, I would suggest including the input/output data of the automation so we will have a better understanding

@tjerman

We’ve encountered some (general and workflow related) performance issues

we highly depend on the before-event to fill out some data of the record. and that in concept will reduce the performance till running all the automation(synchronous approach). while if we have better support for partial-update, we would create/update the record and run automation to fill out the missing fields( asynchronous approach). wouldn’t hurt if it take a few seconds because it’s running in the background.
I tried to do this but I got a lot of Steal status error
cc @darh @peter

@tjerman

If you are able to get us some reproducible setup that would be amazing.

will ask my team for a clone

[darh]

With the last (patch) release, we’ve fixed a couple of memory-leaks. Make sure you upgrde @munawir

[munawir]

@darh already using 2022.3.1

if I can’t control the triggers based on module fields, I would say it’s hard to manage with large records