Guidance on performance and production setup

Hello there,

First and foremost, Corteza is a great low-code platform with numerous out-of-the-box capabilities that is also open source; congrats to the planet crust team and fellow community members :ok_hand:

I’m thinking about using Corteza for a live project and have a few queries. Kindly note that I have successfully installed Corteza locally and have read through the documentation quickly.

  1. Do we have any performance metrics for Corteza? As far as I can tell, the equivalent database structure is as follows:
  • Table gets translated to compose_module
  • Table columns to compose_module_field
  • Each table row mapped to compose_record
  • Actual row values to compose_record_value

I’ve usually worked with configuration in which a certain object, like Products, has its own table with appropriate columns (fields) and records (rows). With the corteza structure described above, I think all records for all tables/modules are stored in the compose record value table, which may expand enormously in a high volume system. For example, if we have 2M records for each module/table, we may have 200M records with 100 modules. I’m curious whether this can have an effect on database and app performance. Maybe someone who is operating such a system may provide some thoughts.

  1. Is there a way to manually insert records (through sql query) and map them according to Corteza’s configuration, i.e. module, field, records?

  2. Is it feasible to archive/move data, for example, if we manage a production instance with recent data and an archive instance with older data (perhaps solely for retrieval/selection through web-app)?

  3. I’m searching for instructions on how to compile the source code so that I may run the different components (Server, corredor, web-app) independently and perhaps in different systems (Windows/Linux). Any advice would be much appreciated.

Thanks,
Drb

1 Like

@peter @AndreB did we not do some benchmarking a while ago? Could you share that?

Yes, this would have an impact on the performance.
The next version (2022.9) introduces support for using stand-alone tables for your records.

You can, but it would probably be faster to do it via the API (to get record IDs generated for you).

Those stand-alone tables would let you add those tables as a source so you could use them for retrieval.

Different components are already available as their own Docker image (look through our DockerHub profile).

If you wish to compile your own, have a look at Docker files and GitHub actions; there are no docs for this as of now.

Thanks @tjerman for your swift response, you’re indeed a Corteza rockstar :smiley:

Do you have any timelines for the next version (2022.9)? I believe it will be the best option in my case, especially because I will be using standalone-tables.

1 Like

The version denotes the year/month of when it is released – 2022.9 → this year, ~end of September.
Dev/RC releases will be available before then (depending on how fast we can make new features stable).

1 Like

Hi @tjerman, thanks for your explanation. 2022.9 Stand-alone tables are good news.

I’m using corteza 2022.3 now. When 2022.9 arrives, what will happen to my data residing in compose_record_value, will data be migrated automatically to stand-alone tables, or will I have to migrate them manually?

Also, I think and “index” option in module’s fields would be suitable for stand-alone tables.

The current state of compose_record and compose_record_value tables is also changing. compose_record_value are getting embedded into compose_record as a JSON to reduce complexity and help with performance (some databases can create indexes over JSON parameters).

I can’t tell you what exactly is going to happen with your current data, but it will probably be migrated to the new compose_record table.
There will be some code to migrate records/modules to dedicated tables but I can’t say for sure what the defaults will be.

Worst case you’ll define indexes manually directly in your DB but having an option in the UI does sound sane to me and I think we talked about it as well.

1 Like

Thanks so much @tjerman.
I hope the defaults to be stand-alone tables and ui indexes, maybe with stand-alone tables as default, some day corteza wants to include optional table partitioning to improve performance even more.

1 Like