Static files and storage

💻 Core Engineering
#1

Hello there

How to configure Corteza to store static files on the cloud? … I know Corteza supports Min.io but our cloud storage is not MinIO.

so I wonder if there is a current native solution for that or if there is a plan to improve data storage in Corteza?

thanks

#2

Minio is an interface to different cloud storage providers. Configure it to the one you wish to use.

#3

it seems I need to read more about MinIO!

as for planning for future redesign or improvements are there any?

#4

Is there anything specific you’d like to see?
We’re planning on redoing how attachments are stored and managed, but we’re not yet sure how or when.

#5

as I understand you’re working on physical schema so the record’s relationship would be changed on 2022.9 … would that anyhow affect the storage ?

#6

Storage of attachments? Probably not, but it might add to how it is right now; can’t provide a better answer on the spot, sorry.

#7

hey @tjerman, could direct me to an article or any resource on how to do that?

or is there a managed service for that?

#8

Minio’s documentation should have everything you need to know.
From Corteza’s side, it should be just some .env options.

Do keep in mind that Minio is deprecating their gateway feature.

#9

exactly that’s why I asked.
since it’s deprecating, the gateway solution wouldn’t work for me
should I deploy a MinIO cluster ? or you have a better idea!
also, what are the common approaches in Corteza community?

1 Like
#10

I also avoided minio because of the deprecation notice. Once I have a solution in production I’ll check back in here.

A quick hack that comes to mind is to use a mounted s3 file system and mount that to corteza’s container

https://hub.docker.com/r/efrecon/s3fs

#11

@shmuel , @munawir

I too was dissapointed with the Minio gateway deprecation :slight_smile:

But there are 2 options, either use Minio cluster (you can setup a one-node simple cluster as well) that actually supports encryption at rest and load balancing.

Or you can use S3 directly with Minio API that Corteza is using.
Minio explicitly states that the Minio API is compatible with S3 API, so that should work out of the box. Do note, we never tested it, so you would be in the pioneering waters on that one.

#12

I’m not confident that I understood how to

If I’m not mistaken you mean that you can setup minio using the latest release and within minio setup your storage endpoint using s3 configuration parameters (bucket,uri,key, and token).

Below is an example of your first idea, setting up your own minio cluster.

I added the following service definition to my docker compose file and it allows me to successfully upload files to s3 via corteza.

  minio:
    image: minio/minio:RELEASE.2022-10-24T18-35-07Z.hotfix.ce525fdaf
    container_name: minio
    volumes:
      - "minio_data:/data"
    environment:
        MINIO_ROOT_USER: YOUR_MINIO_ACCESS_KEY
        MINIO_ROOT_PASSWORD: YOUR_MINIO_SECRET_KEY
    expose:
        - "9000"
        - "9001"
    ports:
        - "9000:9000"
        - "9001:9001"
    command: 'gateway s3 --console-address "127.0.0.1:9001"'
  ...
volumes:
  ...
  minio_data:

The significance of the version RELEASE.2022-10-24T18-35-07Z.hotfix.ce525fdaf is that it’s the latest release that supports minio gateway.

My corteza env file looks so:

MINIO_ENDPOINT=minio:9000
MINIO_SECURE=false
MINIO_ACCESS_KEY=YOUR_MINIO_ACCESS_KEY
MINIO_SECRET_KEY=YOUR_MINIO_SECRET_KEY
MINIO_BUCKET=YOUR_BUCKET
MINIO_PATH_PREFIX={component}/

I’m not sure if we need both the access key/secret key in the env file and docker-compose file, but this does work.

#13

Thanks @shmuel for this nice example.

We are currently not using the gateway feature with Minio as we dont want to have a non-supported version.

What I meant with S3 API was, you should be able to use the s3 host and bucket name with our MINIO conf (so without running Minio).

An example of mc support with S3 would be here:

#14

Ahh, I understand what you mean. Minio essentially just acts as an s3 client. I didn’t realize it could do that without a dedicated service. This is a much lighterweight solution :smiley: .

For future readers the way to set this up in your corteza env file is:

MINIO_ENDPOINT=s3.amazonaws.com
MINIO_ACCESS_KEY=YOUR_MINIO_ACCESS_KEY
MINIO_SECRET_KEY=YOUR_MINIO_SECRET_KEY
MINIO_BUCKET=YOUR_BUCKET
MINIO_PATH_PREFIX={component}/

No extra docker compose service necessary.

1 Like
#15

Does corteza have the ability to cache files retreived from the object store?

Let’s say I fetch a preview of an image from s3, does it have to do the fetch from s3 everytime, or can I define a TTL for caching for faster access?

#16

@shmuel , thanks for the quick check-up, we will definitely include this in the documentation.

No caching of files is supported for now.

2 Likes
#17

my failed try with minio cluster