Upgraded to 2021.3 but cannot access corteza

@darh @Lenny
I have upgraded to 2021.3 but when I try to access Corteza it is stuck in a loading loop

I followed the instructions.
Took Corteza offline (docker-compose down)
I then changed the version in the EML file to 2021.3, all YAML vesions are set to latest.
I then put the container back online (docker-compose up -d)

Now when I try to access corteza it gets stuck in a loading loop.

Loop goes between
https://mycorteza.domain.tld/api-mycorteza.domain.tld/auth/oauth2/default-client?redirect_uri=https://mycorteza.domain.tld/auth/callback&scope=profile%20api
and
https://mycorteza.domain.tld/api-mycorteza.domain.tld/auth/oauth2/default-client?redirect_uri=https%3A%2F%2mycorteza.domain.tld%2Fauth%2Fcallback&scope=profile%20api

My YAML file
version: ‘3.5’

services:
db:
image: percona:8.0
restart: on-failure
environment:
# To be picked up by percona image when creating the database
# Must match with DB_DSN settings inside .env
MYSQL_DATABASE: dbname
MYSQL_USER: user
MYSQL_PASSWORD: pswd
MYSQL_ROOT_PASSWORD: pswd
# healthcheck: { test: [“CMD”, “mysqladmin” ,“ping”, “-h”, “ip_address”], timeout: 20s, retries: 10 }
# Uncomment to use local fs for data persistence
volumes: [ “./data/db:/var/lib/mysql” ]
networks: [ internal ]

server:
image: cortezaproject/corteza-server:latest
restart: on-failure
env_file: [ .env ]
environment:
# Informing Corredor where it he contact us
CORREDOR_ADDR: “corredor:80”
VIRTUAL_HOST: “https://mycorteza.domain.tld
depends_on: [ db, corredor ]
volumes: [ “./data/server:/data” ]
healthcheck: { test: [“CMD”, “mysqladmin” ,“ping”, “-h”, “localhost”], timeout: 20s, retries: 10 }
ports: [ “127.0.0.1:30080:80” ]
networks: [ internal ]

corredor:
image: cortezaproject/corteza-server-corredor:latest
restart: on-failure
env_file: [ .env ]
environment:
# Informing Corredor where it he contact us
CORREDOR_ADDR: “corredor:80”
# Binds internal port to port LOCAL_DEMO_CRD_PORT on localhost
networks: [ internal ]
volumes: [ “./corredor:/corredor/usr” ]

webapp:
image: cortezaproject/corteza-webapp:latest
restart: on-failure
depends_on: [ server ]
environment:
# Monolith server in the backend, all services can be found under one base URL
MONOLITH_API: 1
# Configure web application with API location
API_BASEURL: “api-mycorteza.domain.tld”
VIRTUAL_HOST: mycorteza.domain.tld
ports: [ “127.0.0.1:30081:80” ]

networks:
internal: {}

my env

Version of Corteza Docker images

Domain=mycorteza.domain.tld
VERSION=2021.3

Database connection

DB_DSN=user:pswd@tcp(db:3306)/dbname?collation=utf8mb4_general_ci
AUTH_JWT_SECRET=secret_key

CORREDOR_EXT_SEARCH_PATHS=/extensions:/extensions/:/corredor/usr:/corredor/usr/
CORREDOR_EXEC_CSERVERS_API_HOST=api-mycorteza.domain.tld
CORREDOR_EXEC_CSERVERS_API_BASEURL_TEMPLATE=https://{host}/{service}

########################################################################################################################

SMTP (mail sending) settings

Disable email confirmation for sign-up protocol to allow more seamless setup without the need

for SMTP Server

#PROVISION_SETTINGS_AUTH_INTERNAL_SIGNUP_EMAIL_CONFIRMATION_REQUIRED=false

Point this to your local or external SMTP server

SMTP_HOST=xxx
SMTP_USER=xxx
SMTP_PASS=xxx
SMTP_FROM=’"xxx’

Same Issue

Not isolated.

We are having the same login fail loop.

Looking at source code today to identify the reason(s) - not across the cause of the issue yet.

Julian

Any joy with this.
Ive had a look as well but not entirely sure what Im looking for

Wasnt sure if it maybe ha domsething to do with
https://docs.cortezaproject.org/corteza-docs/2021.3/upgrade-guide/index.html#_authentication
The reworked authentication?

HI!

I would not know how to fix this (I’m more a low-code guy)…

Is this update done in a temporary environment (as recommended in Upgrade Guide :: Corteza Docs)? Or is it in a production environment? If so (and if it’s urgent), the best would be to get dedicated support for a quick solution.

@Lenny
Hi Lenny, thank you for your reply.
The update was done in the live production environment
If Im honest, I am not sure how to run a temporary environment which mimics the live environment. The whole docker thing is totally new to me and I have only ever used it for Corteza.
I am more used to hosting the programs and databases directly on my server

Everything I need to run my business is on Corteza - Customers, Suppliers, job records etc.

Would you have any idea as to where I should maybe look?
I have looked at different debugging solutions online but I am scared to try anything in case I make the problem worse

Solved by setting:

AUTH_BASE_URL=‘https://mydomain.tld/auth

(Thx @darn :tada:)

Fuller context:

@jsmith_dev
Can I just check, Where you added that line of code?
Would it be on the YAML file uder webapp?

ie
API_BASEURL: “api-mycorteza.domain.tld”
VIRTUAL_HOST: mycorteza.domain.tld
AUTH_BASE_URL=‘https://mydomain.tld/auth’


Tried adding it in the Webapp YAML section but no joy :frowning:

also tried in .env
DB_DSN=user:pswd@tcp(db:3306)/dbname?collation=utf8mb4_general_ci
AUTH_JWT_SECRET=secret_key
AUTH_BASE_URL=‘https://mydomain.tld/auth’
LOG_LEVEL=info

Make sure that domain in AUTH_BASE_URL actually matches your domain and that there is HTTPS in you use HTTPS!

Second thing you can check:

If you go to your db and see contents of auth_clients table. Set value on the redirect_uris column to an empty string (might just well be on all clients for now if you have more than one).

Thanks @darh I have set the AUTH_BASE_URL as mentioned above and matches the domain
Can I check… Should it be the main domain? ie - mydomain.tld
or the subdomain that corteza is accessed via, ie mycorteza.mydomain.tld?

I have tried both but none appear to be working.
The one domain I have not tried is api-mycorteza.mydomain.tld but I dont imagine that would work

Can I ask how to access the db contents that you mentioned above?

I can view a lits of tables using
docker exec -it mysql_container_name mysql -uroot -p
USE Name-Of-The-Database
show tables;
and can see the table mentioned is listed.
But not sure how to then view the contents of the specific table

I have accessed the auth_clients.ibd file over ftp and SSH and I get what looks like encrypyed text n{R<BF>^@^@^@^@^@^A8<96>^@^@^@^A^@^@^@^@<9B>.....


When I view the db log file i get
2021-04-13T17:38:39.329682Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.22-13) starting as process 1
2021-04-13T17:38:39.348389Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-04-13T17:38:39.750958Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-04-13T17:38:40.031585Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/lib/mysql/mysqlx.sock
2021-04-13T17:38:40.310420Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-04-13T17:38:40.310917Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-04-13T17:38:40.433053Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.22-13' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release 13, Revision 6f7822f.
mbind: Operation not permitted
mbind: Operation not permitted
mbind: Operation not permitted
mbind: Operation not permitted
......

I have read about cap_add: - SYS_NICE # CAP_SYS_NICE
Not tried it as I dont want to make things potentially worse

1 Like

Maybe it’s best if you revert your upgrade and restore your backup.
This way you’ll have your env running and you can first test it on test/staging env.

What is the safest way to revert the upgrade?

Things seem to be quite fragile and I don’t want to make it any worse

change image version back to .12 and restore db backup.

@darh I have reverted the back up and corteza works fine!
Does that mean that the loop I was caught it is likely something to do with the LetsEncrypt SSL that I have been using?
If so, when will it likely be safe to upgrade?

@darh,
Thus afternoon I have downloaded docker to my laptop and set up a local host version of Corteza.
I started with 2020.12 and then changed the details in the .env and YAML file to the new 2021.3 release
I then pulled the new update and ran the container on localhost.
As you will see from the pic, the update went through fine (although I did need to add AUTH_BASE_URL='http://localhost:18080/auth'

The file I used for the set up was essentially the same as the file that I use on my live environment other than changing the addresses to reflect the local host IP

The only difference I can see, other than my live enviroment being hosed on an ubuntu server and this one being hosted locally on a windows device is that my live set up uses an SSL certificate

Is there anything else that I can perhaps try to get my live enviroment updated to 2021.3

Any news on this?
as mentioned above, I used basically the same YAML file and the same .env file on local host and it has worked fine.
Does that mean that it is SSL issues and will that be sorted in the next release?

Also, I noticed that the new YAML fule does not have the webapp, instead that is added to the .env file.
Would I be better adjusting my YAML/.env files to something similar?

@darh
As mentioned above, the upgrade seemed to work fine when done in a local environment using localhost. Although my files are probably slightly different as I am not 100% sure how to create an exact replica of a live deployment for a localhost set up.

Now though at verstion 2021.3.2 when I enter my login details it just seems to refresh the login page…

I was wondering though… In terms of my live deployment.
I notice that from looking at he previous documentation, for 2020.12 there is no mention of corridor in the YAML file
in 2021.3 there is no version of corridor either and it has also removed any mention of Webapp

Should I ammend my YAML file to remove these and only keep Percona and Server?

If so
How should I ammend the server set up in the YAML
server:
image: cortezaproject/corteza-server:2021.3
restart: on-failure
env_file: [ .env ]
environment:
# Informing Corredor where it he contact us
CORREDOR_ADDR: "corredor:80"
VIRTUAL_HOST: "https://corteza.domain.tld"
depends_on: [ db, corredor ]
volumes: [ "./data/server:/data" ]
healthcheck: { test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost"], timeout: 20s, retries: 10 }
ports: [ "127.0.0.1:30080:80" ]
networks: [ internal ]

particularly
CORREDOR_ADDR: "corredor:80"
and
depends_on: [ db, corredor ]

Also,
Where would I reference:
API_BASEURL: "api-corteza.domain.tld"
and
VIRTUAL_HOST: corteza.domain.tld

Which appear in the webapp section of the YAML

That is the same in my deployment I did locally…

Yes!!! Thank you. With

AUTH_BASE_URL=http://localhost:18080/auth

in the .env - File its working at me too! Thank you!