Upgrading to 2022.3.2 issue

🚧 Installation
#1

Hi everyone

I’m trying to upgrade from 2022.3.1 to 2022.3.2
unfortunately, I’m facing an issue
On the web I get Corteza server initializing
and on the server log, I keep getting these two logs (for more than 2 hours)

server_1    | 2022/05/23 18:28:22 http: superfluous response.WriteHeader call from github.com/cortezaproject/corteza-server/pkg/api/server.panicRecovery (middleware.go:46)
server_1    | 18:28:22.854	DEBUG	crashed on http request	{"error": "runtime error: invalid memory address or nil pointer dereference", "stack": "goroutine 3722 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/cortezaproject/corteza-server/pkg/api/server.panicRecovery({0x342b530, 0xc0007d4400}, {0x3412430, 0xc009fce7e0})\n\tgithub.com/cortezaproject/corteza-server/pkg/api/server/middleware.go:44 +0x2ef\npanic({0x29774e0, 0x45764a0})\n\truntime/panic.go:1038 +0x215\ngoogle.golang.org/grpc.(*ClientConn).GetState(...)\n\tgoogle.golang.org/grpc@v1.45.0/clientconn.go:527\ngithub.com/cortezaproject/corteza-server/pkg/corredor.Healthcheck({0x457c060, 0x0})\n\tgithub.com/cortezaproject/corteza-server/pkg/corredor/healthcheck.go:22 +0x35\ngithub.com/cortezaproject/corteza-server/pkg/healthcheck.(*checks).Run(0xc00046eba0, {0x342b5d8, 0xc00038e780})\n\tgithub.com/cortezaproject/corteza-server/pkg/healthcheck/check.go:68 +0x135\ngithub.com/cortezaproject/corteza-server/pkg/api/server.waitingRoutes.func2({0x3412430, 0xc009fce7e0}, 0xc00a1f5a00)\n\tgithub.com/cortezaproject/corteza-server/pkg/api/server/handlers.go:47 +0x170\nnet/http.HandlerFunc.ServeHTTP(0x29589e0, {0x3412430, 0xc009fce7e0}, 0xc0112ef9e4)\n\tnet/http/server.go:2047 +0x2f\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc00002daa0, {0x3412430, 0xc009fce7e0}, 0xc00a1f5a00)\n\tgithub.com/go-chi/chi/v5@v5.0.7/mux.go:442 +0x216\nnet/http.HandlerFunc.ServeHTTP(0xc000267860, {0x3412430, 0xc009fce7e0}, 0xc00a1f5a00)\n\tnet/http/server.go:2047 +0x2f\ngithub.com/go-chi/cors.(*Cors).Handler.func1({0x3412430, 0xc009fce7e0}, 0xc00a1f5a00)\n\tgithub.com/go-chi/cors@v1.2.0/cors.go:228 +0x1bd\nnet/http.HandlerFunc.ServeHTTP(0x2b584e0, {0x3412430, 0xc009fce7e0}, 0x4575f40)\n\tnet/http/server.go:2047 +0x2f\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc00002daa0, {0x3412430, 0xc009fce7e0}, 0xc00a1f5900)\n\tgithub.com/go-chi/chi/v5@v5.0.7/mux.go:88 +0x442\ngithub.com/cortezaproject/corteza-server/pkg/api/server.(*demux).ServeHTTP(0x0, {0x3412430, 0xc009fce7e0}, 0xc00a1f5900)\n\tgithub.com/cortezaproject/corteza-server/pkg/api/server/demux.go:51 +0x11c\nnet/http.serverHandler.ServeHTTP({0xc00038e6c0}, {0x3412430, 0xc009fce7e0}, 0xc00a1f5900)\n\tnet/http/server.go:2879 +0x43b\nnet/http.(*conn).serve(0xc009f36d20, {0x342b5d8, 0xc000592270})\n\tnet/http/server.go:1930 +0xb08\ncreated by net/http.(*Server).Serve\n\tnet/http/server.go:3034 +0x4e8\n"}

I tried installing it with fresh DB and everything works fine
also, I installed 2022.3.1 on fresh DB and upgraded it to 2022.3.2 and everything works fine too

the above error is on my staging server
note: the staging DB is a backup from the production DB which has a lot of data and the initial installation was 2021.9.0

I need the 2022.3.2 due to a bug fix related to date filtering … thanks

#2

Can you maybe share the request that’s causing this?

#3

@darh I’m not sure how to get you the request (since the platform is not accessible while upgrading) but this is the logs before the above error

server_1    | 19:23:14.671	DEBUG	http.waiting	version route enabled: /version
server_1    | 19:23:14.671	DEBUG	http.waiting	health check route enabled: /healthcheck
server_1    | [Sentry] 2022/05/23 19:23:14 Integration installed: ContextifyFrames
server_1    | [Sentry] 2022/05/23 19:23:14 Integration installed: Environment
server_1    | [Sentry] 2022/05/23 19:23:14 Integration installed: Modules
server_1    | [Sentry] 2022/05/23 19:23:14 Integration installed: IgnoreErrors
server_1    | 19:23:14.679	INFO	http	starting HTTP server	{"path-prefix": "/", "address": "0.0.0.0:80"}
server_1    | 19:23:14.715	DEBUG	store.rdbms	opening connection	{"driver": "postgres", "dsn": "p****s:B****k@***-corteza.ccte*****cp3.*******-1.rds.amazonaws.com/corteza?sslmode=disable"}
server_1    | 19:23:14.715	DEBUG	store.rdbms	setting connection parameters	{"MaxOpenConns": 256, "MaxLifetime": "10m0s", "MaxIdleConns": 32}
server_1    | 19:23:14.731	DEBUG	store.rdbms	connected to the database
server_1    | 19:23:14.731	INFO	running store update
server_1    | 19:23:14.731	INFO	store upgrade running (to enable upgrade debug logging set UPGRADE_DEBUG=true)
server_1    | 19:23:14.949	INFO	updating system role	{"handle": "super-admin", "ID": 0}
server_1    | 19:23:14.949	INFO	updating system role	{"handle": "authenticated", "ID": 0}
server_1    | 19:23:14.949	INFO	updating system role	{"handle": "anonymous", "ID": 0}
server_1    | 19:23:14.968	DEBUG	system entities set	{"users": [268373446688243714, 268373446688309250, 268373446705020930], "roles": [268373446671466498, 268373446671532034, 268373446671597570]}
server_1    | 19:23:15.021	INFO	provision.pre-202109-roles	migrating pre-2021.9 roles
server_1    | 19:23:15.026	INFO	provision.pre-202109-rbac-rules	migrating RBAC rules to new format	{"rules": 301}
server_1    | 19:23:15.348	INFO	provision.pre-202109-settings	cleaning up pre-2021.9 settings
server_1    | 19:23:15.368	INFO	provision.resource-translations	migrating resource locales
server_1    | 19:23:15.385	INFO	provision.report-identifiers	migrating report identifiers
server_1    | 19:23:15.401	INFO	provision.post-202203-rbac-resource-rules-fix	migrating RBAC resource rules to proper format	{"rules": 301}
#4

Nevermind, I see the problem now. We’ll try to address it asap.

You can ignore the error for now.

#5

Looks like corredor-related issue.
Are you using it? If not, can you disable it?

#7

@darh I disabled corredor and I set UPGRADE_DEBUG=true in the env file

the long error doesn’t appear anymore in the logs and on the web, I got this

Corteza server initializing
FAIL Scheduler: stopped
PASS Mail
PASS Corredor
PASS Store/RDBMS/postgres

here are the logs (and they are not continuing)

server_1  | 20:02:26.760	DEBUG	http.waiting	version route enabled: /version
server_1  | 20:02:26.760	DEBUG	http.waiting	health check route enabled: /healthcheck
server_1  | [Sentry] 2022/05/23 20:02:26 Integration installed: ContextifyFrames
server_1  | [Sentry] 2022/05/23 20:02:26 Integration installed: Environment
server_1  | [Sentry] 2022/05/23 20:02:26 Integration installed: Modules
server_1  | [Sentry] 2022/05/23 20:02:26 Integration installed: IgnoreErrors
server_1  | 20:02:26.785	INFO	http	starting HTTP server	{"path-prefix": "/", "address": "0.0.0.0:80"}
server_1  | 20:02:26.830	DEBUG	store.rdbms	opening connection	{"driver": "postgres", "dsn": "p****s:B****k@crm-corteza.ccten64bdcp3.eu-west-1.rds.amazonaws.com/corteza?sslmode=disable"}
server_1  | 20:02:26.830	DEBUG	store.rdbms	setting connection parameters	{"MaxOpenConns": 256, "MaxLifetime": "10m0s", "MaxIdleConns": 32}
server_1  | 20:02:26.871	DEBUG	store.rdbms	connected to the database
server_1  | 20:02:26.871	INFO	running store update
server_1  | 20:02:26.871	INFO	store.upgrade	store upgrade running in debug mode (UPGRADE_DEBUG=true)
server_1  | 20:02:26.879	DEBUG	store.upgrade	skipping sys_actionlog table rename, old table does not exist
server_1  | 20:02:26.892	DEBUG	store.upgrade	skipping sys_reminder table rename, old table does not exist
server_1  | 20:02:26.899	DEBUG	store.upgrade	skipping sys_user table rename, old table does not exist
server_1  | 20:02:26.903	DEBUG	store.upgrade	skipping sys_role table rename, old table does not exist
server_1  | 20:02:26.911	DEBUG	store.upgrade	skipping sys_role_member table rename, old table does not exist
server_1  | 20:02:26.919	DEBUG	store.upgrade	skipping sys_credentials table rename, old table does not exist
server_1  | 20:02:26.927	DEBUG	store.upgrade	skipping sys_application table rename, old table does not exist
server_1  | 20:02:27.179	DEBUG	store.upgrade	skipping rbac rules merge, table sys_permission_rules already Deleted
server_1  | 20:02:27.187	DEBUG	store.upgrade	skipping rbac rules merge, table compose_permission_rules already Deleted
server_1  | 20:02:27.203	DEBUG	store.upgrade	skipping settings merge, table sys_settings already removed
server_1  | 20:02:27.211	DEBUG	store.upgrade	skipping settings merge, table compose_settings already removed
server_1  | 20:02:27.415	INFO	updating system role	{"handle": "super-admin", "ID": 0}
server_1  | 20:02:27.415	INFO	updating system role	{"handle": "authenticated", "ID": 0}
server_1  | 20:02:27.415	INFO	updating system role	{"handle": "anonymous", "ID": 0}
server_1  | 20:02:27.437	DEBUG	system entities set	{"users": [268373446688243714, 268373446688309250, 268373446705020930], "roles": [268373446671466498, 268373446671532034, 268373446671597570]}
server_1  | 20:02:27.471	INFO	provision.pre-202109-roles	migrating pre-2021.9 roles
server_1  | 20:02:27.475	INFO	provision.pre-202109-rbac-rules	migrating RBAC rules to new format	{"rules": 301}
server_1  | 20:02:27.705	INFO	provision.pre-202109-settings	cleaning up pre-2021.9 settings
server_1  | 20:02:27.735	INFO	provision.resource-translations	migrating resource locales
server_1  | 20:02:27.757	INFO	provision.report-identifiers	migrating report identifiers
server_1  | 20:02:27.763	INFO	provision.post-202203-rbac-resource-rules-fix	migrating RBAC resource rules to proper format	{"rules": 301}
#8

also, I made sure to delete all failed reports from the DB

when I was upgrading from 2021.9.1 to 2022.3.0 I had broken reports which result to failed in the upgrade process … more details see here

#9

@darh just to give you full context, I get corredor server back with UPGRADE_DEBUG=true

and I get the same original error … see the logs

#10

hey, @darh I enabled these two variables too … might give some insight

LOG_INCLUDE_CALLER=true
LOG_STACKTRACE_LEVEL=debug

_

server_1    | 12:01:22.463	INFO	provision.post-202203-rbac-resource-rules-fix	provision/migrations_202203_rbac_resource_fix.go:37	migrating RBAC resource rules to proper format	{"rules": 301}
server_1    | github.com/cortezaproject/corteza-server/pkg/provision.migratePost202203RbacRules.func1
server_1    | 	github.com/cortezaproject/corteza-server/pkg/provision/migrations_202203_rbac_resource_fix.go:37
server_1    | github.com/cortezaproject/corteza-server/store/rdbms.Store.Tx.func1
server_1    | 	github.com/cortezaproject/corteza-server/store/rdbms/rdbms.go:260
server_1    | github.com/cortezaproject/corteza-server/store/rdbms.tx
server_1    | 	github.com/cortezaproject/corteza-server/store/rdbms/rdbms.go:382
server_1    | github.com/cortezaproject/corteza-server/store/rdbms.Store.Tx
server_1    | 	github.com/cortezaproject/corteza-server/store/rdbms/rdbms.go:259
server_1    | github.com/cortezaproject/corteza-server/store.Tx
server_1    | 	github.com/cortezaproject/corteza-server/store/tx.go:6
server_1    | github.com/cortezaproject/corteza-server/pkg/provision.migratePost202203RbacRules
server_1    | 	github.com/cortezaproject/corteza-server/pkg/provision/migrations_202203_rbac_resource_fix.go:31
server_1    | github.com/cortezaproject/corteza-server/pkg/provision.Run.func8
server_1    | 	github.com/cortezaproject/corteza-server/pkg/provision/provision.go:37
server_1    | github.com/cortezaproject/corteza-server/pkg/provision.Run
server_1    | 	github.com/cortezaproject/corteza-server/pkg/provision/provision.go:54
server_1    | github.com/cortezaproject/corteza-server/app.(*CortezaApp).Provision
server_1    | 	github.com/cortezaproject/corteza-server/app/boot_levels.go:299
server_1    | github.com/cortezaproject/corteza-server/app.(*CortezaApp).InitServices
server_1    | 	github.com/cortezaproject/corteza-server/app/boot_levels.go:314
server_1    | github.com/cortezaproject/corteza-server/app.(*CortezaApp).Activate
server_1    | 	github.com/cortezaproject/corteza-server/app/boot_levels.go:547
server_1    | github.com/cortezaproject/corteza-server/app.(*CortezaApp).InitCLI.func2
server_1    | 	github.com/cortezaproject/corteza-server/app/cli.go:95
server_1    | github.com/cortezaproject/corteza-server/pkg/cli.ServeCommand.func1
server_1    | 	github.com/cortezaproject/corteza-server/pkg/cli/commands.go:67
server_1    | github.com/spf13/cobra.(*Command).execute
server_1    | 	github.com/spf13/cobra@v1.4.0/command.go:856
server_1    | github.com/spf13/cobra.(*Command).ExecuteC
server_1    | 	github.com/spf13/cobra@v1.4.0/command.go:974
server_1    | github.com/spf13/cobra.(*Command).Execute
server_1    | 	github.com/spf13/cobra@v1.4.0/command.go:902
server_1    | github.com/cortezaproject/corteza-server/app.(*CortezaApp).Execute
server_1    | 	github.com/cortezaproject/corteza-server/app/cli.go:151
server_1    | main.main
server_1    | 	./main.go:13
server_1    | runtime.main
server_1    | 	runtime/proc.go:255
server_1    | 12:01:26.175	DEBUG	server/middleware.go:44	crashed on http request	{"error": "runtime error: invalid memory address or nil pointer dereference", "stack": "goroutine 502 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/cortezaproject/corteza-server/pkg/api/server.panicRecovery({0x342b530, 0xc000256180}, {0x3412430, 0xc00046e460})\n\tgithub.com/cortezaproject/corteza-server/pkg/api/server/middleware.go:44 +0x2ef\npanic({0x29774e0, 0x45764a0})\n\truntime/panic.go:1038 +0x215\ngoogle.golang.org/grpc.(*ClientConn).GetState(...)\n\tgoogle.golang.org/grpc@v1.45.0/clientconn.go:527\ngithub.com/cortezaproject/corteza-server/pkg/corredor.Healthcheck({0x457c060, 0x0})\n\tgithub.com/cortezaproject/corteza-server/pkg/corredor/healthcheck.go:22 +0x35\ngithub.com/cortezaproject/corteza-server/pkg/healthcheck.(*checks).Run(0xc00046cba0, {0x342b5d8, 0xc0005d6210})\n\tgithub.com/cortezaproject/corteza-server/pkg/healthcheck/check.go:68 +0x135\ngithub.com/cortezaproject/corteza-server/pkg/api/server.waitingRoutes.func2({0x3412430, 0xc00046e460}, 0xc0000f8500)\n\tgithub.com/cortezaproject/corteza-server/pkg/api/server/handlers.go:47 +0x170\nnet/http.HandlerFunc.ServeHTTP(0x29589e0, {0x3412430, 0xc00046e460}, 0xc004ea7e84)\n\tnet/http/server.go:2047 +0x2f\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc00025baa0, {0x3412430, 0xc00046e460}, 0xc0000f8500)\n\tgithub.com/go-chi/chi/v5@v5.0.7/mux.go:442 +0x216\nnet/http.HandlerFunc.ServeHTTP(0xc0003dcfa0, {0x3412430, 0xc00046e460}, 0xc0000f8500)\n\tnet/http/server.go:2047 +0x2f\ngithub.com/go-chi/cors.(*Cors).Handler.func1({0x3412430, 0xc00046e460}, 0xc0000f8500)\n\tgithub.com/go-chi/cors@v1.2.0/cors.go:228 +0x1bd\nnet/http.HandlerFunc.ServeHTTP(0x2b584e0, {0x3412430, 0xc00046e460}, 0x4575f40)\n\tnet/http/server.go:2047 +0x2f\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc00025baa0, {0x3412430, 0xc00046e460}, 0xc0000f8400)\n\tgithub.com/go-chi/chi/v5@v5.0.7/mux.go:88 +0x442\ngithub.com/cortezaproject/corteza-server/pkg/api/server.(*demux).ServeHTTP(0x0, {0x3412430, 0xc00046e460}, 0xc0000f8400)\n\tgithub.com/cortezaproject/corteza-server/pkg/api/server/demux.go:51 +0x11c\nnet/http.serverHandler.ServeHTTP({0xc0005d6120}, {0x3412430, 0xc00046e460}, 0xc0000f8400)\n\tnet/http/server.go:2879 +0x43b\nnet/http.(*conn).serve(0xc00046a3c0, {0x342b5d8, 0xc0002c9800})\n\tnet/http/server.go:1930 +0xb08\ncreated by net/http.(*Server).Serve\n\tnet/http/server.go:3034 +0x4e8\n"}
server_1    | github.com/cortezaproject/corteza-server/pkg/api/server.panicRecovery
server_1    | 	github.com/cortezaproject/corteza-server/pkg/api/server/middleware.go:44
server_1    | runtime.gopanic
server_1    | 	runtime/panic.go:1038
server_1    | runtime.panicmem
server_1    | 	runtime/panic.go:221
server_1    | runtime.sigpanic
server_1    | 	runtime/signal_unix.go:735
server_1    | google.golang.org/grpc.(*ClientConn).GetState
server_1    | 	google.golang.org/grpc@v1.45.0/clientconn.go:527
server_1    | github.com/cortezaproject/corteza-server/pkg/corredor.Healthcheck
server_1    | 	github.com/cortezaproject/corteza-server/pkg/corredor/healthcheck.go:22
server_1    | github.com/cortezaproject/corteza-server/pkg/healthcheck.(*checks).Run
server_1    | 	github.com/cortezaproject/corteza-server/pkg/healthcheck/check.go:68
server_1    | github.com/cortezaproject/corteza-server/pkg/api/server.waitingRoutes.func2
server_1    | 	github.com/cortezaproject/corteza-server/pkg/api/server/handlers.go:47
server_1    | net/http.HandlerFunc.ServeHTTP
server_1    | 	net/http/server.go:2047
server_1    | github.com/go-chi/chi/v5.(*Mux).routeHTTP
server_1    | 	github.com/go-chi/chi/v5@v5.0.7/mux.go:442
server_1    | net/http.HandlerFunc.ServeHTTP
server_1    | 	net/http/server.go:2047
server_1    | github.com/go-chi/cors.(*Cors).Handler.func1
server_1    | 	github.com/go-chi/cors@v1.2.0/cors.go:228
server_1    | net/http.HandlerFunc.ServeHTTP
server_1    | 	net/http/server.go:2047
server_1    | github.com/go-chi/chi/v5.(*Mux).ServeHTTP
server_1    | 	github.com/go-chi/chi/v5@v5.0.7/mux.go:88
server_1    | github.com/cortezaproject/corteza-server/pkg/api/server.(*demux).ServeHTTP
server_1    | 	github.com/cortezaproject/corteza-server/pkg/api/server/demux.go:51
server_1    | net/http.serverHandler.ServeHTTP
server_1    | 	net/http/server.go:2879
server_1    | net/http.(*conn).serve
server_1    | 	net/http/server.go:1930
server_1    | 2022/05/24 12:01:26 http: superfluous response.WriteHeader call from github.com/cortezaproject/corteza-server/pkg/api/server.panicRecovery (middleware.go:46)
server_1    | 12:01:26.201	DEBUG	server/middleware.go:44	crashed on http request	{"error": "runtime error: invalid memory address or nil pointer dereference", "stack": "goroutine 505 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/cortezaproject/corteza-server/pkg/api/server.panicRecovery({0x342b530, 0xc000256500}, {0x3412430, 0xc00046ee00})\n\tgithub.com/cortezaproject/corteza-server/pkg/api/server/middleware.go:44 +0x2ef\npanic({0x29774e0, 0x45764a0})\n\truntime/panic.go:1038 +0x215\ngoogle.golang.org/grpc.(*ClientConn).GetState(...)\n\tgoogle.golang.org/grpc@v1.45.0/clientconn.go:527\ngithub.com/cortezaproject/corteza-server/pkg/corredor.Healthcheck({0x457c060, 0x0})\n\tgithub.com/cortezaproject/corteza-server/pkg/corredor/healthcheck.go:22 +0x35\ngithub.com/cortezaproject/corteza-server/pkg/healthcheck.(*checks).Run(0xc00046cba0, {0x342b5d8, 0xc0005d6450})\n\tgithub.com/cortezaproject/corteza-server/pkg/healthcheck/check.go:68 +0x135\ngithub.com/cortezaproject/corteza-server/pkg/api/server.waitingRoutes.func2({0x3412430, 0xc00046ee00}, 0xc000192000)\n\tgithub.com/cortezaproject/corteza-server/pkg/api/server/handlers.go:47 +0x170\nnet/http.HandlerFunc.ServeHTTP(0x29589e0, {0x3412430, 0xc00046ee00}, 0xc005007874)\n\tnet/http/server.go:2047 +0x2f\ngithub.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc00025baa0, {0x3412430, 0xc00046ee00}, 0xc000192000)\n\tgithub.com/go-chi/chi/v5@v5.0.7/mux.go:442 +0x216\nnet/http.HandlerFunc.ServeHTTP(0xc0003dcfa0, {0x3412430, 0xc00046ee00}, 0xc000192000)\n\tnet/http/server.go:2047 +0x2f\ngithub.com/go-chi/cors.(*Cors).Handler.func1({0x3412430, 0xc00046ee00}, 0xc000192000)\n\tgithub.com/go-chi/cors@v1.2.0/cors.go:228 +0x1bd\nnet/http.HandlerFunc.ServeHTTP(0x2b584e0, {0x3412430, 0xc00046ee00}, 0x4575f40)\n\tnet/http/server.go:2047 +0x2f\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc00025baa0, {0x3412430, 0xc00046ee00}, 0xc0000f8e00)\n\tgithub.com/go-chi/chi/v5@v5.0.7/mux.go:88 +0x442\ngithub.com/cortezaproject/corteza-server/pkg/api/server.(*demux).ServeHTTP(0x0, {0x3412430, 0xc00046ee00}, 0xc0000f8e00)\n\tgithub.com/cortezaproject/corteza-server/pkg/api/server/demux.go:51 +0x11c\nnet/http.serverHandler.ServeHTTP({0xc0005d6390}, {0x3412430, 0xc00046ee00}, 0xc0000f8e00)\n\tnet/http/server.go:2879 +0x43b\nnet/http.(*conn).serve(0xc00046a820, {0x342b5d8, 0xc0002c9800})\n\tnet/http/server.go:1930 +0xb08\ncreated by net/http.(*Server).Serve\n\tnet/http/server.go:3034 +0x4e8\n"}
#11

Hi, I got all the info I needed from the first post but thank you :+1:

Please be sure to set LOG_STACKTRACE_LEVEL back to error (or comment it out). Logs will fill your disk quickly if you dont.

Crashing corredor health check is fixed on 2022.3.x branch and you can expect it in the release 2022.3.3. Can not say when exactlyč

#12

that’s great news thank you @darh

I disabled Corredor completely and the upgrade process still failing !!
I’m not sure if you aware of that

lastly, couldn’t you guys update the docker image of 2022.3.2 since it’s a blocker to try 2022.3.2 ?

#13

lastly, couldn’t you guys update the docker image of 2022.3.2 since it’s a blocker to try 2022.3.2 ?

Thank you for your input munawir. We’ll stick to our current release process for now.

Regarding your upgrade issues: you raised an issue with a crash in a certain subprocess (corredor health check). This was handled anyway by internal procedures and was only recorded as a DEBUG notice + with that fix, it should not even do that anymore… This does not and can not interfere with the upgrade procedure.

At least not from what I’m seeing here.

But… your logs are are always cur off at provision.post-202203-rbac-resource-rules-fix. Is this the last line and server halts at that point (in your words – they are not continuing)?

Can you export your rbac_rules table and attach it here?

#14

@darh yes, it stops at this

provision.post-202203-rbac-resource-rules-fix provision/migrations_202203_rbac_resource_fix.go:37 migrating RBAC resource rules to proper format {"rules": 301}

and here are my rbac_rules table as csv
here

#15

Thank you for the information, I am working on it.

1 Like
#16

Hi @vicky, any luck?

#17

Hey @munawir,
Yeah refactored script(migrations_202203_rbac_resource_fix) and made bit changes to prevent those nil pointer issues, currently adding more tests for it.

Expect this to be fixed in next release(~2 days).

1 Like
#18

great news @vicky … massive thanks

waiting for the next release and I’ll update from 2022.3.1 to 2022.3.3 directly

1 Like
#19

Hey @munawir,
If you wish to test the patch, you can either run it from source code, or snapshot and commit for above fix.

In the future, Please use the public forum so others can benefit from it as well.

1 Like
#21

hey guys @vicky @darh, thank you so much for your great support

I’ve just tried 2022.3.3-rc.1 version and the table migrated successfully … thanks a lot

2 Likes
#22