How to setup proper monitoring and logging

#1

Hello there

I’ve been using corteza for about 3 months
I don’t have monitoring for failed automation and when I debug something I ssh the server to see the logs

is there any built-in solution for such monitoring/logging Corteza and wouldn’t be possible to send these logs to slack channel or email ?

#2

Hi @munawir, we have support for Sentry ready; I asked our devs to give you some pointers on setting it up.

1 Like
#3

Sentry would be one way to collect serious issues (when go panics for example).

Enable logging of mem usage, goroutines etc

And endpoint for collecting metrics (prometheus)

You can also pipe (container) logs to wherever you desire and process them there (ELK?).

2 Likes
#4

thanks guys, will take a look at them

did I just relized that you released the new version! :slight_smile:

#5

2022.3 is still in RC and the entire team is working hard to find and squash those bugs.
Expect 2022.3.0 sometime next week!

But the ENV options references from my prev. post are valid for older versions too!

1 Like
#6

Hey @darh, I’ve tried the Sentry integration but I received nothing at Sentry
here is my .env file

SENTRY_DSN=
ACTIONLOG_DEBUG=true
LOG_LEVEL=debug
LOG_INCLUDE_CALLER=true
LOG_STACKTRACE_LEVEL=debug
LOG_DEBUG=true

when I logged into the server I found the logs at docker’s logs for example

17:12:59.403 DEBUG actionlog service/module.go:458 looked-up for a User {“timestamp”: “17:12:59.403”, “requestOrigin”: “api/rest”, “requestID”: “077c9980785b/0JQb7k8o5j-003292”, “actorIPAddr”: “93.168.34.253”, “actorID”: 268378038629040130, “resource”: “compose:module”, “action”: “lookup”, “severity”: 6, “error”: “”, “description”: “looked-up for a User”, “policy-match”: true, “meta”: {“module.ID”:“268373859994959874”,“module.handle”:“User”,“module.name”:“User”,“module.namespaceID”:“268373449590702082”,“namespace.ID”:“268373449590702082”,“namespace.name”:“CRM”,“namespace.slug”:“crm”}}
17:12:59.855 DEBUG actionlog service/record.go:341 searched for records {“timestamp”: “17:12:59.855”, “requestOrigin”: “api/rest”, “requestID”: “077c9980785b/0JQb7k8o5j-003293”, “actorIPAddr”: “93.168.34.253”, “actorID”: 268378038629040130, “resource”: “compose:record”, “action”: “search”, “severity”: 6, “error”: “”, “description”: “searched for records”, “policy-match”: true, “meta”: {“filter.deleted”:0,“filter.limit”:“500”,“filter.moduleID”:“268373859994959874”,“filter.namespaceID”:“268373449590702082”,“filter.query”:“user_id=229364 or user_id=196597 or user_id=262133 or user_id=262135 or user_id=229372 or user_id=196605 or user_id=163838”,“filter.sort”:""}}
17:12:59.892 DEBUG actionlog service/record.go:341 searched for records {“timestamp”: “17:12:59.892”, “requestOrigin”: “api/rest”, “requestID”: “077c9980785b/0JQb7k8o5j-003292”, “actorIPAddr”: “93.168.34.253”, “actorID”: 268378038629040130, “resource”: “compose:record”, “action”: “search”, “severity”: 6, “error”: “”, “description”: “searched for records”, “policy-match”: true, “meta”: {“filter.deleted”:0,“filter.limit”:“500”,“filter.moduleID”:“268373859994959874”,“filter.namespaceID”:“268373449590702082”,“filter.query”:“user_id=229364 or user_id=262133 or user_id=196597 or user_id=262135 or user_id=229372 or user_id=196605 or user_id=163838”,“filter.sort”:""}}

appreciate your help

#7

Looks like you left SENTRY_DSN var unset. Kinda hard for Corteza to know where your Sentry server is if you do not tell it…

#8

hey @darh
of course, I sat it up

SENTRY_DSN=https://75#####b9ea#####a80beb1#####a4f6@o4#####.ingest.sentry.io/63#####

#9

Hi @munawir

Can you explain what are you trying to achieve?

Sentry will tell you when and why something crashed
HTTP_METRICS_* options will enable HTTP endpoints for Prometheus to pickup
MONITOR_INTERVAL will output a couple of stats in the logs
LOG_LEVEL will limit what is logged and what not

These and the rest are explained in the docs and/or example

#10

Hi @darh, my goal is to have proper visibility on what’s going on in the platform

currently, I’m only interested in workflows,

  • what’s the error message if they fail?
  • what’s the input/output data of the workflow?
  • what are the scopes and their data in a workflow?

I tried to run a workflow with an error but Sentry didn’t capture the error so I guess Sentry is only interested in code exceptions! is that right?
regarding Prometheus, we are currently not interested in gathering metrics about the platform

I’ve read the docs, and I see several types of logs, how to capture the logs on all levels for workflows (or even any component). if I could capture them, I’ll somehow push them to our Slack channel

please let me know if you needed anything … thanks for bearing with me

1 Like
Tracking workflow information