Discovery installation

nabl · October 6, 2022, 9:16pm

Hi There ,

I tried to install es and discovery and followed this:

I adjusted the docker-compose file:

  es:
    image: opensearchproject/opensearch:1.3.0
    restart: on-failure
    networks: [ internal ]
    environment:
      - cluster.name=es-docker-cluster
      - node.name=es
      - cluster.initial_master_nodes=es
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - OPENSEARCH_JAVA_OPTS=-Xms8000m -Xmx8000m # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - DISABLE_INSTALL_DEMO_CONFIG=true
      - DISABLE_SECURITY_PLUGIN=true
      - VIRTUAL_HOST=es.${DOMAIN}
      - LETSENCRYPT_HOST=es.${DOMAIN}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./data/es:/usr/share/elasticsearch/data

  discovery:
    image: cortezaproject/corteza-server-discovery:${VERSION}
    restart: on-failure
    env_file: [ .env ]
    depends_on: [ es, server ]
    networks: [ proxy, internal ]
    environment:
      VIRTUAL_HOST: discovery.${DOMAIN}
      VIRTUAL_PORT: 80
      LETSENCRYPT_HOST: discovery.${DOMAIN}

In .env i inserted the lines like in description but i have no starting point at what keys to insert or how to generate them:

DISCOVERY_INDEXER_PRIVATE_INDEX_CLIENT_KEY=${PRIVATE_KEY_EXAMPLE}
DISCOVERY_INDEXER_PRIVATE_INDEX_CLIENT_SECRET=${SECRET_EXAMPLE}
DISCOVERY_SEARCHER_CLIENT_KEY=${CLIENT_KEY_EXAMPLE}
DISCOVERY_SEARCHER_CLIENT_SECRET=${CLIENT_SECRET_EXAMPLE}
DISCOVERY_SEARCHER_JWT_SECRET=${JWT_SECRET_EXAMPLE}
DISCOVERY_SEARCHER_ALLOWED_ROLE=${ROLE_ALLOWED_EXAMPLE}

opensearch should be started with no security plugin enabled, so i dont quite get why these keys are required in the first place. but discovery refuses to start without them.
could you please give me a direction how to generate these keys?

Thanks in Advance and for the great System in general, please keep up the great work

vicky · October 8, 2022, 7:24am

Hey @nabl

opensearch should be started with no security plugin enabled, so i dont quite get why these keys are required in the first place. but discovery refuses to start without them.
could you please give me a direction how to generate these keys?

Corteza Discovery is a stand alone application, independent from the rest of the Corteza system. In order to make Discovery functional, you must grant access by creating an auth client along with a user and a role.

what keys to insert or how to generate them:

Access control defines what data the Discovery indexer has access to; more in the index configuration sections.

Take a look at our Low-Code Platform Developer Guide > Corteza Discovery for more info

nabl · October 8, 2022, 1:37pm

Hi @vicky

Thank you for the links i dont know how i have overread this.
I sat it up now and discovery service is up and running.
If i make the discovery url it availible from the outside world: is there a security problem with it? because now i see a response like this without any authentication:

{"response":{"hits":null,"total_hits":0,"aggregations":[],"size":999,"from":0,"total_results":0}}

so my guess is that the service is waiting for querys to be run without auth, is that correct?
My second question is about how to add Discovery Application to access it.
There is no Discovery Button on the App chooser in Corteza.
But maybe that is a general problem what i have. also the reporter never showed up by itself.
I added the App config for reporter by myself copying from a plain setup of corteza. Am i missing something here?

Thank you very much and best regards
Lukas

grisgruis · October 10, 2022, 10:22am

got this issue also, enabled discovery, but can’t seem to access it from the app

vicky · October 12, 2022, 12:24pm

If i make the discovery url it availible from the outside world: is there a security problem with it?

Discovery currently support private indexing policy but also have planning to support Public and Protected(Currently not in use; support will be added in a future release). hence, that response is for public index(users that are not logged-in). It is empty as there is not support as of now apart from that Private data will not be accessible or returned as response without proper authentication.

Check out Low-Code Platform Developer Guide > Corteza Discovery#Indexing Policy for more info

vicky · October 12, 2022, 12:27pm

CC @nabl,

From Corteza Admin webapp > From left menu > Applications, There you will be able to add application and also allow listing to your application list. You can take reference from the any of the existing applications. Doc for it, is being prepared it will be soon out. Cheers

grisgruis · October 13, 2022, 10:38am

Actually tried that before, by copying settings from latest.cortezaproject.org

left latest, right my own setup

And yes the discovery tile shows up, but when clicked, it redirects me back to the ‘home’ with all tiles/apps, no errors, no logs.

vicky · October 17, 2022, 10:18am

Hey @grisgruis,

Apologies for that, This might be due to Discovery is not default web application.
Please add below server configuration to your .env file to enable discovery web application.

HTTP_WEBAPP_LIST=admin,compose,workflow,reporter,discovery

Note: List out only all the webapps which are being used

Corteza doc reference for Server configuration :: Corteza Docs

grisgruis · October 17, 2022, 12:42pm

haha this is properly hidden… but thanks!

yajuve · November 1, 2022, 2:59pm

@nabl how did you arrived to start Discovery Server and linked it with Corteza Server, thanks

yajuve · November 1, 2022, 3:07pm

Hi @vicky, i followed all your comments concerning the configuration of Corteza Discovery, i didn’t undestand where can i find your-discovery-server-base-url in

DISCOVERY_BASE_URL=your-discovery-server-base-url

Following the documentation in: Corteza Discovery :: Corteza Docs

grisgruis · December 8, 2022, 9:26am

@vicky

Got a discovery service running, and it’s actually filled according to OpenSearch dashboard

Using version 2022.9.3

But now when I want to either update the ‘Discovery settings’ on a Module i get this error

And when Iam trying to search from the discovery webapp I get a 500 with this message:

There is no complete (local) docker-compose file anywhere to be found which just includes all
Server/Discovery/Corredor which you can just start en tryout, is it possible so supply such a file?

vicky · December 15, 2022, 11:29am

Thanks for reporting this, It will be fixed in next patch

vicky · December 15, 2022, 11:30am

It suppose to be the URL for discovery server host url.

MrMetal1986 · January 27, 2023, 2:11pm

Hello,

I’m also trying to get discovery to work according to the documentation (2022.09).

So far, it seems that opensearch is working as expected and scans corteza:

`[o.o.c.r.a.AllocationService] [es] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[corteza-private-compose-modules][0]]]).`

`[2023-01-27T13:41:15,335][INFO ][o.o.j.s.JobSweeper       ] [es] Running full sweep`

Also, when I check opensearch dashboards, data seem to be present.

As far as corteza discovery goes, it also seems also working, e.g.:

`13:58:50.490 DEBUG reindex/reindex.go:300 reindexing {"docs": 2, "index": "corteza-private-system-users"}`

`13:58:50.509 DEBUG reindex/reindex.go:300 reindexing {"docs": 2, "index": "corteza-private-compose-namespaces"}`

`13:58:50.517 DEBUG reindex/reindex.go:300 reindexing {"docs": 11, "index": "corteza-private-compose-modules"}`

`13:58:50.533 DEBUG reindex/reindex.go:300 reindexing {"docs": 32, "index": "corteza-private-compose-modules"}`

`13:58:50.541 DEBUG reindex/reindex.go:300 reindexing {"docs": 1, "index": "corteza-private-compose-records-321193550236745733-321193550236876805"}`

But, we accessing discovery web interface, the standard interface is returned with any results:

Logs:

`14:05:42.553 DEBUG service rest/es.go:413 searching  {"for": "mAggregation"}`

`14:05:42.562 DEBUG service rest/es.go:474 search completed {"for": "mAggregation", "query": "crm*", "indexPrefix": "corteza-private-", "from": 0, "size": 999, "status": "200 OK", "took": 7, "timedOut": false, "hits": 0, "hitsRelation": "eq", "namespaceAggs": 0, "moduleAggs": 0}`

When trying to configure discovery to compose module settings, I get the same error as already posted

Error
Could not save this module: error parsing http request body: json: cannot unmarshal array into Go struct field AccessJson.Config.discovery.public.result of type types.Result

My .env (stripped)

DOMAIN=corteza.local
VERSION=2022.9
ENVIRONMENT=dev
DB_DSN=postgres://corteza:corteza@db:5432/corteza?sslmode=disable
HTTP_WEBAPP_LIST=admin,compose,workflow,reporter,discovery
HTTP_WEBAPP_ENABLED=true
ACTIONLOG_DEBUG=false
LOG_LEVEL=ALL
LOG_DEBUG=true
APIGW_PROFILER_ENABLED=true
APIGW_LOG_REQUEST_BODY=true
APIGW_PROFILER_GLOBAL=true
LOG_INCLUDE_CALLER=true
LOG_STACKTRACE_LEVEL=debug
AUTH_JWT_SECRET=this-is-only-for-demo-purpose--make-sure-you-change-it-for-production
DISCOVERY_ENABLED=true
DISCOVERY_DEBUG=true
DISCOVERY_CORTEZA_DOMAIN=http://corteza.local/
DISCOVERY_BASE_URL=http://discovery.corteza.local
ES_ADDRESS=http://es:9200
ES_INDEX_INTERVAL=300
HTTP_ADDR=0.0.0.0:80
DISCOVERY_INDEXER_ENABLED=true
DISCOVERY_INDEXER_PRIVATE_INDEX_CLIENT_KEY=321193684118798341
DISCOVERY_INDEXER_PRIVATE_INDEX_CLIENT_SECRET=<<redacted>>
DISCOVERY_INDEXER_PROTECTED_INDEX_CLIENT_KEY=321193684118798341
DISCOVERY_INDEXER_PROTECTED_INDEX_CLIENT_SECRET=<<redacted>>
DISCOVERY_INDEXER_PUBLIC_INDEX_CLIENT_KEY=321193684118798341
DISCOVERY_INDEXER_PUBLIC_INDEX_CLIENT_SECRET=<<redacted>>
DISCOVERY_SEARCHER_ENABLED=true
DISCOVERY_SEARCHER_CLIENT_KEY=321447375639937027
DISCOVERY_SEARCHER_CLIENT_SECRET=<<redacted>>
DISCOVERY_SEARCHER_JWT_SECRET=this-is-only-for-demo-purpose--make-sure-you-change-it-for-production
#DISCOVERY_SEARCHER_ALLOWED_ROLE=

The auth clients are impersonating the super admin.

Am I missing something?

MrMetal1986 · January 28, 2023, 1:22pm

So I did a little digging. I’m most probably mistaken but there seems to be a problem in
corteza/server/discovery/types/resource_meta.go

	/*
		"discovery": {
		    "public": {
		      "result": [
		          {
		            lang: "", // en, fr
		            fields: [] // “names”, ..
		          }
		        ]
		      } // struct
		    },
		    "private": {
		      "response": []
		    }
		  }
	*/
	ModuleMeta struct {
		Public    AccessJson `json:"public"`
		Private   AccessJson `json:"private"`
		Protected AccessJson `json:"protected"`
	}

	AccessJson struct {
		Result Result `json:"result"`
	}

	Result struct {
		Lang   string   `json:"lang"`
		Fields []string `json:"fields"`

		// @todo? TBD? excludeModuleFields, includeModuleFields <- if passed filter module field accordingly.
	}

In particular, “fields” is a [] and noted a such in the Result struc Fields []string json:"fields".

The same for “result” however, in the AccessJson struct it is not the case. Shoudn’t it be Result []Result json:"result"?

If I send a post request according to the below snippet, no unmarshall error

		"discovery": {
		    "public": {
		      "result": 
		          {
		            lang: "", // en, fr
		            fields: [] // “names”, ..
		          }
		        
		      } // struct
		    },
		    "private": {
		      "response": []
		    }
		  }

jfortun · February 1, 2023, 2:00pm

Hey, this has been fixed and will be released with 2022.9.7