How to modify throttling settings?

shmuel · December 1, 2022, 7:21am

I opened 10 chart pages in quick succession and got a Too Many Requests error.

I definitely need more relaxed settings. Is there a config option for that? I couldn’t find one in the example .env file.

Also, I plan on making thousands of asynchronous API requests in short bursts from time to time, so I want to make sure that no issues will arise during those “bulk processing” periods.

peter · December 2, 2022, 8:10am

Hi @shmuel ,

we did come by with this error once already and it is in our backlog, but not in the next week or so.
Do you happen to run this locally / via docker / or proxied behind nginx?

shmuel · December 2, 2022, 8:34am

Hey @peter ,

This was encountered both locally and remotely. Corteza was running in docker with no nginx proxy.

shmuel · December 2, 2022, 11:09am

I’ve filed some github issues related to throttling and many requests

github.com/cortezaproject/corteza

CortezaJS unhhandled rejection bug

opened 10:43AM - 02 Dec 22 UTC

skamensky

### Is there an existing issue for this? - [X] I have searched the existing i…ssues ### Version of Corteza 2022.9.x ### Current Behavior When processing many requests concurrently, I get the following error: ``` corteza_corredor_1 | (node:40) UnhandledPromiseRejectionWarning: Error: field moduleID is empty corteza_corredor_1 | at Compose.<anonymous> (/corredor/node_modules/@cortezaproject/corteza-js/src/api-clients/compose.ts:1304:13) corteza_corredor_1 | at Generator.next (<anonymous>) corteza_corredor_1 | at /corredor/node_modules/@cortezaproject/corteza-js/dist/index.js:277:71 corteza_corredor_1 | at new Promise (<anonymous>) corteza_corredor_1 | at __awaiter (/corredor/node_modules/@cortezaproject/corteza-js/dist/index.js:273:12) corteza_corredor_1 | at Compose.recordList (/corredor/node_modules/@cortezaproject/corteza-js/dist/index.js:15334:16) corteza_corredor_1 | at Object.exec (/extensions/workflow-helpers/server-scripts/import-merchant-merge.js:141:31) corteza_corredor_1 | at processTicksAndRejections (internal/process/task_queues.js:95:5) corteza_corredor_1 | (Use `node --trace-warnings ...` to show where the warning was created) corteza_corredor_1 | (node:40) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2) corteza_corredor_1 | (node:40) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code. ```` ### Expected Behavior The promise errors out so I can handle it gracefully ### Steps To Reproduce 1. Launch 30 thousand requests at the server asynchronously from corredor ### Environment and versions _No response_ ### Anything else? This is part of a series of bugs that happen when corteza is hit with many concurrent requests Other issues: https://github.com/cortezaproject/corteza/issues/591 https://github.com/cortezaproject/corteza/issues/593

github.com/cortezaproject/corteza

Postgres connection limit reached

opened 10:48AM - 02 Dec 22 UTC

skamensky

### Is there an existing issue for this? - [X] I have searched the existing i…ssues ### Version of Corteza 2022.9.x ### Current Behavior When doing batch processing inside corredor, I get hit with the following DB error: ```corteza_db_1 | 2022-12-02 10:37:49.093 UTC [527] FATAL: sorry, too many clients already``` Is there an issue with connection pooling? At the same time I recieved this error from within corredor: ``` corteza_corredor_1 | (node:40) UnhandledPromiseRejectionWarning: Error: Request failed with status code 401 corteza_corredor_1 | at createError (/extensions/workflow-helpers/node_modules/axios/lib/core/createError.js:16:15) corteza_corredor_1 | at settle (/extensions/workflow-helpers/node_modules/axios/lib/core/settle.js:17:12) corteza_corredor_1 | at IncomingMessage.handleStreamEnd (/extensions/workflow-helpers/node_modules/axios/lib/adapters/http.js:269:11) corteza_corredor_1 | at IncomingMessage.emit (events.js:412:35) corteza_corredor_1 | at IncomingMessage.emit (domain.js:475:12) corteza_corredor_1 | at endReadableNT (internal/streams/readable.js:1333:12) corteza_corredor_1 | at processTicksAndRejections (internal/process/task_queues.js:82:21) corteza_corredor_1 | (node:40) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 10) ``` ### Expected Behavior Coreza can handle multiple thousands of concurrent requests. If not, the error should be caught from within Corteza and not thrown from the DB (this way the client has a way of dealing with the error). ### Steps To Reproduce 1. Launch 30 thousand requests at the server asynchronously from corredor ### Environment and versions _No response_ ### Anything else? This is part of a series of bugs that happen when corteza is hit with many concurrent requests Other issues: https://github.com/cortezaproject/corteza/issues/590 https://github.com/cortezaproject/corteza/issues/593

github.com/cortezaproject/corteza

Corteza returns 401 when dealing with many requests

opened 11:03AM - 02 Dec 22 UTC

skamensky

### Is there an existing issue for this? - [X] I have searched the existing i…ssues ### Version of Corteza 2022.9.x ### Current Behavior When making many requests you get a 401 error instead of throttling errors. Here you can see the full axios log of the failed request. ``` Request failed with status code 401 corteza_corredor_1 | at createError (/extensions/workflow-helpers/node_modules/axios/lib/core/createError.js:16:15) corteza_corredor_1 | at settle (/extensions/workflow-helpers/node_modules/axios/lib/core/settle.js:17:12) corteza_corredor_1 | at IncomingMessage.handleStreamEnd (/extensions/workflow-helpers/node_modules/axios/lib/adapters/http.js:269:11) corteza_corredor_1 | at IncomingMessage.emit (events.js:412:35) corteza_corredor_1 | at IncomingMessage.emit (domain.js:475:12) corteza_corredor_1 | at endReadableNT (internal/streams/readable.js:1333:12) corteza_corredor_1 | at processTicksAndRejections (internal/process/task_queues.js:82:21) { corteza_corredor_1 | config: { corteza_corredor_1 | url: '/namespace/', corteza_corredor_1 | method: 'get', corteza_corredor_1 | headers: { corteza_corredor_1 | Accept: 'application/json, text/plain, */*', corteza_corredor_1 | 'Content-Type': 'application/json', corteza_corredor_1 | Authorization: 'Bearer REMOVED, corteza_corredor_1 | 'User-Agent': 'axios/0.21.4' corteza_corredor_1 | }, corteza_corredor_1 | params: { corteza_corredor_1 | query: undefined, corteza_corredor_1 | slug: undefined, corteza_corredor_1 | limit: undefined, corteza_corredor_1 | incTotal: undefined, corteza_corredor_1 | labels: undefined, corteza_corredor_1 | pageCursor: undefined, corteza_corredor_1 | sort: undefined corteza_corredor_1 | }, corteza_corredor_1 | baseURL: 'http://corteza_backend/api/compose', corteza_corredor_1 | transformRequest: [ [Function: transformRequest] ], corteza_corredor_1 | transformResponse: [ [Function: transformResponse] ], corteza_corredor_1 | timeout: 0, corteza_corredor_1 | withCredentials: true, corteza_corredor_1 | adapter: [Function: httpAdapter], corteza_corredor_1 | xsrfCookieName: 'XSRF-TOKEN', corteza_corredor_1 | xsrfHeaderName: 'X-XSRF-TOKEN', corteza_corredor_1 | maxContentLength: -1, corteza_corredor_1 | maxBodyLength: -1, corteza_corredor_1 | validateStatus: [Function: validateStatus], corteza_corredor_1 | transitional: { corteza_corredor_1 | silentJSONParsing: true, corteza_corredor_1 | forcedJSONParsing: true, corteza_corredor_1 | clarifyTimeoutError: false corteza_corredor_1 | }, corteza_corredor_1 | data: undefined corteza_corredor_1 | }, corteza_corredor_1 | request: <ref *1> ClientRequest { corteza_corredor_1 | _events: [Object: null prototype] { corteza_corredor_1 | abort: [Function (anonymous)], corteza_corredor_1 | aborted: [Function (anonymous)], corteza_corredor_1 | connect: [Function (anonymous)], corteza_corredor_1 | error: [Function (anonymous)], corteza_corredor_1 | socket: [Function (anonymous)], corteza_corredor_1 | timeout: [Function (anonymous)], corteza_corredor_1 | prefinish: [Function: requestOnPrefinish] corteza_corredor_1 | }, corteza_corredor_1 | _eventsCount: 7, corteza_corredor_1 | _maxListeners: undefined, corteza_corredor_1 | outputData: [], corteza_corredor_1 | outputSize: 0, corteza_corredor_1 | writable: true, corteza_corredor_1 | destroyed: false, corteza_corredor_1 | _last: true, corteza_corredor_1 | chunkedEncoding: false, corteza_corredor_1 | shouldKeepAlive: false, corteza_corredor_1 | _defaultKeepAlive: true, corteza_corredor_1 | useChunkedEncodingByDefault: false, corteza_corredor_1 | sendDate: false, corteza_corredor_1 | _removedConnection: false, corteza_corredor_1 | _removedContLen: false, corteza_corredor_1 | _removedTE: false, corteza_corredor_1 | _contentLength: 0, corteza_corredor_1 | _hasBody: true, corteza_corredor_1 | _trailer: '', corteza_corredor_1 | finished: true, corteza_corredor_1 | _headerSent: true, corteza_corredor_1 | socket: Socket { corteza_corredor_1 | connecting: false, corteza_corredor_1 | _hadError: false, corteza_corredor_1 | _parent: null, corteza_corredor_1 | _host: 'corteza_backend', corteza_corredor_1 | _readableState: [ReadableState], corteza_corredor_1 | _events: [Object: null prototype], corteza_corredor_1 | _eventsCount: 7, corteza_corredor_1 | _maxListeners: undefined, corteza_corredor_1 | _writableState: [WritableState], corteza_corredor_1 | allowHalfOpen: false, corteza_corredor_1 | _sockname: null, corteza_corredor_1 | _pendingData: null, corteza_corredor_1 | _pendingEncoding: '', corteza_corredor_1 | server: null, corteza_corredor_1 | _server: null, corteza_corredor_1 | parser: null, corteza_corredor_1 | _httpMessage: [Circular *1], corteza_corredor_1 | [Symbol(async_id_symbol)]: 37036, corteza_corredor_1 | [Symbol(kHandle)]: [TCP], corteza_corredor_1 | [Symbol(kSetNoDelay)]: false, corteza_corredor_1 | [Symbol(lastWriteQueueSize)]: 0, corteza_corredor_1 | [Symbol(timeout)]: null, corteza_corredor_1 | [Symbol(kBuffer)]: null, corteza_corredor_1 | [Symbol(kBufferCb)]: null, corteza_corredor_1 | [Symbol(kBufferGen)]: null, corteza_corredor_1 | [Symbol(kCapture)]: false, corteza_corredor_1 | [Symbol(kBytesRead)]: 0, corteza_corredor_1 | [Symbol(kBytesWritten)]: 0, corteza_corredor_1 | [Symbol(RequestTimeout)]: undefined corteza_corredor_1 | }, corteza_corredor_1 | _header: 'GET /api/compose/namespace/ HTTP/1.1\r\n' + corteza_corredor_1 | 'Accept: application/json, text/plain, */*\r\n' + corteza_corredor_1 | 'Content-Type: application/json\r\n' + corteza_corredor_1 | 'Authorization: REMOVED\r\n' + corteza_corredor_1 | 'User-Agent: axios/0.21.4\r\n' + corteza_corredor_1 | 'Host: corteza_backend\r\n' + corteza_corredor_1 | 'Connection: close\r\n' + corteza_corredor_1 | '\r\n', corteza_corredor_1 | _keepAliveTimeout: 0, corteza_corredor_1 | _onPendingData: [Function: noopPendingOutput], corteza_corredor_1 | agent: Agent { corteza_corredor_1 | _events: [Object: null prototype], corteza_corredor_1 | _eventsCount: 2, corteza_corredor_1 | _maxListeners: undefined, corteza_corredor_1 | defaultPort: 80, corteza_corredor_1 | protocol: 'http:', corteza_corredor_1 | options: [Object], corteza_corredor_1 | requests: {}, corteza_corredor_1 | sockets: [Object], corteza_corredor_1 | freeSockets: {}, corteza_corredor_1 | keepAliveMsecs: 1000, corteza_corredor_1 | keepAlive: false, corteza_corredor_1 | maxSockets: Infinity, corteza_corredor_1 | maxFreeSockets: 256, corteza_corredor_1 | scheduling: 'lifo', corteza_corredor_1 | maxTotalSockets: Infinity, corteza_corredor_1 | totalSocketCount: 911, corteza_corredor_1 | [Symbol(kCapture)]: false corteza_corredor_1 | }, corteza_corredor_1 | socketPath: undefined, corteza_corredor_1 | method: 'GET', corteza_corredor_1 | maxHeaderSize: undefined, corteza_corredor_1 | insecureHTTPParser: undefined, corteza_corredor_1 | path: '/api/compose/namespace/', corteza_corredor_1 | _ended: true, corteza_corredor_1 | res: IncomingMessage { corteza_corredor_1 | _readableState: [ReadableState], corteza_corredor_1 | _events: [Object: null prototype], corteza_corredor_1 | _eventsCount: 3, corteza_corredor_1 | _maxListeners: undefined, corteza_corredor_1 | socket: [Socket], corteza_corredor_1 | httpVersionMajor: 1, corteza_corredor_1 | httpVersionMinor: 1, corteza_corredor_1 | httpVersion: '1.1', corteza_corredor_1 | complete: true, corteza_corredor_1 | headers: [Object], corteza_corredor_1 | rawHeaders: [Array], corteza_corredor_1 | trailers: {}, corteza_corredor_1 | rawTrailers: [], corteza_corredor_1 | aborted: false, corteza_corredor_1 | upgrade: false, corteza_corredor_1 | url: '', corteza_corredor_1 | method: null, corteza_corredor_1 | statusCode: 401, corteza_corredor_1 | statusMessage: 'Unauthorized', corteza_corredor_1 | client: [Socket], corteza_corredor_1 | _consuming: false, corteza_corredor_1 | _dumped: false, corteza_corredor_1 | req: [Circular *1], corteza_corredor_1 | responseUrl: 'http://corteza_backend/api/compose/namespace/', corteza_corredor_1 | redirects: [], corteza_corredor_1 | [Symbol(kCapture)]: false, corteza_corredor_1 | [Symbol(RequestTimeout)]: undefined corteza_corredor_1 | }, corteza_corredor_1 | aborted: false, corteza_corredor_1 | timeoutCb: null, corteza_corredor_1 | upgradeOrConnect: false, corteza_corredor_1 | parser: null, corteza_corredor_1 | maxHeadersCount: null, corteza_corredor_1 | reusedSocket: false, corteza_corredor_1 | host: 'corteza_backend', corteza_corredor_1 | protocol: 'http:', corteza_corredor_1 | _redirectable: Writable { corteza_corredor_1 | _writableState: [WritableState], corteza_corredor_1 | _events: [Object: null prototype], corteza_corredor_1 | _eventsCount: 2, corteza_corredor_1 | _maxListeners: undefined, corteza_corredor_1 | _options: [Object], corteza_corredor_1 | _ended: true, corteza_corredor_1 | _ending: true, corteza_corredor_1 | _redirectCount: 0, corteza_corredor_1 | _redirects: [], corteza_corredor_1 | _requestBodyLength: 0, corteza_corredor_1 | _requestBodyBuffers: [], corteza_corredor_1 | _onNativeResponse: [Function (anonymous)], corteza_corredor_1 | _currentRequest: [Circular *1], corteza_corredor_1 | _currentUrl: 'http://corteza_backend/api/compose/namespace/', corteza_corredor_1 | [Symbol(kCapture)]: false corteza_corredor_1 | }, corteza_corredor_1 | [Symbol(kCapture)]: false, corteza_corredor_1 | [Symbol(kNeedDrain)]: false, corteza_corredor_1 | [Symbol(corked)]: 0, corteza_corredor_1 | [Symbol(kOutHeaders)]: [Object: null prototype] { corteza_corredor_1 | accept: [Array], corteza_corredor_1 | 'content-type': [Array], corteza_corredor_1 | authorization: [Array], corteza_corredor_1 | 'user-agent': [Array], corteza_corredor_1 | host: [Array] corteza_corredor_1 | } corteza_corredor_1 | }, corteza_corredor_1 | response: { corteza_corredor_1 | status: 401, corteza_corredor_1 | statusText: 'Unauthorized', corteza_corredor_1 | headers: { corteza_corredor_1 | 'content-type': 'application/json', corteza_corredor_1 | vary: 'Origin, Origin', corteza_corredor_1 | 'x-request-id': '2a93e2df528e/S59Qh5WpUC-002735', corteza_corredor_1 | date: 'Fri, 02 Dec 2022 10:52:33 GMT', corteza_corredor_1 | 'content-length': '525', corteza_corredor_1 | connection: 'close' corteza_corredor_1 | }, corteza_corredor_1 | config: { corteza_corredor_1 | url: '/namespace/', corteza_corredor_1 | method: 'get', corteza_corredor_1 | headers: [Object], corteza_corredor_1 | params: [Object], corteza_corredor_1 | baseURL: 'http://corteza_backend/api/compose', corteza_corredor_1 | transformRequest: [Array], corteza_corredor_1 | transformResponse: [Array], corteza_corredor_1 | timeout: 0, corteza_corredor_1 | withCredentials: true, corteza_corredor_1 | adapter: [Function: httpAdapter], corteza_corredor_1 | xsrfCookieName: 'XSRF-TOKEN', corteza_corredor_1 | xsrfHeaderName: 'X-XSRF-TOKEN', corteza_corredor_1 | maxContentLength: -1, corteza_corredor_1 | maxBodyLength: -1, corteza_corredor_1 | validateStatus: [Function: validateStatus], corteza_corredor_1 | transitional: [Object], corteza_corredor_1 | data: undefined corteza_corredor_1 | }, corteza_corredor_1 | request: <ref *1> ClientRequest { corteza_corredor_1 | _events: [Object: null prototype], corteza_corredor_1 | _eventsCount: 7, corteza_corredor_1 | _maxListeners: undefined, corteza_corredor_1 | outputData: [], corteza_corredor_1 | outputSize: 0, corteza_corredor_1 | writable: true, corteza_corredor_1 | destroyed: false, corteza_corredor_1 | _last: true, corteza_corredor_1 | chunkedEncoding: false, corteza_corredor_1 | shouldKeepAlive: false, corteza_corredor_1 | _defaultKeepAlive: true, corteza_corredor_1 | useChunkedEncodingByDefault: false, corteza_corredor_1 | sendDate: false, corteza_corredor_1 | _removedConnection: false, corteza_corredor_1 | _removedContLen: false, corteza_corredor_1 | _removedTE: false, corteza_corredor_1 | _contentLength: 0, corteza_corredor_1 | _hasBody: true, corteza_corredor_1 | _trailer: '', corteza_corredor_1 | finished: true, corteza_corredor_1 | _headerSent: true, corteza_corredor_1 | socket: [Socket], corteza_corredor_1 | _header: 'GET /api/compose/namespace/ HTTP/1.1\r\n' + corteza_corredor_1 | 'Accept: application/json, text/plain, */*\r\n' + corteza_corredor_1 | 'Content-Type: application/json\r\n' + corteza_corredor_1 | 'Authorization: Bearer REMOVEDr\n' + corteza_corredor_1 | 'User-Agent: axios/0.21.4\r\n' + corteza_corredor_1 | 'Host: corteza_backend\r\n' + corteza_corredor_1 | 'Connection: close\r\n' + corteza_corredor_1 | '\r\n', corteza_corredor_1 | _keepAliveTimeout: 0, corteza_corredor_1 | _onPendingData: [Function: noopPendingOutput], corteza_corredor_1 | agent: [Agent], corteza_corredor_1 | socketPath: undefined, corteza_corredor_1 | method: 'GET', corteza_corredor_1 | maxHeaderSize: undefined, corteza_corredor_1 | insecureHTTPParser: undefined, corteza_corredor_1 | path: '/api/compose/namespace/', corteza_corredor_1 | _ended: true, corteza_corredor_1 | res: [IncomingMessage], corteza_corredor_1 | aborted: false, corteza_corredor_1 | timeoutCb: null, corteza_corredor_1 | upgradeOrConnect: false, corteza_corredor_1 | parser: null, corteza_corredor_1 | maxHeadersCount: null, corteza_corredor_1 | reusedSocket: false, corteza_corredor_1 | host: 'corteza_backend', corteza_corredor_1 | protocol: 'http:', corteza_corredor_1 | _redirectable: [Writable], corteza_corredor_1 | [Symbol(kCapture)]: false, corteza_corredor_1 | [Symbol(kNeedDrain)]: false, corteza_corredor_1 | [Symbol(corked)]: 0, corteza_corredor_1 | [Symbol(kOutHeaders)]: [Object: null prototype] corteza_corredor_1 | }, corteza_corredor_1 | data: { error: [Object] } corteza_corredor_1 | }, corteza_corredor_1 | isAxiosError: true, corteza_corredor_1 | toJSON: [Function: toJSON] corteza_corredor_1 | } ``` ### Expected Behavior We should get a throttling headers so clients can automatically response in kind E.g. https://www.ietf.org/archive/id/draft-polli-ratelimit-headers-02.html ### Steps To Reproduce 1. Launch 30 thousand requests at the server asynchronously from corredor ### Environment and versions _No response_ ### Anything else? This is part of a series of bugs that happen when corteza is hit with many concurrent requests Other issues: https://github.com/cortezaproject/corteza/issues/591 https://github.com/cortezaproject/corteza/issues/590

peter · February 16, 2023, 8:10am

@shmuel , thanks for the feedback.

As you’ve probably already seen on the last issue and also for somebody else looking for info here, the way to fix the connection limits to Corteza API is using the:

These settings are currently the effective way to remove the limits (which should be somewhat thought about on your production systems).

Db polling limits are handled via db connection limit settings, either in postgres or mysql:

Returning proper headers (as per RFC) to the limited connections is currently a low prio issue.

shmuel · February 16, 2023, 10:10am

@peter ,

Thanks for the response.

I’d rather the Corteza server be able to pool connections (like many other web servers do), than increase my max DB connections to the number of parallel requests I expect to make.

It seems that Corteza does pool connections per session. For example, I can make thousands of workflow DB queries without an issue. I’m assuming this is done through connection pooling.

For those interested, I ended up moving all bulk automation out of corredor scripts (and my opinion is that the current corredor->server architecture is not fit for bulk automation) into a combination of raw SQL procedures and workflows.

shmuel · February 16, 2023, 10:11am

Which RFC are you referring to?