AER-4020 Added configuration option to set max number of workers available#118
AER-4020 Added configuration option to set max number of workers available#118Hilbrand wants to merge 1 commit intoaerius:mainfrom
Conversation
…lable When scaling workers the maxCapacityUse value can limit the dispatching of tasks because when the capacity a specific queue can use is based on the actual workers it might never cause tasks not send to workers and therefor the load of the workers might never exceed the trigger value to start adding more workers. By setting a maxWorkerAvailable value to the value the system can scale to tasks will be scheduled if the full capacity is available. So when a low number of workers available more tasks would still be scheduled, which will result in triggering the scaling value.
SerhatG
left a comment
SerhatG
left a comment
There was a problem hiding this comment.
Ah, I didn't get this directly.. am I correct to assume this would be perfect for queues (chunkers come to mind) where the maxCapacityUse is lower than the scaling threshold, so by setting this you can get it to scale out, even though only this relatively small queue is 'full'.
Example scenario to see if I understand the change
connect_gml_export is set to 40% by default.
For a production environment where it starts with 16 chunkers in the morning, 40% would be 6.4 chunkers. But this will never trigger a scale out if this is the only type of chunkers running at that moment, as the threshold for starting a scale out is 50% for chunkers.
Instead if say, the maximum amount of chunkers is fictitiously 100, you could set maxWorkersAvailable to 40, which is what it is allowed to use for a fully scaled out environment.
| try { | ||
| final JsonNode jsonObject = getJsonResultFromApi(apiPath); | ||
|
|
||
| LOG.info("Making RabbitMQ api call: {}", apiPath); |
There was a problem hiding this comment.
Probably not meant to be .info, or was it to debug while developing?
There was a problem hiding this comment.
Ah seems I forgot to push my last update of commit that already had removed this log line (as it should not be here indeed).
Hilbrand
force-pushed
the
AER-4020-max-workers-available
branch
from
3a63d33 to
e37392d
Compare
|
@SerhatG Yes that is exactly the intended usage. |
BertScholten
left a comment
BertScholten
left a comment
There was a problem hiding this comment.
Not exactly sure how this is going to be used in practice. Don't think we have automatic configuration files per environment (yet)? Or do you want to configure it based on the (rather minimal) dev environment?
| * Returns the number of workers that are available or the number of configured maximum numbers that are potential available. Which ever number | ||
| * is the highest. This number can be used to see if tasks can still be scheduled. If there is still room for more workers to be started | ||
| * (i.e. max is not reached) than a task should be able to be scheduled it it doesn't have reach it maximum relative to the total potential | ||
| * workers, not the actual number of workers running. |
There was a problem hiding this comment.
| * Returns the number of workers that are available or the number of configured maximum numbers that are potential available. Which ever number | |
| * is the highest. This number can be used to see if tasks can still be scheduled. If there is still room for more workers to be started | |
| * (i.e. max is not reached) than a task should be able to be scheduled it it doesn't have reach it maximum relative to the total potential | |
| * workers, not the actual number of workers running. | |
| * Returns the number of workers that are available, or the configured maximum that are potentially available, | |
| * whichever number is the highest. | |
| * This number can be used to see if tasks can still be scheduled. | |
| * If there is still room for more workers to be started (i.e. the max is not reached yet) then a task can be scheduled. | |
| * This then serves as a preload mechanism while workers are starting/scaled up. |
Had a hard time understanding this comment. Code was simpler.
| For example if worker scaling is based on the percentage workers being used it can mean the system will never scale up because the `maxCapacityUse` for certain input queues will never exceed the scaling threshold percentage. | ||
| Especially when the system runs with a low number of workers the `maxCapacityUse` is easily reached before the scaling threshold is reached. | ||
| By setting `maxWorkersAvailable` the scheduler will determine if tasks already have reached the `maxCapacityUse` value based on the `maxWorkersAvailable` value and not the actual number of workers running. | ||
| It will than schedule tasks beyond the `maxCapacityUse` assuming this would trigger the system to scale up if needed, |
There was a problem hiding this comment.
| It will than schedule tasks beyond the `maxCapacityUse` assuming this would trigger the system to scale up if needed, | |
| It will then schedule tasks beyond the `maxCapacityUse`, assuming this will trigger the system to scale up as needed, |
| Especially when the system runs with a low number of workers the `maxCapacityUse` is easily reached before the scaling threshold is reached. | ||
| By setting `maxWorkersAvailable` the scheduler will determine if tasks already have reached the `maxCapacityUse` value based on the `maxWorkersAvailable` value and not the actual number of workers running. | ||
| It will than schedule tasks beyond the `maxCapacityUse` assuming this would trigger the system to scale up if needed, | ||
| and only for a short amount of time claim more resources for a specific queue than would be allowed by the `maxCapacityUse` percentage. |
There was a problem hiding this comment.
only for a short time if the maxWorkersAvailable matches the actual system max scaling. If that isn't configured correctly, it might end up claiming more for longer periods (bit nitpicky perhaps).
|
Something to discuss at the technical meeting? |
3 participants
When scaling workers the maxCapacityUse value can limit the dispatching of tasks because when the capacity a specific queue can use is based on the actual workers it might never cause tasks not send to workers and therefor the load of the workers might never exceed the trigger value to start adding more workers. By setting a maxWorkerSize value to the value the system can scale to tasks will be scheduled if the full capacity is available. So when a low number of workers available more tasks would still be scheduled, which will result in triggering the scaling value.