[rush] Improve task scheduler to handle tasks that consume 100% CPU

[octogonz]

Summary

@LPegasus and I recently discussed a problem where certain Rush phase scripts are heavily multithreaded, designed to consume 100% CPU. Examples:

• ESBuild
• Rspack
• Jest

When Rush attempts to run several of these tasks in parallel, it overloads the machine. For ESBuild it was causing Golang deadlocks, perhaps due to an ESBuild bug that causes thread starvation. (?)

Details

This can be avoided by specifying weight=N in rush-project.json, where N is equal to the --parallelism limit. But if CI uses a value like --parallelism=100%, then N is machine dependent.

We can break this problem into two complementary design proposals:

Proposal 1: Allow % units for weight

The rush-project.json file should allow percentage values like "weight": "100%" similar to --parallelism=100%. For example if --parallelism=100% corresponds to a maximum of 8 parallel jobs, then "weight": "75%" would indicate a weight of 6.

Proposal 2: Allow weight to be specified using pattern matching

In order to use the "weight": "100%" feature, we would need to audit every operation script for every project in our monorepo, identify which ones are invoking ESBuild, and then add a corresponding rush-project.json or rig profile to adjust the weight.

However in practice, that work could be captured by a general rule like this:

Examine the package.json shell script: Does it match the RegExp /\besbuild\b/? If yes, then use "100%" as the default weight, unless it is overridden by rush-project.json.

Such a rule could be configured centrally in a common/config file.

@dmichon-msft

[dmichon-msft]

Discussed this during Rush Hour today; recommendation is that weight and parallelism use the same units; percentage weights will be a fraction of os.availableParallelism(). Note that the value should get normalized into a number when being forwarded onto the Operation because the queue computes critical path lengths as a sum of weights.

If you want to dynamically control operation weights, write your own Rush plugin that taps the runPhasedCommand hook and it is free to tamper with the weights of all operations in the graph before they get scheduled. It could even do things like read historical data from somewhere and use that to guess an appropriate value, including analyzing based on RAM usage instead of CPU.

Edit:
For clarity on why weight should be a percentage of os.availableParallelism() directly and not of parallelism, consider a machine with 8 cores and 4 tasks that are each configured with a weight: 25%.

• User runs with parallelism: 100%, max concurrency is 8*100% = 8, effective weight = 8*100%*25% = 2. Number of actual concurrent tasks is 8/2=4.
• User runs with parallelism: 50%, max concurrency is 8*50% = 4; if weight depends on parallelism, effective weight = 8*50%*25% = 1. Number of actual concurrent tasks is 4/1=4. However, these tasks still do the same amount of work as above, so likely this will result in 100% CPU utilization
• User runs with parallelism: 50%, max concurrency is 8*50% = 4; if weight does not depend on parallelism, effective weight = 8*25% = 2. Number of actual concurrent tasks is 4/2=2, leaving 50% of CPUs free.

[LPegasus]

Hi, @dmichon-msft . I have a question about the code here https://github.com/microsoft/rushstack/blob/main/libraries/rush-lib/src/logic/operations/OperationExecutionManager.ts#L264.

For example: The current device has parallelism=10, and there are 4 tasks A(Weight=4), B(Weight=4), C(Weight=4), D(Weight=4). Since the total number of Operations is only 4, the concurrency of Async.forEachAsync will be set to 4 (Math.min(4, 10)). As a result, these 4 tasks will execute sequentially.

However, the expected behavior is that, assuming allowOversubscription is now false, there should be two Operations running in parallel simultaneously.