tahoe-lafs/trac-2024-07-25

Use CircleCI's "parallelism" feature to speed up runs of the integration test suite #4054

New issue

Open

opened 2023-08-01 14:02:05 +00:00 by exarkun · 0 comments

exarkun commented

2023-08-01 14:02:05 +00:00

Owner

CircleCI has a feature where you can tell it to run several executors concurrently for a single job. It tells each executor how many concurrent executors it is running and what its unique index in that sequence is. Then it is up to the job to pick a slice of work which, when combined with the choice made on all the other jobs, combine to represent the full test suite run.

For a test suite (as opposed to something like the compilation of a large C program), the "slice of work" can be some subset of the tests in the test suite.

There are a few possible approaches for convincing pytest to run such a subset. If we combine one of these with the CircleCI feature then we can leverage more CPU power to (possibly) complete the integration tests more quickly.

CircleCI has a feature where you can tell it to run several executors concurrently for a single job. It tells each executor how many concurrent executors it is running and what its unique index in that sequence is. Then it is up to the job to pick a slice of work which, when combined with the choice made on all the other jobs, combine to represent the full test suite run. For a test suite (as opposed to something like the compilation of a large C program), the "slice of work" can be some subset of the tests in the test suite. There are a few possible approaches for convincing pytest to run such a subset. If we combine one of these with the CircleCI feature then we can leverage more CPU power to (possibly) complete the integration tests more quickly.

tahoe-lafs added the

labels 2023-08-01 14:02:05 +00:00

tahoe-lafs added this to the undecided milestone 2023-08-01 14:02:05 +00:00

Sign in to join this conversation.

No labels

LeastAuthority.com automation

cannot reproduce

code-frontend-cli

code-frontend-ftp-sftp

code-frontend-magic-folder

code-frontend-web

code-peerselection

dev-infrastructure

somebody else's problem

was already fixed

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/trac-2024-07-25#4054

No description provided.