Backend: Remove N+1 for Gitaly requests when fetching `includes`
Before being able to increase the number of includes while properly monitoring performance, we should look into some possible N+1s.
Context:
We have Gitaly N+1 requests because each include makes one request to the the file's contents.
You can observe in the following dashboard that the ..Config::Process usually takes an average of 3s, but there are several spikes where it goes up to 12s.
@mbobin did some local debugging, and found that the GitLab project uses 31 includes and the Config::Process was taking about 5-6 seconds.
Implementation
| Work Type | Description | Issue link |
|---|---|---|
|
NOTE: |
||
| backend | Backend: The gitlab-ci.yml is limited to 100 includes | #207270 (closed) |
| backend | Backend: Remove N+1 for Gitaly requests when fetching includes |
|
| backend frontend | Improve the error messaging when fetching remote includes are timing out | #351168 (closed) |
| backend | Backend: Improve CI Linter performance through parallelizing HTTP calls | #351250 (closed) |
| backend | Backend: Caching includes to improve performance when using remote includes | #351252 |
| backend | Backend: Batch request calls to Gitaly when fetching include
|
#382531 (closed) |
| backend | Backend: Group files by projects in config_file_project_validate_access | #382751 (closed) |
Edited by Mark Nuzzo