Backend: Caching includes to improve performance when using remote includes

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

Some users have been receiving a timeout error when using remote includes. In an effort to help the CI linter perform better, exploring and implementing the ability for includes to be cached when using remote includes.

Proposal

Need to validate through a smaller set of tests whether incorporating caching into includes will help with the performance of when external includes are being referenced in gitlab-ci.yml when running a pipeline or if it would adversely affect the use of Redis.

Implementation

Work Type	Description	Issue link
	NOTE: 🚨 All below issues can be done in parallel
backend	Backend: The gitlab-ci.yml is limited to 100 includes	#207270 (closed)
backend	Backend: Remove N+1 for Gitaly requests when fetching includes	#344829 (closed)
backend frontend	Improve the error messaging when fetching remote includes are timing out	#351168 (closed)
backend	Backend: Improve CI Linter performance through parallelizing HTTP calls	#351250 (closed)
backend	Backend: Caching includes to improve performance when using remote includes	👈 You are here
backend	Backend: Batch request calls to Gitaly when fetching include	#382531 (closed)
backend	Backend: Group files by projects in config_file_project_validate_access	#382751 (closed)

Cache the response from the Gitlab::HTTP request in Gitlab::Ci::Config::External::File::Remote using a cache key like ci-remote-include::LOCATION (so, ci-remote-include::https://gitlab.com/another-ci-file.yml) and set the cache expiry to 1 hour