Repository navigation
Tags
Tags
feat: set up terraform for real postgres (#6280) Experiment db seems to be doing good, time to set up the real deal: - Extract the OSV API terraform stuff from the `osv` terraform module to a dedicated `osv_api` module (with `moved` blocks to retain the state across them), so that it's easier to avoid setting up postgres read replicas in `private-osv`, which does not run the API - Create the Postgres database in `osv_pipeline` module (and moved a few resources from the `temp` module which would cause problems if we tried to recreate them) - `oss-vdb` and `oss-vdb-test` are created in `us-central1` so we can have the API read-replicas in the same region as the Cloud Run, `private-osv` is in `us-west2`. - The primary machine (for writes) is probably a bit undersized at the moment (only 4 CPUs / 2 on private) - I'll look at how much we end up actually needing when the dual writes is up and running. - We have 2x16 CPU read pools for the prod API, based on performance from the experiment (which was happy with 1x32) - Document the postgres password generation steps in the README
chore(deps): bump filelock from 3.32.7 to 4.2.4 (#6236) For vulnerabilities...
refactor: stop OSS-Fuzz workers writing directly to the main database (… …#6172) Enabling some database migration by fully separating OSS-Fuzz concerns from main OSV database stuff. - OSS-Fuzz vulns now are imported into the main OSV database via the git repo like a regular datasource, rather than writing vulns directly - The `OSS-Fuzz` pipeline now only writes `Bug` entities (and commits them to the oss-fuzz-vulns repo) - Removed a bunch of now unused code: `Bug._post_put_hook` and `AffectedCommits` writing functions, PyPi notification, put_if_newer. - Withdrawn OSS-Fuzz records are now published to the oss-fuzz-vulns repo (if they had already been published to the repo before) so we don't accidentally drop those updates (though I'm not sure if that would ever happen) This needs to be merged carefully, so as to not muck up the database. What needs to happen is this: 1. Pause most/all cron jobs to pause updates 2. Merge this PR, which should create a new `SourceRepository` 3. Set the last synced commit of the new source repo to the latest oss-fuzz-vulns commit 4. Run a script (that I have locally) to update all the `OSV-` records `source` to the new `oss-fuzz-vulns` source 5. Resume all cron jobs 6. ??? 7. Profit I'm on release next week, so I can manage this stuff.
test: update apitester snapshots (#6145) The snapshots have changed, probably due to OSV advisories being changed. Please review the differences to make sure that they're expected!
feat(bindings/go): add ExperimentalQueryUbuntuPackageMapping client m… …ethod (#6136) > [!NOTE] > Stacked on top of #6135. Adds `ExperimentalQueryUbuntuPackageMapping` to `osvdev.OSVClient` in `bindings/go` for calling `POST /v1experimental/ubuntu/binary-to-source`. - Automatically chunks requests with more than `MaxPackagesPerUbuntuMappingRequest` (1000) binary names and executes them concurrently while preserving input order. - Adds unit tests for single-batch queries, automatic chunking (>1000 packages), and error propagation. <!-- AI-PR -->
test: update apitester snapshots (#6040) The snapshots have changed, probably due to OSV advisories being changed. Please review the differences to make sure that they're expected!
test: update apitester snapshots (#6016) The snapshots have changed, probably due to OSV advisories being changed. Please review the differences to make sure that they're expected!
chore(sources): update chainguard source to v3 feed URL (#5944) Point the Chainguard entry in `source.yaml` at the new Chainguard OSV v3 feed, as was previously already done in source_test.yaml. See: - #5606 Chainguard v3 feed docs are in: - https://github.com/chainguard-dev/vulnerability-scanner-support/blob/main/docs/osv_v3_feed.md This has had a few weeks of testing now in the test instance, and behaviour observed there is the desired behaviour for the CGA- entries. Signed-off-by: Dimitri John Ledkov <dimitri.ledkov@chainguard.dev> Signed-off-by: Dimitri John Ledkov <dimitri.ledkov@chainguard.dev>
feat: Optimize linting by checking changed files (#5939) Trying to do quick fixes really make you find out where the pain points in the CI. Turned the lint and format script into a python script (with no dependencies). We now tune a few settings to make it run a lot faster. 1. Make use of the parallel processing flags for python lints 3. Only run it on files that changed (In CI it's against the master branch).
PreviousNext