Repository navigation
[FEATURE] Local and Brickflow Metadata Cleanup #167
Description
A really common problem experienced by developers is when deployments fail because metadata in the local and/or databricks workspace does not match what resources are actually deployed to databricks.
The error message often looks like:
Error: cannot read job: User ff59715d-07ae-465b-9a29-5539c4e7f687 does not have Admin or Manage Run or Owner or View permissions on job 1020683971056165
Often times even under the instruction of support developers that try to resolve this issue manually end up making mistakes. This wastes the time of everyone involved.
Cloud Information
- AWS
- Azure
- GCP
- Other
I'm proposing extending the CLI to include a 'clean' feature that performs this cleanup per project.
The command would look something like:
bf projects clean --project test_project --env qa
Describe alternatives you've considered
I wrote a shell script that performed this same functionality and have tested it to success, but it would be better if it was integrated into the CLI.
Activity
maxim-mityutko commentedon May 9, 2025
I think this is a valuable feature / fix, cause it pops rather often in the support channel.
nogitting commentedon May 10, 2025
To be put on hold, at best, for now. Data shows it is rather due to (local or CI/CD) setup-related issues. Plus, the clean-up, in this very case, may seem simple but not necessarily safe.
maxim-mityutko commentedon May 14, 2025
@nogitting Can you elaborate what are the caveats for doing this for local deployments?