[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: database is locked #1201

Closed
1 task done
auipga opened this issue Sep 5, 2023 · 17 comments
Closed
1 task done

[Bug]: database is locked #1201

auipga opened this issue Sep 5, 2023 · 17 comments
Labels
S-Accepted This will be worked on S-Investigation This issue needs further investigation or design to figure out a solution T-Bug Something isn't working

Comments

@auipga
Copy link
auipga commented Sep 5, 2023

What happened?

Regularly reading this error while changing nothing and just rerunning the deployment:

error returned from database: (code: 5) database is locked

Version

v0.24.0

Which operating system(s) are you seeing the problem on?

In deployment

Which CPU architectures are you seeing the problem on?

In deployment

Relevant log output

cargo shuttle logs *****

 INFO Entering queued state
TRACE shuttle_deployer::deployment::queue: getting a build slot
TRACE {uri="http://gateway:8001/stats/load"} shuttle_deployer::deployment::gateway_client: calling gateway
DEBUG hyper::client::connect::http: connecting to 10.99.0.14:8001
DEBUG hyper::client::connect::http: connected to 10.99.0.14:8001
DEBUG hyper::client::pool: pooling idle connection for ("http", gateway:8001)
TRACE {response="Response { status: 200, version: HTTP/1.1, headers: {\"content-type\": \"application/json\", \"content-length\": \"38\", \"date\": \"Tue, 05 Sep 2023 21:59:58 GMT\"}, body: Body(Streaming) }"} shuttle_deployer::deployment::gateway_client: Load response

 INFO Entering building state
 INFO     Finished release [optimized] target(s) in 0.17s

 INFO Entering crashed state
ERROR {error="Build error: Database error: error returned from database: (code: 5) database is locked"} shuttle_deployer::deployment::queue: service build encountered an error

Duplicate declaration

  • I have searched the issues and there are none like this.
@auipga auipga added S-Triage Awaiting decision for what to do T-Bug Something isn't working labels Sep 5, 2023
@auipga
Copy link
Author
auipga commented Sep 5, 2023

Same with v0.25.0

@oddgrd
Copy link
Contributor
oddgrd commented Sep 6, 2023

Hey @auipga, we have a ticket for this here: #1067, but perhaps it's not so easy to find without "database is locked" in the title. Did you run just a single deployment, or did you attempt several? We're working on resolving the issue, but in the meantime you should be able to avoid the bug by letting a deployment finish before starting a new one (which will be the case if you don't exit the deployment stream). If you do end up with multiple deployments in the building/loading states, you should be able to reset it with a cargo shuttle project restart. Sorry for the trouble!

@auipga
Copy link
Author
auipga commented Sep 7, 2023

cargo deploy I ran a single deployment until it crashed.
cargo logs ... I read the logs.
update the code
cargo deploy rerun the deployment.

Although waiting a minute or more between the deployments reduced the likelihood of the mentioned error it still occurred sometimes. (It drove me a little crazy because I changed things back and forth but getting inconsistent results)

I read #1067 back when I created this issue but I thought that a crashed deployment is not running anymore hence no concurrency.

I'll try your suggestion to cargo shuttle project restart and let you know if it helps.

@oddgrd
Copy link
Contributor
oddgrd commented Sep 8, 2023

Ah, okay, thanks for the info! It does seem like this error can occur when having an active deployment and one building too, which is unexpected. We're about to make a change in the coming week that will stop writing logs to the deployer sqlite database, this should alleviate this issue since logs are the majority of writes to the DB. In the slightly longer term we aim to completely remove this DB from deployers (the container your service runs in), and of course support interuptible builds as stated in #1067.

@oddgrd oddgrd mentioned this issue Sep 14, 2023
1 task
@jonaro00
Copy link
Member

Although this bug is more rare now, it can still happen.

@elbaro
Copy link
elbaro commented Dec 18, 2023

Installed cargo-shuttle for the first time.
cargo shuttle init --template axum hello-world followed by cargo shuttle login, cargo shuttle deploy crashed with this error.

Also project restart should accept --allow-dirty.

❯ cargo shuttle project restart
Project "helo" is destroyed
Run `cargo shuttle project start` to recreate project environment on Shuttle.
Project "helo" is ready
Your project will sleep if it is idle for 30 minutes.
To change the idle time refer to the docs: https://docs.shuttle.rs/getting-started/idle-projects

Run `cargo shuttle deploy --allow-dirty` to deploy your Shuttle service.

❯ cargo shuttle project restart --allow-dirty
error: unexpected argument '--allow-dirty' found

Usage: cargo-shuttle project restart [OPTIONS]

@auipga
Copy link
Author
auipga commented Dec 20, 2023

@elbaro

  1. hello-world != helo
  2. Output: Run cargo shuttle project start
    You ran: cargo shuttle project **re**start + --allow-dirty, why?
  3. --allow-dirty is for deployment.
    $ cargo shuttle deploy -h
    ...
    Options:
          --allow-dirty
              Allow deployment with uncommitted files [aliases: ad]
    ...
  4. cargo shuttle project restart has no --allow-dirty option. It would make no sense to have this option for restart. Maybe I did't get the point, so please explain in more detail.
  5. Your issue seems not to be linked to this specific issue, so please open a new issue for your situation.

@elbaro
Copy link
elbaro commented Dec 20, 2023

Well the first project start gave [Bug]: database is locked and the second project start gave it's already running error. I ran restart to clear any corrupted state. The dashboard page doesn't indicate it's running and I couldn't access the deployed page even after a few restarts. I am providing a data point where the cloud state might be corrupted from this bug.

--allow-dirty is a different issue as you said and probably I mixed the concept of deploy and start at the moment.

@jonaro00
Copy link
Member

This bug has become a lot more common now and we have raised the priority to solve it.

@lazyfuhrer
Copy link
2024-01-03T03:55:09.866+05:30 [Deployer]  INFO Starting service
2024-01-03T03:55:14.870+05:30 [Deployer]  INFO Entering Crashed state
2024-01-03T03:55:14.870+05:30 [Deployer]  INFO Cleaning up crashed deployment
2024-01-03T03:55:14.870+05:30 [Deployer] ERROR shuttle_deployer::deployment::run: {error="Run error: errored while setting the new deployer address: Database error: error returned from database: (code: 5) database is locked"} Service encountered an error and crashed

Any idea how can I fix this ?

@jonaro00
Copy link
Member
jonaro00 commented Jan 3, 2024

The solution is usually to restart the project and deploy again.

@jonaro00
Copy link
Member

Update: This bug has been hard to reproduce, but it looks to be less frequent now than in previous weeks. We are also working on a bigger change that should eliminate this bug once and for all (getting rid of the sqlite dbs in deployers).

@jonaro00 jonaro00 added S-Investigation This issue needs further investigation or design to figure out a solution S-Accepted This will be worked on and removed S-Triage Awaiting decision for what to do labels Jan 31, 2024
@gnomeria
Copy link
gnomeria commented Feb 3, 2024

I just got realized I got this issue. Even after a cargo shuttle project restart it kept erroring out. :(
Now after the second project restart, the status has not changed from This project is booting up, please refresh the page in a few seconds for more than 15 minutes. Did the 3rd restart, still back to original error:

2024-02-03T00:11:50.929698Z [Deployer]  INFO shuttle_deployer::deployment::run: successfully loaded service
2024-02-03T00:11:50.929734Z [Deployer]  INFO shuttle_deployer::persistence: Uploading resources to resource-recorder
2024-02-03T00:11:50.930351Z [Deployer]  INFO Entering Running state
2024-02-03T00:11:50.930353Z [Deployer]  INFO Starting service
2024-02-03T00:11:55.934314Z [Deployer]  INFO Entering Crashed state
2024-02-03T00:11:55.934325Z [Deployer]  INFO Cleaning up crashed deployment
2024-02-03T00:11:55.934364Z [Deployer] ERROR shuttle_deployer::deployment::run: {error="Run error: errored while setting the new deployer address: Database error: error returned from database: (code: 5) database is locked"} Service encountered an error and crashed

I tried to update the cargo-runtime and cargo-axum to 0.38.0 and did another redeploy, it worked.
But then after updating the code and redeployed, got the original issue again, with several attempts at cargo shuttle project restart to no avail 😞

@jonaro00 jonaro00 pinned this issue Feb 8, 2024
@jonaro00
Copy link
Member

We have continued investigating this, and we made some improvements in 0.39.0 that we hope help reduce the frequency of this bug.

@jonaro00
Copy link
Member

The probability of getting this error seems to be reduced since 0.39.0

@jonaro00
Copy link
Member
jonaro00 commented Mar 7, 2024

Improvements in recent releases has made this error very rare now. We are continuing work on getting rid of it completely.

@jonaro00
Copy link
Member

Closing due to being solved on the shuttle.dev platform.

@jonaro00 jonaro00 unpinned this issue Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-Accepted This will be worked on S-Investigation This issue needs further investigation or design to figure out a solution T-Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

6 participants