[go: up one dir, main page]

Azure Blob Storage backend fails to resume interrupted backup.

I have:

  • ([x] when completed)
  • searched https://gitlab.com/duplicity/duplicity/-/issues for similar issues. If you find a similar issue and the issue is still open, add a comment to the existing issue instead of opening a new one. If you find a Closed issue that seems like it is the same thing that you're experiencing, open a new issue and include a link to the original issue in the body of your new one.
  • searched https://bugs.launchpad.net/duplicity for similar issues. If you find a similar issue, open a new issue on here and include a link to the original issue in the body of your new one.
  • [N/A on FreeBSD] tested that this issue still occurs on the latest stable snap (install instructions: https://snapcraft.io/duplicity), please include the snap version (snap info duplicity | grep installed) output: installed: x.xx.xx (xx)
  • [N/A on FreeBSD] ideally, tested that this issue still occurs on the latest edge snap, if you can test without risking your data. Please include the snap version output: installed: x.xx.xx (xx)

Summary

(Summarize the bug encountered concisely.)

It appears that when an interrupted backup is resumed, duplicity may re-upload backend chunks that were already uploaded as part of the previous backup attempt. On older versions of duplicity, this was not a problem, because the Azure API call that duplicity was using allowed existing blobs to be overwritten by default. However, when the Azure backend was updated to use the latest Azure storage API, the new upload_blob API call does not allow a blob to be overwritten by default. This causes resumed backups to fail: 1a3441f3

There is an existing issue that was closed without a fix: #120 (closed)

Environment

(OS and Version)

FreeBSD 13.1-RELEASE

(duplicity --version)

0.8.23

(command line used)

duplicity -v7

Steps to reproduce

(How one can reproduce the issue - this is very important)

Start a backup using Azure blob storage, interrupt it, and then attempt to resume.

What is the current bug behaviour?

(What actually happens)

An interrupted backup resumes and attempts to re-upload an already uploaded chunk to Azure blob storage. It will fail with an error when doing so, aborting the backup.

What is the expected correct behaviour?

(What you should see instead)

The resumed backup should be able to re-upload chunks and continue the backup.

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)

Writing duplicity-full Giving up after 1 attempts. ResourceExistsError: The specified blob already exists.

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

I have manually patched my own duplicity install to use the overwrite=True parameter for the Azure upload_blob API. This restores the behavior that duplicity was relying on with the old create_blob_from_path API call. After this, I was successfully able to resume my interrupted backup and have been operating without issue since. The fix is simply to add the overwrite: True to the upload_blob kwargs here: https://gitlab.com/duplicity/duplicity/-/blob/main/duplicity/backends/azurebackend.py#L102

102c102
<         kwargs = {}
---
>         kwargs = { "overwrite": True }

I will submit a merge request with my change shortly.

Here are the docs I consulted while root causing this issue:

Old API (note it will overwrite an existing blob by default): https://learn.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.blockblobservice.blockblobservice?view=azure-python-previous#azure-storage-blob-blockblobservice-blockblobservice-create-blob-from-path

New API (see the overwrite parameter description): https://learn.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.blobclient?view=azure-python#azure-storage-blob-blobclient-upload-blob

Formatting

PLEASE DO NOT copy/paste long listings to the issue. Use the attach file option to attach a file instead. This makes it much easier to read and to process by the developers.

Edited by Charlie Stanley