Repository navigation
Unreadable files make the checksum step run twice per file (doubling) and get the SHA-1 of empty input as id #236
Description
When the file index contains files that HomeGallery cannot read (e.g. EACCES),
the checksum step of the import does not just skip them:
- Every unreadable file doubles the number of parallel checksum loops:
the warningCould not create checksum of ...is logged once for the first
unreadable file, twice for the second, 4 times for the third, and so on. With
a few dozen unreadable files, the import seems to hang and floods the log. - Unreadable files are stored in the index with the SHA-1 of empty input
(da39a3ee5e6b4b0d3255bfef95601890afd80709) as their id. - The parallel loops add
SIGINTlisteners concurrently:
MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process].
Cause
In packages/index/src/checksum.ts, createSha1() calls its callback from
both the error and the close listener of the read stream. When the file
cannot be opened, Node emits error and then close, so the callback runs
twice: once with the error (calculateAll() logs the warning and continues
with the next file), and once with the digest of no data (calculateAll()
stores it as the id and continues with the next file again). From then on,
two loops process the remaining files, and each further unreadable file
doubles them again. Each loop also calls done at the end.
Steps to reproduce
With the Docker image xemle/home-gallery:1.21 (1.21.0, commit baa1f87,
Node 24.14.0; checksum.ts is the same on master):
docker run --rm --entrypoint sh xemle/home-gallery:1.21 -c '
mkdir -p /tmp/pics && cd /tmp/pics
for i in 1 2 3 4 5 6 7 8; do head -c 2000 /dev/urandom > photo$i.jpg; done
chmod 000 photo2.jpg photo4.jpg photo6.jpg
chown -R node:node /tmp/pics
su node -s /bin/sh -c "node /app/gallery.js index create -d /tmp/pics -i /tmp/test.idx -l warn" 2>&1 |
grep -o "checksum of [^:]*" | sort | uniq -c
zcat /tmp/test.idx | grep -o "photo[246].jpg\",\"sha1sum\":\"[0-9a-f]*"'Output:
1 checksum of /tmp/pics/photo2.jpg
2 checksum of /tmp/pics/photo4.jpg
4 checksum of /tmp/pics/photo6.jpg
photo2.jpg","sha1sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709
photo4.jpg","sha1sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709
photo6.jpg","sha1sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709
Expected: one warning per unreadable file, and no id for them.
Suggested fix
Call the callback only once, e.g.:
const createSha1 = (filename, progress, cb) => {
var input = fs.createReadStream(filename);
var digest = crypto.createHash('sha1');
let failed = false;
const destroyStream = () => {
input.destroy();
}
process.on('SIGINT', destroyStream)
input.addListener('error', err => {
failed = true;
cb(err);
});
input.addListener('data', data => digest.update(data));
input.addListener('data', progress);
input.addListener('close', () => {
process.off('SIGINT', destroyStream);
if (!failed) {
cb(null, digest.digest('hex'));
}
});
};Tested with the same sequential loop as calculateAll() on the files above:
one warning per unreadable file, done is called once (instead of 8 times),
and the unreadable files get no id.
Environment
- HomeGallery 1.21.0, Docker image
xemle/home-gallery:1.21,linux/arm64
(Raspberry Pi 5), rootless Docker - Found with a photo library on a read-only CIFS mount, where some files were
not readable for the gallery's user because of the permissions on the NAS
Activity