Replace vinyl-fs.src with glob-stream inside stream.pipeline

Antora currently uses the src function from vinyl-fs to locate (i.e., glob) and read files from the filesystem. This function is used in the content aggregator to collect files from the worktree of a local repository and in the UI loader to load the bundle from a local directory and to overlay supplemental files. This issue brings forth the proposal to switch to using the low-level globStream function from glob-stream (which vinyl-fs.src wraps) inside a stream.pipeline (from Node.js stdlib).

While the src function is convenient, it aborts at the first file that it cannot read [1]. This blocks us from being able to deal with these errors gracefully by logging a message, skipping the file, and continuing on (as proposed in #707 (closed)). The function also provides functionally that we simply don't need, such as resolving symlinks (which is already done by glob-stream) and adding a sourcemap. We also need to use our own File implementation that use posix paths, so we have to do a follow-up step to replace the File objects that vinyl-fs.src creates anyway. Plus, the pipeline capabilities that vinyl-fs.src provides are now a part of the Node.js stdlib in the form of stream.pipeline. So we might end up with better performance overall.

Here's the current way:

return new Promise((resolve, reject) =>
  vfs
    .src('**/*[!~]', { cwd, follow: true, nomount: true, nosort: true, nounique: true, etc... })
    .on('error', (err) => {
      // decorate error here; there is no skip
      reject(err)
    })
    .pipe(relativizeFiles)
    .pipe(gatherStreamData(resolve))

Here's the new way:

return new Promise((resolve, reject) => {
  const files = []
  pipeline(
    gs('**/*[!~]', { cwd, follow: true, nomount: true, nosort: true, nounique: true, etc... }),
    map(({ path: abspathPosix }, _, next) => {
      const abspath = posixify ? ospath.normalize(abspathPosix) : abspathPosix
      const relpath = abspath.substr(cwd.length + 1)
      smartStat(abspath).then(
        (stat) => {
          if (stat.isDirectory()) return next()
          fsp.readFile(abspath).then(
            (contents) => {
              files.push(
                new File({ path: posixify ? posixify(relpath) : relpath, contents, stat, src: { abspath } })
              )
              next()
            },
            (readErr) => {
              // decorate or skip error here
              next(readErr)
            }
          )
        },
        (statErr) => {
          // decorate or skip error here
          next(statErr)
        }
      )
    }),
    (err) => (err ? reject(err) : resolve(files))
  )
})

(smartStat is our own function that follows a symlink and flags a symlink error as such)

Although this is more code, it's because it gives us more control. It could be encapsulated in a function. Note that glob-stream always returns posixified paths, so the function has to renormalize those paths for error messages, then back again when making the File when running on Windows.

While this issue doesn't address #707 (closed), it lays the foundation for what we need to address it.

[1] The error also lacks important information for making a helpful error message, such as when it's a broken symlink.