8000 Cucumber Messages miss test cases when features start running during compile_features! · Issue #71 · huddlz-hq/cucumber · GitHub
[go: up one dir, main page]

Skip to content

Cucumber Messages miss test cases when features start running during compile_features! #71

Open

Description

@mwoods79

What happens

With config :cucumber, messages: path, the NDJSON stream can be missing test cases for scenarios that ran and passed. In huddlz (895 scenarios, compile_features! called from test/features_test.exs), each run dropped between 140 and 325 test cases, a different number every time. Cucumber 1.1.1 and current main behave the same.

The missing scenarios belong to the features compiled first: some of each early @async feature is missing, and every feature compiled later is complete. Their source, gherkinDocument and pickle envelopes are present; the testCase, testCaseStarted, testStep* and testCaseFinished envelopes are not.

Why

  1. Messages turn on after every feature module is compiled. Cucumber.Compiler.compile_all!/4 defines each feature's test module, and only after the last one calls Emitter.configure to turn the recorder on. ExUnit hands an async: true module to the runner as soon as it is defined, while test files are still loading (ExUnit.Server.take_modules/1). When compile_features! runs from a test file, the early async features start running before messages are on, and their scenarios are never recorded. Calling it from test_helper.exs, as the README shows, hides the race, because the runner hasn't started yet.

  2. A second compile_features! call resets the run. Each call runs RunCoordinator.ensure_started/0, which resets the coordinator. A project that calls compile_features! twice in one run loses whatever the first call had set up if the second runs while the first's scenarios are going. (huddlz has two calls, in test/features_test.exs and test/browser/browser_features_test.exs, but the browser file only loads with HUDDLZ_BROWSER_TEST=1, so they never share a run and huddlz isn't affected by this part.) The reset loses:

    • the before_all result and its once-guard
    • the registered after_all hooks
    • recorded attachments
    • the message stream

Expected

  • Every scenario that runs appears in the stream, wherever compile_features! is called.
  • Calling compile_features! more than once in a run adds to that run instead of resetting it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0