You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cucumber Messages miss test cases when features start running during compile_features! #71
With config :cucumber, messages: path, the NDJSON stream can be missing test cases for scenarios that ran and passed. In huddlz (895 scenarios, compile_features! called from test/features_test.exs), each run dropped between 140 and 325 test cases, a different number every time. Cucumber 1.1.1 and current main behave the same.
The missing scenarios belong to the features compiled first: some of each early @async feature is missing, and every feature compiled later is complete. Their source, gherkinDocument and pickle envelopes are present; the testCase, testCaseStarted, testStep* and testCaseFinished envelopes are not.
Why
Messages turn on after every feature module is compiled.Cucumber.Compiler.compile_all!/4 defines each feature's test module, and only after the last one calls Emitter.configure to turn the recorder on. ExUnit hands an async: true module to the runner as soon as it is defined, while test files are still loading (ExUnit.Server.take_modules/1). When compile_features! runs from a test file, the early async features start running before messages are on, and their scenarios are never recorded. Calling it from test_helper.exs, as the README shows, hides the race, because the runner hasn't started yet.
A second compile_features! call resets the run. Each call runs RunCoordinator.ensure_started/0, which resets the coordinator. A project that calls compile_features! twice in one run loses whatever the first call had set up if the second runs while the first's scenarios are going. (huddlz has two calls, in test/features_test.exs and test/browser/browser_features_test.exs, but the browser file only loads with HUDDLZ_BROWSER_TEST=1, so they never share a run and huddlz isn't affected by this part.) The reset loses:
the before_all result and its once-guard
the registered after_all hooks
recorded attachments
the message stream
Expected
Every scenario that runs appears in the stream, wherever compile_features! is called.
Calling compile_features! more than once in a run adds to that run instead of resetting it.
Activity