Repository navigation
Require explicit opt-in to run shell commands for real #123
Description
I just spent nearly an hour working why a test worked in a local environment, but not remotely. Basically, it's decidedly non-obvious that system calls are run for real by default.
To take a simple example, if you write the following:
:echomsg system('uname')
! uname
$ Linux
~ Linux
... then you have a test that verifies that a system call is made to uname, and you've faked it to return "Linux". (Presumably in a real test, :echomsg system('uname') would actually be something more useful, and the uname command would be called from some internal implementation detail.)
If you accidentally forget the $ hijack line, then (assuming the command is valid on your machine), the tests will still pass, but now they'll be non-hermetic and will execute the command for real.
That might be good in some cases — it's effectively an integration test, and that can be valuable too — but it's far too easy to do by mistake.
I think a better option would be to require an explicit decision in the test to either mock each command (by providing a hijack for stdout and/or the exit status) or to opt-in to run the command for real.
Perhaps this might look something like:
:echomsg system('uname')
! uname (allow-execution-in-test)
~ Linux
#4 is also related, since currently Vroom allows unexpected commands to execute as well.
Activity
dbarnett commentedon Jan 13, 2023
Ooh I like that!
dbarnett commentedon Mar 18, 2023
FWIW, I just ran into a case for the same and remembered this FR.
I intentionally broke some code for testing and got ugly errors with
Press ENTERprompts from it trying to shell out to weird commands instead of the straightforward errors I'd intuitively expected saying it was trying to improperly run weird shell commands.