process

Clojure library for shelling out / spawning sub-processes.

API

In 90% of the use cases you will probably need shell and for the remaining use cases you will probably need a combination of process and check. Start reading the docs for those, skim over the rest and revisit the remaining functions when you need them.

See API docs as generated by quickdoc.

Installation

This library is included in babashka since 0.2.3 but is also intended as a JVM library:

$ clojure -Sdeps '{:deps {babashka/process {:mvn/version "0.3.11"}}}'

user=> (require '[clojure.string :as str])
nil
user=> (require '[babashka.process :refer [shell sh process check]])
nil
user=> (shell "ls -la")
total 144
drwxr-xr-x@ 22 borkdude  staff    704 Dec  4 13:39 .
drwxr-xr-x@ 75 borkdude  staff   2400 Dec  3 14:18 ..
drwxr-xr-x@  4 borkdude  staff    128 Mar 10  2022 .circleci
drwxr-xr-x@  5 borkdude  staff    160 Mar 10  2022 .clj-kondo
drwxr-xr-x@ 50 borkdude  staff   1600 Dec  3 20:55 .cpcache
...
user=> (-> (shell {:out :string} "ls -la") :out str/split-lines first)
"total 136776"

Usage

user=> (require '[babashka.process :refer [process check sh pipeline pb]])

shell

Most commonly you will use shell. It executes a command and streams the output to stdout and stderr while the process is running:

user=> (shell "ls -la")
total 144
drwxr-xr-x@ 22 borkdude  staff    704 Dec  4 13:39 .
drwxr-xr-x@ 75 borkdude  staff   2400 Dec  3 14:18 ..
drwxr-xr-x@  4 borkdude  staff    128 Mar 10  2022 .circleci
drwxr-xr-x@  5 borkdude  staff    160 Mar 10  2022 .clj-kondo
drwxr-xr-x@ 50 borkdude  staff   1600 Dec  3 20:55 .cpcache

The first string argument to shell is tokenized automatically: "ls -la" is broken up into "ls" and "-la". You can provide more arguments if you need to:

user=> (shell "ls -la" "src" "test")
src:
total 0
drwxr-xr-x@  3 borkdude  staff   96 Mar 10  2022 .
drwxr-xr-x@ 22 borkdude  staff  704 Dec  4 14:13 ..
drwxr-xr-x@  4 borkdude  staff  128 Dec  4 14:01 babashka

test:
total 0
drwxr-xr-x@  3 borkdude  staff   96 Mar 10  2022 .
drwxr-xr-x@ 22 borkdude  staff  704 Dec  4 14:13 ..
drwxr-xr-x@  3 borkdude  staff   96 Dec  4 14:01 babashka

This is particularly handy when you want to supply commands coming from the command line:

(apply shell "-ls -la" *command-line-args*)

The shell function checks the exit code and throws if it is non-zero:

user=> (shell "ls nothing")
ls: nothing: No such file or directory
Execution error (ExceptionInfo) at babashka.process/check (process.cljc:113).

To avoid throwing, you can use :continue true. You will still see the error being printed to stderr, but no exception will be thrown. That is convenient when you want to handle the :exit code yourself:

user=> (-> (shell {:continue true} "ls nothing") :exit)
ls: nothing: No such file or directory
1

To collect output as a :string, use the :out :string option as the first argument:

user=> (-> (shell {:out :string} "ls -la") :out str/split-lines first)
"total 144"

To change the working directory, use the :dir option:

user=> (-> (shell {:out :string :dir "src/babashka"} "ls -la") :out str/split-lines first)
"total 48"

To add environment variables, use :extra-env:

user=> (-> (shell {:out :string :extra-env {"FOO" "BAR"}} "bb -e '(System/getenv \"FOO\")'") :out print)
"BAR"

process

The shell function is a combination of process and deref and check. The process function is the lower level function of this library that doesn't make any opiniated choices:

It does not provide a default for :out, :in and :err: in shell these default to :inherit which means: read and write from and to the console. In process they default to the default of java.lang.ProcessBuilder.
It does not wait until the process completes.
It does not check the exit code and throw an exception.

Use process when you need to change one of the above and shell's options do not support it. In practice this means: whenever you need async processing, e.g. reading output from a process while it is running.

The return value of process implements clojure.lang.IDeref. When dereferenced, it will wait for the process to finish and will add the :exit value.

user=> (-> (process "ls foo") deref :exit)
1

The function check takes a process, waits for it to finish (so you can omit deref) and returns it. When the exit code is non-zero, it will throw.

user=> (-> (process {:out :string} "ls") check :out str/split-lines first)
"API.md"
user=> (-> (process {:out :string} "ls foo") check :out str/split-lines first)
Execution error (ExceptionInfo) at babashka.process/check (process.clj:74).
ls: foo: No such file or directory

Both :in, :out may contain objects that are compatible with clojure.java.io/copy:

user=> (with-out-str (check (process {:in "foo" :out *out*} "cat")))
"foo"

user=> (->> (with-out-str (check (process {:out *out*} "ls"))) str/split-lines (take 2))
("API.md" "CHANGELOG.md")

The :out option also supports :string. You will need to deref the process in order for the string to be there, since the output can't be finalized if the process hasn't finished running:

user=> (-> @(process {:out :string} "ls") :out str/split-lines first)
"API.md"

Piping output

Both shell and process support piping output from one process to the next using but note that shell writes the output to the system's stdout by default, so you have to provide it with {:out :string} for the next process to capture the input, while process uses the default java.lang.ProcessBuilder setting which defaults to writing to a stream:

user=> (let [stream (-> (process "ls") :out)]
         @(process {:in stream
                    :out :inherit} "cat")
         nil)
API.md
CHANGELOG.md
LICENSE
README.md
...

Forwarding the output of a process as the input of another process can also be done with thread-first (->):

user=> (-> (process "ls")
           (process {:out :string} "grep README") deref :out)
"README.md\n"

Redirecting output to a file

To write to a file use :out :write and set :out-file to a file:

user=> (require '[clojure.java.io :as io])
nil
user=> (do @(process {:out :write :out-file (io/file "/tmp/out.txt")} "ls") nil)
nil
user=> (slurp "/tmp/out.txt")
"CHANGELOG.md\nLICENSE\nREADME.md..."

or simply:

(do (shell {:out "/tmp/out.txt"} "ls") nil)

To append to a file, use :out :append:

user=> (do @(process {:out :append :out-file (io/file "/tmp/out.txt")} "ls") nil)
nil
user=> (slurp "/tmp/out.txt")
"CHANGELOG.md\nLICENSE\nREADME.md..."

Feeding input

Here is an example of a cat process to which we send input while the process is running, then close stdin and read the output of cat afterwards:

(ns cat-demo
  (:require [babashka.process :refer [process alive?]]
            [clojure.java.io :as io]))

(def catp (process "cat"))

(alive? catp) ;; true

(def stdin (io/writer (:in catp)))

(binding [*out* stdin]
  (println "hello"))

(.close stdin)

(slurp (:out catp)) ;; "hello\n"

(:exit @catp) ;; 0

(alive? catp) ;; false

Processing output

Here is an example where we read the output of yes line by line and print it ourselves:

(require '[babashka.process :as p :refer [process destroy-tree]]
         '[clojure.java.io :as io])

(def yesp (process
           {:err :inherit
            :shutdown destroy-tree}
           "yes"))

;; Beware: infinite loop reading lines of "y" from the `yes` process:

(with-open [rdr (io/reader (:out yesp))]
  (binding [*in* rdr]
    (loop []
      (let [line (read-line)]
        (println :line line))
      (recur))))

Printing command

The :pre-start-fn option can be used to report commands being run:

(require '[babashka.process :refer [process]])

(doseq [file ["LICENSE" "CHANGELOG.md"]]
  (-> (process
        {:out :string
         :pre-start-fn #(apply println "Running" (:cmd %))}
        "head" "-1" file)
      deref :out println))

Running head -1 LICENSE
Eclipse Public License - v 1.0

Running head -1 CHANGELOG.md
# Changelog

sh

sh is a convenience function around process which sets :out and :err to :string and blocks automatically, similar to clojure.java.shell/sh:

user=> (def config {:output {:format :edn}})
#'user/config
user=> (-> (sh ["clj-kondo" "--lint" "src"]) :out slurp edn/read-string)
{:findings [], :summary {:error 0, :warning 0, :info 0, :type :summary, :duration 34}}

Tokenization

All of shell, process and sh support tokenization on the first string argument using tokenize:

user=> (require '[babashka.process :refer [sh tokenize]])
nil
user=> (tokenize "hello there")
["hello" "there"]
user=> (-> (sh "echo hello there") :out)
"hello there\n"

user=> (-> (sh {:in "(inc)"} "clj-kondo --lint -") :out)
"<stdin>:1:1: error: clojure.core/inc is called with 0 args but expects 1\nlinting took 10ms, errors: 1, warnings: 0\n"

Output buffering

Note that check will wait for the process to end in order to check the exit code. When the process has lots of data to write to stdout, it is recommended to add an explicit :out option to prevent deadlock due to buffering. This example will deadlock because the process is buffering the output stream but it's not being consumed, so the process won't be able to finish:

user=> (-> (process {:in (apply str (repeat 1000000 "hello\n"))} "cat") check :out count)

The way to deal with this is providing an explicit :out option so the process can finish writing its output:

user=> (-> (process {:out :string :in (apply str (repeat 1000000 "hello\n"))} "cat") check :out count)
6000000

Add Environment

The :env option replaces your entire environment with the provided map. To add environment variables you can use :extra-env instead:

:extra-env {"FOO" "BAR"}

Windows TIP: Unlike in the CMD and Powershell shells, environment variable names are case sensitive for :extra-env. For example, "PATH" will not update the value of "Path" on Windows. Here's an example of a babashka task that understands this nuance.

Pipelines

The pipeline function returns a sequential of processes from a process that was created with -> or by passing multiple objects created with pb:

user=> (require '[babashka.process :refer [pipeline pb process check]])
nil
user=> (mapv :cmd (pipeline (-> (process "ls") (process "ls"))))
[["ls"] ["cat"]]
user=> (mapv :cmd (pipeline (pb "ls") (pb "cat")))
[["ls"] ["cat"]]

To obtain the right-most process from the pipeline, use last (or peek):

user=> (-> (pipeline (pb "ls") (pb "cat")) last :out slurp)
"LICENSE\nREADME.md\ndeps.edn\nsrc\ntest\n..."

Calling pipeline on the right-most process returns the pipeline:

user=> (def p (pipeline (pb "ls") (pb "cat")))
#'user/p
user=> (= p (pipeline (last p)))
true

To check an entire pipeline for non-zero exit codes, you can use:

user=> (run! check (pipeline (pb "ls foo") (pb "cat")))
Execution error (ExceptionInfo) at babashka.process/check (process.clj:37).
ls: foo: No such file or directory

Although you can create pipelines with ->, for some applications it may be preferable to create a pipeline with pipeline which defers to ProcessBuilder/startPipeline. In the following case it takes a long time before you would see any output due to buffering.

(future
  (loop []
    (spit "log.txt" (str (rand-int 10) "\n") :append true)
    (Thread/sleep 10)
    (recur)))

(-> (process "tail" "-f" "log.txt")
    (process "cat")
    (process {:out :inherit} "grep "5"))

The solution then it to use pipeline + pb:

(pipeline (pb "tail" "-f" "log.txt")
          (pb "cat")
          (pb {:out :inherit} "grep" "5"))

The varargs arity of pipeline is only available in JDK9 or higher due to the availability of ProcessBuilder/startPipeline. If you are on JDK8 or lower, the following solution that reads the output of tail line by line may work for you:

user=> (require '[clojure.java.io :as io])
nil

(def tail (process {:err :inherit} "tail" "-f" "log.txt"))

(def cat-and-grep
  (-> (process {:err :inherit} "cat")
      (process {:out :inherit
                :err :inherit} "grep 5")))

(binding [*in*  (io/reader (:out tail))
          *out* (io/writer (:in cat-and-grep))]
  (loop []
    (when-let [x (read-line)]
      (println x)
      (recur))))

Another solution is to let bash handle the pipes by shelling out with bash -c.

Differences with `clojure.java.shell/sh`

If clojure.java.shell works for your purposes, keep using it. But there are contexts in which you need more flexibility. The major differences compared with this library:

sh is blocking, process makes blocking explicit via deref
sh focuses on convenience but limits what you can do with the underlying process, process exposes as much as possible while still offering an ergonomic API
process supports piping processes via -> or pipeline
sh offers integration with clojure.java.io/copy for :in, process extends this to :out and :err

###Script termination

Because process spawns threads for non-blocking I/O, you might have to run (shutdown-agents) at the end of your Clojure JVM scripts to force termination. Babashka does this automatically.

Clojure.pprint

When pretty-printing a process, by default you will get an exception:

user=> (require '[babashka.process :refer [process]])
nil
user=> (require '[clojure.pprint :as pprint])
nil
user=> (pprint/pprint (process "ls"))
Execution error (IllegalArgumentException) at user/eval257 (REPL:1).
Multiple methods in multimethod 'simple-dispatch' match dispatch value: class babashka.process.Process -> interface clojure.lang.IDeref and interface clojure.lang.IPersistentMap, and neither is preferred

The reason is that a process is both a record and a clojure.lang.IDeref and pprint does not have a preference for how to print this. The recommended solution is to require the babashka.process.pprint namespace, which will define a pprint implementation for a Process record:

user=> (require '[babashka.process.pprint])
nil

user=> (pprint/pprint (process "ls"))
{:proc
 #object[java.lang.ProcessImpl 0x1d61a348 "Process[pid=43771, exitValue=\"not exited\"]"],
 :exit nil,
...

Promesa

You can combine this library with promesa in the following way. This requires :exit-fn which was released in version 0.2.10.

(require '[babashka.process :as proc]
         '[promesa.core :as prom])

(defn process
  "Returns promise that will be resolved upon process termination. The promise is rejected when the exit code is non-zero."
  [opts & cmd]
  (prom/create
   (fn [resolve reject]
     (let [exit-fn (fn [response]
                     (let [{:keys [exit] :as r} response]
                       (if (zero? exit)
                         (resolve r)
                         (reject r))))]
       (apply proc/process (assoc opts :exit-fn exit-fn) cmd)))))

(prom/let [ls (process
               {:out :string
                :err :inherit}
               "ls")
           ls-out (:out ls)]
  (prn ls-out))

License

Distributed under the EPL License. See LICENSE.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

process

API

Installation

Usage

shell

process

Piping output

Redirecting output to a file

Feeding input

Processing output

Printing command

sh

Tokenization

Output buffering

Add Environment

Pipelines

Differences with `clojure.java.shell/sh`

Clojure.pprint

Promesa

License

About

Releases

Packages

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 228 Commits
.circleci		.circleci
.clj-kondo		.clj-kondo
.github/workflows		.github/workflows
doc		doc
script		script
src/babashka		src/babashka
test/babashka		test/babashka
.gitignore		.gitignore
API.md		API.md
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
bb.edn		bb.edn
deps.edn		deps.edn
project.clj		project.clj

License

pesterhazy/process

Folders and files

Latest commit

History

Repository files navigation

process

API

Installation

Usage

shell

process

Piping output

Redirecting output to a file

Feeding input

Processing output

Printing command

sh

Tokenization

Output buffering

Add Environment

Pipelines

Differences with clojure.java.shell/sh

Clojure.pprint

Promesa

License

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Differences with `clojure.java.shell/sh`

Packages