[go: up one dir, main page]
Skip to content

agilecreativity/babashka.process

BranchesTags
 
 

Repository files navigation

process

Clojars Project bb built-in

Clojure library for shelling out / spawning subprocesses.

API

You will probably mostly need process and check so it would be good to start reading the docs for these. Skim over the rest and come back when you need it.

See API docs as generated by quickdoc.

Installation

This library is included in babashka since 0.2.3 but is also intended as a JVM library:

$ clojure -Sdeps '{:deps {babashka/process {:mvn/version "0.2.9"}}}'

user=> (require '[clojure.string :as str])
nil
user=> (require '[babashka.process :refer [process check]])
nil
user=> (-> (process ["ls" "-la"] {:out :string}) check :out str/split-lines first)
"total 136776"

Differences with clojure.java.shell/sh

If clojure.java.shell works for your purposes, keep using it. But there are contexts in which you need more flexibility. The major differences compared with this library:

  • sh is blocking, process makes blocking explicit via deref
  • sh focuses on convenience but limits what you can do with the underlying process, process exposes as much as possible while still offering an ergonomic API
  • process supports piping processes via -> or pipeline
  • sh offers integration with clojure.java.io/copy for :in, process extends this to :out and :err

Usage

user=> (require '[babashka.process :refer [process check sh pipeline pb]])

Invoke ls:

user=> (-> (process '[ls]) :out slurp)
"LICENSE\nREADME.md\nsrc\n"

Change working directory:

user=> (-> (process '[ls] {:dir "test/babashka"}) :out slurp)
"process_test.clj\n"

Set the process environment.

user=> (-> (process '[sh -c "echo $FOO"] {:env {:FOO "BAR"}}) :out slurp)
"BAR\n"

The return value of process implements clojure.lang.IDeref. When dereferenced, it will wait for the process to finish and will add the :exit value:

user=> (-> @(process '[ls foo]) :exit)
1

The function check takes a process, waits for it to finish and returns it. When the exit code is non-zero, it will throw.

user=> (-> (process '[ls foo]) check :out slurp)
Execution error (ExceptionInfo) at babashka.process/check (process.clj:74).
ls: foo: No such file or directory

Redirect output to stdout:

user=> (do @(process '[ls] {:out :inherit}) nil)
LICENSE		README.md	deps.edn	src		test
nil

Both :in, :out may contain objects that are compatible with clojure.java.io/copy:

user=> (with-out-str (check (process '[cat] {:in "foo" :out *out*})))
"foo"

user=> (with-out-str (check (process '[ls] {:out *out*})))
"LICENSE\nREADME.md\ndeps.edn\nsrc\ntest\n"

The :out option also supports :string. You will need to deref the process in order for the string to be there:

user=> (-> @(process '[ls] {:out :string}) :out)
"LICENSE\nREADME.md\ndeps.edn\nsrc\ntest\n"

Redirect output stream from one process to input stream of the next process:

user=> (let [is (-> (process '[ls]) :out)]
         @(process ["cat"] {:in is
                            :out :inherit})
         nil)
LICENSE
README.md
deps.edn
src
test
nil

Forwarding the output of a process as the input of another process can also be done with thread-first:

user=> (-> (process '[ls])
           (process '[grep "README"]) :out slurp)
"README.md\n"

Redirecting output to a file

To write to a file use :out :write and set :out-file to a file:

user=> (require '[clojure.java.io :as io])
nil
user=> (do @(p/process ["ls"] {:out :write :out-file (io/file "/tmp/out.txt")}) nil)
nil
user=> (slurp "/tmp/out.txt")
"CHANGELOG.md\nLICENSE\nREADME.md\ndeps.edn\nproject.clj\nscript\nsrc\ntest\n"

To append to a file, use :out :append:

user=> (do @(p/process ["ls"] {:out :append :out-file (io/file "/tmp/out.txt")}) nil)
nil
user=> (slurp "/tmp/out.txt")
"CHANGELOG.md\nLICENSE\nREADME.md\ndeps.edn\nproject.clj\nscript\nsrc\ntest\nCHANGELOG.md\nLICENSE\nREADME.md\ndeps.edn\nproject.clj\nscript\nsrc\ntest\n"

Feeding input

Here is an example of a cat process to which we send input while the process is running, then close stdin and read the output of cat afterwards:

(ns cat-demo
  (:require [babashka.process :refer [process]]
            [clojure.java.io :as io]))

(def catp (process '[cat]))

(.isAlive (:proc catp)) ;; true

(def stdin (io/writer (:in catp)))

(binding [*out* stdin]
  (println "hello"))

(.close stdin)

(slurp (:out catp)) ;; "hello\n"

(def exit (:exit @catp)) ;; 0

(.isAlive (:proc catp)) ;; false

Processing output

Here is an example where we read the output of yes line by line and print it ourselves:

(require '[babashka.process :as p :refer [process]]
         '[clojure.java.io :as io])

(def yes (process ["yes"] {:err :inherit
                           :shutdown p/destroy}))

(with-open [rdr (io/reader (:out yes))]
  (binding [*in* rdr]
    (loop []
      (let [line (read-line)]
        (println :line line))
      (recur))))

Printing command

We can use :pre-start-fn to report commands being run:

(require '[babashka.process :refer [process]])

(doseq [file ["LICENSE" "CHANGELOG.md"]]
         (-> (process ["head" "-1" file] {:out :string 
                                           :pre-start-fn #(apply println "Running" (:cmd %))})
             deref :out println))

Running head -1 LICENSE
Eclipse Public License - v 1.0

Running head -1 CHANGELOG.md
# Changelog

sh

$ is a convenience macro around process:

user=> (def config {:output {:format :edn}})
#'user/config
user=> (-> ($ clj-kondo --config ~config --lint "src") deref :out slurp edn/read-string)
{:findings [], :summary {:error 0, :warning 0, :info 0, :type :summary, :duration 34}}

sh is a convenience function around process which sets :out and :err to :string and blocks automatically, similar to clojure.java.shell/sh:

user=> (def config {:output {:format :edn}})
#'user/config
user=> (-> (sh ["clj-kondo" "--lint" "src"]) :out slurp edn/read-string)
{:findings [], :summary {:error 0, :warning 0, :info 0, :type :summary, :duration 34}}

Tokenization

Both process and sh support tokenization when passed a single string argument:

user=> (-> (sh "echo hello there") :out)
"hello there\n"
user=> (-> (sh "clj-kondo --lint -" {:in "(inc)"}) :out print)
<stdin>:1:1: error: clojure.core/inc is called with 0 args but expects 1
linting took 11ms, errors: 1, warnings: 0

Output buffering

Note that check will wait for the process to end in order to check the exit code. When the process has lots of data to write to stdout, it is recommended to add an explicit :out option to prevent deadlock due to buffering. This example will deadlock because the process is buffering the output stream but it's not being consumed, so the process won't be able to finish:

user=> (-> (process ["cat"] {:in (slurp "https://datahub.io/datahq/1mb-test/r/1mb-test.csv")}) check :out slurp count)

The way to deal with this is providing an explicit :out option so the process can finish writing its output:

user=> (-> (process ["cat"] {:in (slurp "https://datahub.io/datahq/1mb-test/r/1mb-test.csv") :out :string}) check :out count)
1043005

Add Environment

The :env option replaces your entire environment with the provided map. To add environment variables you can use :extra-env instead:

:extra-env {"FOO" "BAR"}

Windows TIP: Unlike in the CMD and Powershell shells, environment variable names are case sensitive for :extra-env. For example, "PATH" will not update the value of "Path" on Windows. Here's an example of a babashka task that understands this nuance.

Pipelines

The pipeline function returns a sequential of processes from a process that was created with -> or by passing multiple objects created with pb:

user=> (mapv :cmd (pipeline (-> (process '[ls]) (process '[cat]))))
[["ls"] ["cat"]]
user=> (mapv :cmd (pipeline (pb '[ls]) (pb '[cat])))
[["ls"] ["cat"]]

To obtain the right-most process from the pipeline, use last (or peek):

user=> (-> (pipeline (pb ["ls"]) (pb ["cat"])) last :out slurp)
"LICENSE\nREADME.md\ndeps.edn\nsrc\ntest\n"

Calling pipeline on the right-most process returns the pipeline:

user=> (def p (pipeline (pb ["ls"]) (pb ["cat"])))
#'user/p
user=> (= p (pipeline (last p)))
true

To check an entire pipeline for non-zero exit codes, you can use:

user=> (run! check (pipeline (-> (process '[ls "foo"]) (process '[cat]))))
Execution error (ExceptionInfo) at babashka.process/check (process.clj:37).
ls: foo: No such file or directory

Although you can create pipelines with ->, for some applications it may be preferable to create a pipeline with pipeline which defers to ProcessBuilder/startPipeline. In the following case it takes a long time before you would see any output due to buffering.

(future
  (loop []
    (spit "log.txt" (str (rand-int 10) "\n") :append true)
    (Thread/sleep 10)
    (recur)))

(-> (process '[tail -f "log.txt"])
    (process '[cat])
    (process '[grep "5"] {:out :inherit}))

The solution then it to use pipeline + pb:

(pipeline (pb '[tail -f "log.txt"])
          (pb '[cat])
          (pb '[grep "5"] {:out :inherit}))

The varargs arity of pipeline is only available in JDK9 or higher due to the availability of ProcessBuilder/startPipeline. If you are on JDK8 or lower, the following solution that reads the output of tail line by line may work for you:

(def tail (process '[tail -f "log.txt"] {:err :inherit}))

(def cat-and-grep
  (-> (process '[cat]      {:err :inherit})
      (process '[grep "5"] {:out :inherit
                            :err :inherit})))

(binding [*in*  (io/reader (:out tail))
          *out* (io/writer (:in cat-and-grep))]
  (loop []
    (when-let [x (read-line)]
      (println x)
      (recur))))

Another solution is to let bash handle the pipes by shelling out with bash -c.

Notes

Script termination

Because process spawns threads for non-blocking I/O, you might have to run (shutdown-agents) at the end of your Clojure JVM scripts to force termination. Babashka does this automatically.

Clojure.pprint

When pretty-printing a process, by default you will get an exception:

(require '[clojure.pprint :as pprint])
(pprint/pprint (process ["ls"]))
Execution error (IllegalArgumentException) at user/eval257 (REPL:1).
Multiple methods in multimethod 'simple-dispatch' match dispatch value: class babashka.process.Process -> interface clojure.lang.IDeref and interface clojure.lang.IPersistentMap, and neither is preferred

The reason is that a process is both a record and a clojure.lang.IDeref and pprint does not have a preference for how to print this. Two potential resolutions for this are:

  • require the babashka.process.pprint namespace, which will define a pprint implementation for a Process record:
(require '[babashka.process.pprint]
         '[clojure.pprint :as pprint])

(pprint/pprint (process ["ls"]))

=> {:proc
    #object[java.lang.ProcessImpl...]
    ...
    }
  • define a preference for pprint's dispatch mulitmethod:
(prefer-method pprint/simple-dispatch clojure.lang.IPersistentMap clojure.lang.IDeref)

Promesa

You can combine this library with promesa in the following way. This requires :exit-fn which was released in version 0.2.10.

(require '[babashka.process :as proc]
         '[promesa.core :as prom])

(defn process
  "Returns promise that will be resolved upon process termination. The promise is rejected when the exit code is non-zero."
  [cmd opts]
  (prom/create
   (fn [resolve reject]
     (let [exit-fn (fn [response]
                     (let [{:keys [exit] :as r} response]
                       (if (zero? exit)
                         (resolve r)
                         (reject r))))]
       (proc/process cmd (assoc opts :exit-fn exit-fn))))))

(prom/let [ls (process "ls" {:out :string
                             :err :inherit})
           ls-out (:out ls)]
  (prn ls-out))

License

Copyright © 2020-2021 Michiel Borkent

Distributed under the EPL License. See LICENSE.

About

Clojure wrapper for java.lang.ProcessBuilder

Resources

Readme

License

EPL-1.0 license
Activity

Stars

0 stars

Watchers

2 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages

  • Clojure 99.6%
  • Shell 0.4%