1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269

# Plumbing and Graph: the Clojure utility belt
<img src="https://raw.github.com/wiki/plumatic/plumbing/images/prismaticswissarmyknife.png" alt="prismatic/plumbing logo" title="prismatic/plumbing logo" align="right" width="250" />
This first release includes our '[Graph](http://plumatic.github.io/prismaticsgraphatstrangeloop)' library, our `plumbing.core` library of very commonly used functions (the only namespace we `:use` across our codebase), and a few other supporting namespaces.
*New in 0.3.0: support for ClojureScript*
*New in 0.2.0: support for schema.core/defnstyle schemas on fnks and Graphs. See `(doc fnk)` for details.*
Leiningen dependency (Clojars):
[![Clojars Project](http://clojars.org/prismatic/plumbing/latestversion.svg)](http://clojars.org/prismatic/plumbing)
[Latest API docs](http://plumatic.github.io/plumbing).
**This is an alpha release. We are using it internally in production, but the API and organizational structure are subject to change. Comments and suggestions are much appreciated.**
Check back often, because we'll keep adding more useful namespaces and functions as we work through cleaning up and opensourcing our stack of Clojure libraries.
## Graph: the Functional SwissArmy Knife
Graph is a simple and *declarative* way to specify a structured computation, which is easy to analyze, change, compose, and monitor. Here's a simple example of an ordinary function definition, and its Graph equivalent:
```clojure
(require '[plumbing.core :refer (sum)])
(defn stats
"Take a map {:xs xs} and return a map of simple statistics on xs"
[{:keys [xs] :as m}]
(assert (contains? m :xs))
(let [n (count xs)
m (/ (sum identity xs) n)
m2 (/ (sum #(* % %) xs) n)
v ( m2 (* m m))]
{:n n ; count
:m m ; mean
:m2 m2 ; meansquare
:v v ; variance
}))
(require '[plumbing.core :refer (fnk sum)])
(def statsgraph
"A graph specifying the same computation as 'stats'"
{:n (fnk [xs] (count xs))
:m (fnk [xs n] (/ (sum identity xs) n))
:m2 (fnk [xs n] (/ (sum #(* % %) xs) n))
:v (fnk [m m2] ( m2 (* m m)))})
```
A Graph is just a map from keywords to keyword functions ([learn more](#fnk)). In this case, `statsgraph` represents the steps in taking a sequence of numbers (`xs`) and producing univariate statistics on those numbers (i.e., the mean `m` and the variance `v`). The names of arguments to each `fnk` can refer to other steps that must happen before the step executes. For instance, in the above, to execute `:v`, you must first execute the `:m` and `:m2` steps (mean and meansquare respectively).
We can "compile" this Graph to produce a single function (equivalent to `stats`), which also checks that the map represents a valid Graph:
```clojure
(require '[plumbing.graph :as graph] '[schema.core :as s])
(def statseager (graph/compile statsgraph))
(= {:n 4
:m 3
:m2 (/ 25 2)
:v (/ 7 2)}
(into {} (statseager {:xs [1 2 3 6]})))
;; Missing :xs key exception
(thrown? Throwable (statseager {:ys [1 2 3]}))
```
Moreover, as of the 0.1.0 release, `statseager` is *fast*  only about 30% slower than the handcoded `stats` if `xs` has a single element, and within 5% of `stats` if `xs` has ten elements.
Unlike the opaque `stats` fn, however, we can modify and extend `statsgraph` using ordinary operations on maps:
```clojure
(def extendedstats
(graph/compile
(assoc statsgraph
:sd (fnk [^double v] (Math/sqrt v)))))
(= {:n 4
:m 3
:m2 (/ 25 2)
:v (/ 7 2)
:sd (Math/sqrt 3.5)}
(into {} (extendedstats {:xs [1 2 3 6]})))
```
A Graph encodes the structure of a computation, but not how it happens, allowing for many execution strategies. For example, we can compile a Graph lazily so that step values are computed as needed. Or, we can parallelcompile the Graph so that independent step functions are run in separate threads:
```clojure
(def lazystats (graph/lazycompile statsgraph))
(def output (lazystats {:xs [1 2 3 6]}))
;; Nothing has actually been computed yet
(= (/ 25 2) (:m2 output))
;; Now :n and :m2 have been computed, but :v and :m are still behind a delay
(def parstats (graph/parcompile statsgraph))
(def output (parstats {:xs [1 2 3 6]}))
;; Nodes are being computed in futures, with :m and :m2 going in parallel after :n
(= (/ 7 2) (:v output))
```
We can also ask a Graph for information about its inputs and outputs (automatically computed from its definition):
```clojure
(require '[plumbing.fnk.pfnk :as pfnk])
;; statsgraph takes a map with one required key, :xs
(= {:xs s/Any}
(pfnk/inputschema statsgraph))
;; statsgraph outputs a map with four keys, :n, :m, :m2, and :v
(= {:n s/Any :m s/Any :m2 s/Any :v s/Any}
(pfnk/outputschema statsgraph))
```
If schemas are provided on the inputs and outputs of the node functions, these propagate through into the Graph schema as expected.
We can also have higherorder functions on Graphs to wrap the behavior on each step. For instance, we can automatically profile each subfunction in 'stats' to see how long it takes to execute:
```clojure
(def profiledstats (graph/compile (graph/profiled ::profiledata statsgraph)))
;;; times in milliseconds for each step:
(= {:n 1.001, :m 0.728, :m2 0.996, :v 0.069}
@(::profiledata (profiledstats {:xs (range 10000)})))
```
… and so on. For more examples and details about Graph, check out the [graph examples test](https://github.com/plumatic/plumbing/blob/master/test/plumbing/graph_examples_test.cljx).
<a name="fnk"/>
## Bring on (de)fnk
Many of the functions we write (in Graph and elsewhere) take a single (nested) map argument with keyword keys and have expectations about which keys must be present and which are optional. We developed a new style of binding ([read more here](https://github.com/plumatic/plumbing/tree/master/src/plumbing/fnk)) to make this a lot easier and to check that input data has the right 'shape'. We call these 'keyword functions' (defined by `defnk`) and here's what one looks like:
```clojure
(use 'plumbing.core)
(defnk simplefnk [a b c]
(+ a b c))
(= 6 (simplefnk {:a 1 :b 2 :c 3}))
;; Below throws: Key :c not found in (:a :b)
(thrown? Throwable (simplefnk {:a 1 :b 2}))
```
You can declare a key as optional and provide a default:
```clojure
(defnk simpleoptfnk [a b {c 1}]
(+ a b c))
(= 4 (simpleoptfnk {:a 1 :b 2}))
```
You can do nested map bindings:
```clojure
(defnk simplenestedfnk [a [:b b1] c]
(+ a b1 c))
(= 6 (simplenestedfnk {:a 1 :b {:b1 2} :c 3}))
;; Below throws: Expected a map at keypath [:b], got type class java.lang.Long
(thrown? Throwable (simplenestedfnk {:a 1 :b 1 :c 3}))
```
Of course, you can bind multiple variables from an inner map and do multiple levels of nesting:
```clojure
(defnk simplenestedfnk2 [a [:b b1 [:c {d 3}]]]
(+ a b1 d))
(= 4 (simplenestedfnk2 {:a 1 :b {:b1 2 :c {:d 1}}}))
(= 5 (simplenestedfnk2 {:a 1 :b {:b1 1 :c {}}}))
```
You can also use this binding style in a `let` statement using `letk`
or within an anonymous function by using `fnk`.
## More good stuff
There are a bunch of functions in `plumbing.core` that we can't live without. Here are a few of our favorites.
When we build maps, we often use `formap`, which works like `for` but for maps:
```clojure
(use 'plumbing.core)
(= (formap [i (range 3)
j (range 3)
:let [s (+ i j)]
:when (< s 3)]
[i j]
s)
{[0 0] 0, [0 1] 1, [0 2] 2, [1 0] 1, [1 1] 2, [2 0] 2})
```
`safeget` is like `get` but throws when the key doesn't exist:
```clojure
;; IllegalArgumentException Key :c not found in {:a 1, :b 2}
(thrown? Exception (safeget {:a 1 :b 2} :c))
```
Another frequently used map function is `mapvals`:
```clojure
;; return k > (f v) for [k, v] in map
(= (mapvals inc {:a 0 :b 0})
{:a 1 :b 1})
```
Ever wanted to conditionally do steps in a `>>` or `>`? Now you can with our
'penguin' operators. Here's a few examples:
```clojure
(use 'plumbing.core)
(= (let [addb? false]
(> {:a 1}
(merge {:c 2})
(?> addb? (assoc :b 2))))
{:a 1 :c 2})
(= (let [incall? true]
(>> (range 10)
(filter even?)
(?>> incall? (map inc))))
[1 3 5 7 9])
```
Check out [`plumbing.core`](https://github.com/plumatic/plumbing/blob/master/src/plumbing/core.cljx) for many other useful functions.
## ClojureScript
As of 0.3.0, plumbing is available in ClojureScript! The vast majority of the
library supports ClojureScript, with the only exceptions that are JVMspecific
optimizations.
Here's an example usage of `formap`:
```clojure
(ns plumbing.readme
(:require [plumbing.core :refermacros [formap]]))
(defn jsobj>map
"Recursively converts a JavaScript object into a map with keyword keys"
[obj]
(formap [k (jskeys obj)
:let [v (aget obj k)]]
(keyword k) (if (object? v) (jsobj>map v) v)))
(is (= {:a 1 :b {:x "x" :y "y"}}
(jsobj>map
(jsobj "a" 1
"b" (jsobj "x" "x"
"y" "y")))))
;; Note: this is a contrived example; you would normally use `cljs.core/clj>js`
```
## Community
Plumbing now has a [mailing list](https://groups.google.com/forum/#!forum/prismaticplumbing). Please feel free to join and ask questions or discuss how you're using Plumbing and Graph.
## Supported Clojure versions
Plumbing is currently supported on Clojure 1.5.x and 1.6.x.
## License
Distributed under the Eclipse Public License, the same as Clojure.
