1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
|
<!-- README.md is generated from README.Rmd. Please edit that file -->
# sparseMatrixStats <a href='https://github.com/const-ae/sparseMatrixStats'><img src='man/figures/logo.png' align="right" height="209" /></a>
<!-- badges: start -->
[](https://codecov.io/gh/const-ae/sparseMatrixStats)
<!-- badges: end -->
The goal of `sparseMatrixStats` is to make the API of
[matrixStats](https://github.com/HenrikBengtsson/matrixStats) available
for sparse matrices.
## Installation
You can install the release version of
*[sparseMatrixStats](https://bioconductor.org/packages/sparseMatrixStats)*
from BioConductor:
``` r
if (!requireNamespace("BiocManager", quietly = TRUE))
install.packages("BiocManager")
BiocManager::install("sparseMatrixStats")
```
Alternatively, you can get the development version of the package from
[GitHub](https://github.com/const-ae/sparseMatrixStats) with:
``` r
# install.packages("devtools")
devtools::install_github("const-ae/sparseMatrixStats")
```
If you have trouble with the installation, see the end of the README.
## Example
``` r
library(sparseMatrixStats)
#> Loading required package: MatrixGenerics
#> Loading required package: matrixStats
#>
#> Attaching package: 'MatrixGenerics'
#> The following objects are masked from 'package:matrixStats':
#>
#> colAlls, colAnyNAs, colAnys, colAvgsPerRowSet, colCollapse,
#> colCounts, colCummaxs, colCummins, colCumprods, colCumsums,
#> colDiffs, colIQRDiffs, colIQRs, colLogSumExps, colMadDiffs,
#> colMads, colMaxs, colMeans2, colMedians, colMins, colOrderStats,
#> colProds, colQuantiles, colRanges, colRanks, colSdDiffs, colSds,
#> colSums2, colTabulates, colVarDiffs, colVars, colWeightedMads,
#> colWeightedMeans, colWeightedMedians, colWeightedSds,
#> colWeightedVars, rowAlls, rowAnyNAs, rowAnys, rowAvgsPerColSet,
#> rowCollapse, rowCounts, rowCummaxs, rowCummins, rowCumprods,
#> rowCumsums, rowDiffs, rowIQRDiffs, rowIQRs, rowLogSumExps,
#> rowMadDiffs, rowMads, rowMaxs, rowMeans2, rowMedians, rowMins,
#> rowOrderStats, rowProds, rowQuantiles, rowRanges, rowRanks,
#> rowSdDiffs, rowSds, rowSums2, rowTabulates, rowVarDiffs, rowVars,
#> rowWeightedMads, rowWeightedMeans, rowWeightedMedians,
#> rowWeightedSds, rowWeightedVars
```
``` r
mat <- matrix(0, nrow=10, ncol=6)
mat[sample(seq_len(60), 4)] <- 1:4
# Convert dense matrix to sparse matrix
sparse_mat <- as(mat, "dgCMatrix")
sparse_mat
#> 10 x 6 sparse Matrix of class "dgCMatrix"
#>
#> [1,] 4 . . . . .
#> [2,] . . . . . .
#> [3,] . . . . . .
#> [4,] 2 . . . . .
#> [5,] . . . . . .
#> [6,] . . . . . .
#> [7,] . . . . . 1
#> [8,] . . . . . .
#> [9,] . . . 3 . .
#> [10,] . . . . . .
```
The package provides an interface to quickly do common operations on the
rows or columns. For example calculate the variance:
``` r
apply(mat, 2, var)
#> [1] 1.822222 0.000000 0.000000 0.900000 0.000000 0.100000
matrixStats::colVars(mat)
#> [1] 1.822222 0.000000 0.000000 0.900000 0.000000 0.100000
sparseMatrixStats::colVars(sparse_mat)
#> [1] 1.822222 0.000000 0.000000 0.900000 0.000000 0.100000
```
On this small example data, all methods are basically equally fast, but
if we have a much larger dataset, the optimizations for the sparse data
start to show.
I generate a dataset with 10,000 rows and 50 columns that is 99% empty
``` r
big_mat <- matrix(0, nrow=1e4, ncol=50)
big_mat[sample(seq_len(1e4 * 50), 5000)] <- rnorm(5000)
# Convert dense matrix to sparse matrix
big_sparse_mat <- as(big_mat, "dgCMatrix")
```
I use the `bench` package to benchmark the performance difference:
``` r
bench::mark(
sparseMatrixStats=sparseMatrixStats::colVars(big_sparse_mat),
matrixStats=matrixStats::colVars(big_mat),
apply=apply(big_mat, 2, var)
)
#> # A tibble: 3 x 6
#> expression min median `itr/sec` mem_alloc `gc/sec`
#> <bch:expr> <bch:tm> <bch:tm> <dbl> <bch:byt> <dbl>
#> 1 sparseMatrixStats 37.3µs 42.71µs 20836. 2.93KB 14.6
#> 2 matrixStats 1.48ms 1.65ms 584. 156.8KB 2.03
#> 3 apply 10.61ms 11.18ms 88.9 9.54MB 48.2
```
As you can see `sparseMatrixStats` is ca. 35 times fast than
`matrixStats`, which in turn is 7 times faster than the `apply()`
version.
# API
The package now supports all functions from the `matrixStats` API for
column sparse matrices (`dgCMatrix`). And thanks to the
[`MatrixGenerics`](https://bioconductor.org/packages/MatrixGenerics/) it
can be easily integrated along-side
[`matrixStats`](https://cran.r-project.org/package=matrixStats) and
[`DelayedMatrixStats`](https://bioconductor.org/packages/DelayedMatrixStats/).
Note that the `rowXXX()` functions are called by transposing the input
and calling the corresponding `colXXX()` function. Special optimized
implementations are available for `rowSums2()`, `rowMeans2()`, and
`rowVars()`.
| Method | matrixStats | sparseMatrixStats | Notes |
| :------------------- | :---------- | :---------------- | :--------------------------------------------------------------------------------------- |
| colAlls() | ✔ | ✔ | |
| colAnyMissings() | ✔ | ❌ | Not implemented because it is deprecated in favor of `colAnyNAs()` |
| colAnyNAs() | ✔ | ✔ | |
| colAnys() | ✔ | ✔ | |
| colAvgsPerRowSet() | ✔ | ✔ | |
| colCollapse() | ✔ | ✔ | |
| colCounts() | ✔ | ✔ | |
| colCummaxs() | ✔ | ✔ | |
| colCummins() | ✔ | ✔ | |
| colCumprods() | ✔ | ✔ | |
| colCumsums() | ✔ | ✔ | |
| colDiffs() | ✔ | ✔ | |
| colIQRDiffs() | ✔ | ✔ | |
| colIQRs() | ✔ | ✔ | |
| colLogSumExps() | ✔ | ✔ | |
| colMadDiffs() | ✔ | ✔ | |
| colMads() | ✔ | ✔ | |
| colMaxs() | ✔ | ✔ | |
| colMeans2() | ✔ | ✔ | |
| colMedians() | ✔ | ✔ | |
| colMins() | ✔ | ✔ | |
| colOrderStats() | ✔ | ✔ | |
| colProds() | ✔ | ✔ | |
| colQuantiles() | ✔ | ✔ | |
| colRanges() | ✔ | ✔ | |
| colRanks() | ✔ | ✔ | |
| colSdDiffs() | ✔ | ✔ | |
| colSds() | ✔ | ✔ | |
| colsum() | ✔ | ❌ | Base R function |
| colSums2() | ✔ | ✔ | |
| colTabulates() | ✔ | ✔ | |
| colVarDiffs() | ✔ | ✔ | |
| colVars() | ✔ | ✔ | |
| colWeightedMads() | ✔ | ✔ | Sparse version behaves slightly differently, because it always uses `interpolate=FALSE`. |
| colWeightedMeans() | ✔ | ✔ | |
| colWeightedMedians() | ✔ | ✔ | Only equivalent if `interpolate=FALSE` |
| colWeightedSds() | ✔ | ✔ | |
| colWeightedVars() | ✔ | ✔ | |
| rowAlls() | ✔ | ✔ | |
| rowAnyMissings() | ✔ | ❌ | Not implemented because it is deprecated in favor of `rowAnyNAs()` |
| rowAnyNAs() | ✔ | ✔ | |
| rowAnys() | ✔ | ✔ | |
| rowAvgsPerColSet() | ✔ | ✔ | |
| rowCollapse() | ✔ | ✔ | |
| rowCounts() | ✔ | ✔ | |
| rowCummaxs() | ✔ | ✔ | |
| rowCummins() | ✔ | ✔ | |
| rowCumprods() | ✔ | ✔ | |
| rowCumsums() | ✔ | ✔ | |
| rowDiffs() | ✔ | ✔ | |
| rowIQRDiffs() | ✔ | ✔ | |
| rowIQRs() | ✔ | ✔ | |
| rowLogSumExps() | ✔ | ✔ | |
| rowMadDiffs() | ✔ | ✔ | |
| rowMads() | ✔ | ✔ | |
| rowMaxs() | ✔ | ✔ | |
| rowMeans2() | ✔ | ✔ | |
| rowMedians() | ✔ | ✔ | |
| rowMins() | ✔ | ✔ | |
| rowOrderStats() | ✔ | ✔ | |
| rowProds() | ✔ | ✔ | |
| rowQuantiles() | ✔ | ✔ | |
| rowRanges() | ✔ | ✔ | |
| rowRanks() | ✔ | ✔ | |
| rowSdDiffs() | ✔ | ✔ | |
| rowSds() | ✔ | ✔ | |
| rowsum() | ✔ | ❌ | Base R function |
| rowSums2() | ✔ | ✔ | |
| rowTabulates() | ✔ | ✔ | |
| rowVarDiffs() | ✔ | ✔ | |
| rowVars() | ✔ | ✔ | |
| rowWeightedMads() | ✔ | ✔ | Sparse version behaves slightly differently, because it always uses `interpolate=FALSE`. |
| rowWeightedMeans() | ✔ | ✔ | |
| rowWeightedMedians() | ✔ | ✔ | Only equivalent if `interpolate=FALSE` |
| rowWeightedSds() | ✔ | ✔ | |
| rowWeightedVars() | ✔ | ✔ | |
# Installation Problems
`sparseMatrixStats` uses features from C++14 and as the standard is more
than 6 years old, I thought this wouldn’t cause problems. In most
circumstances this is true, but there are reoccuring reports, that the
installation fails for some people and that is of course annoying. The
typical error message is:
Error: C++14 standard requested but CXX14 is not defined
The main reason that the installation fails is that the compiler is too
old. Sufficient support for C++14 came in
- `clang` version 3.4
- `gcc` version 4.9
Accordingly, you must have a compiler available that is at least that
new. If you run on the command line
``` bash
$ gcc --version
```
and it says 4.8, you will have to install a newer compiler. At the end
of the section, I have collected a few tips to install an appropriate
version on different distributions.
If you have recent version of `gcc` (\>=4.9) or `clang` (\>= 3.4)
installed, but you still see the error message
Error: C++14 standard requested but CXX14 is not defined
the problem is that R doesn’t yet know about it.
The solution is to either create a `~/.R/Makevars` file and define
CXX14 = g++
CXX14FLAGS = -g -O2 $(LTO)
CXX14PICFLAGS = -fpic
CXX14STD = -std=gnu++14
or simply call
``` r
withr::with_makevars(
new = c(CXX14 = "g++", CXX14FLAGS = "-g -O2 $(LTO)",
CXX14PICFLAGS = "-fpic", CXX14STD = "-std=gnu++14"),
code = {
BiocManager::install("sparseMatrixStats")
})
```
### Update Compiler
#### CentOS / Scientic Linux / RHEL
One of the main culprits causing trouble is CentOS 7. It is popular in
scientific computing and is still supported until 2024. It does,
however, by default come with a very old version of `gcc` (4.8.5).
To install a more recent compiler, we can use
[devtoolset](https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/).
First, we enable the Software Collection Tools and then install for
example `gcc` version 7:
``` bash
$ yum install centos-release-scl
$ yum install devtoolset-7-gcc*
```
We can now either activate the new compiler for an R session
``` bash
$ scl enable devtoolset-7 R
```
and then call
``` r
withr::with_makevars(
new = c(CXX14 = "g++", CXX14FLAGS = "-g -O2 $(LTO)",
CXX14PICFLAGS = "-fpic", CXX14STD = "-std=gnu++14"),
code = {
BiocManager::install("sparseMatrixStats")
})
```
or we refer to the full path of the newly installed g++ from a standard
R session
``` r
withr::with_makevars(
new = c(CXX14 = "/opt/rh/devtoolset-7/root/usr/bin/g++",
CXX14FLAGS = "-g -O2 $(LTO)", CXX14PICFLAGS = "-fpic",
CXX14STD = "-std=gnu++14"),
code = {
BiocManager::install("sparseMatrixStats")
})
```
Note, that our shenanigans are only necessary once, when we install
`sparseMatrixStats`. After the successful installation of the package,
we can use R as usual.
#### Debian
All Debian releases later than Jessie (i.e. Stretch, Buster, Bullseye)
are recent enough and should install sparseMatrixStats without problems.
I was able to install `sparseMatrixStats` on Debian Jessie (which comes
with `gcc` version 4.9.2) by providing the necessary Makefile arguments
``` r
withr::with_makevars(
new = c(CXX14 = "g++",
CXX14FLAGS = "-g -O2 $(LTO)", CXX14PICFLAGS = "-fpic",
CXX14STD = "-std=gnu++14"),
code = {
BiocManager::install("sparseMatrixStats")
})
```
Debian Wheezy comes with `gcc` 4.7, which does not support C++14. On the
other hand, the last R release that was backported to Wheezy is 3.2.5
(see information on
[CRAN](https://cloud.r-project.org/bin/linux/debian/#debian-wheezy-oldoldoldstable)).
Thus, if you are still on Wheezy, I would encourage you to update your
OS.
#### Ubuntu
Since 16.04, Ubuntu comes with a recent enough compiler.
Ubuntu 14.04 comes with `gcc` 4.8.5, but updating to `gcc-5` is easy:
``` bash
$ sudo add-apt-repository ppa:ubuntu-toolchain-r/test
$ sudo apt-get update
$ sudo apt-get install gcc-5 g++-5
```
After that, you can install `sparseMatrixStats` with a custom Makevars
variables that refer to the new compiler
``` r
withr::with_makevars(
new = c(CXX14 = "g++-5",
CXX14FLAGS = "-g -O2 $(LTO)", CXX14PICFLAGS = "-fpic",
CXX14STD = "-std=gnu++14"),
code = {
BiocManager::install("sparseMatrixStats")
})
```
#### MacOS
No trouble reported so far. Just do:
``` r
BiocManager::install("sparseMatrixStats")
```
#### Windows
It is important that you have
[RTools40](https://cran.r-project.org/bin/windows/Rtools/) installed.
After that, you shouldn’t have any troubles installing
`sparseMatrixStats` directly from Bioconductor:
``` r
BiocManager::install("sparseMatrixStats")
```
### But I still have a problems
1. *Please* make sure to carefully read the full problem section.
2. Make sure that you are using at least R 4.0.0.
3. Make sure your compiler is new enough to support C++14 (ie. `gcc`
\>= 4.9 and `clang` \>= 3.4)
If your problems nonetheless persist, please file an
[issue](https://github.com/const-ae/sparseMatrixStats/issues/) including
the following information:
- Operating system with exact version (e.g. ‘Linux Ubuntu 18.04’)
- Compiler and compiler version (e.g. ‘gcc version 7.2.5’)
- The output of `sessionInfo()`
- Information if you have a `~/.R/Makevars` file and what it contains
- The exact call that you use to install `sparseMatrixStats` including
the full error message
|
