---
title: "EDASeq: Exploratory Data Analysis and Normalization for RNA-Seq"
author: "Davide Risso"
date: "Last modified: May 17, 2018; Compiled: `r format(Sys.time(), '%B %d, %Y')`"
bibliography: biblio.bib
output:
  BiocStyle::html_document:
    toc: true
vignette: >
  %\VignetteEncoding{UTF-8}
---

<!--
%\VignetteEngine{knitr::rmarkdown}
%\VignetteIndexEntry{EDASeq Vignette}
-->

# Introduction

In this document, we show how to conduct Exploratory Data Analysis
(EDA) and normalization for a typical RNA-Seq experiment using the
package `EDASeq`.

One can think of EDA for RNA-Seq as a two-step process: "read-level"
EDA helps in discovering lanes with low sequencing depths, quality
issues, and unusual nucleotide frequencies, while ``gene-level'' EDA
can capture mislabeled lanes, issues with distributional assumptions
(e.g., over-dispersion), and GC-content bias.

The package also implements both "within-lane" and "between-lane"
normalization procedures, to account, respectively, for within-lane
gene-specific (and possibly lane-specific) effects on read counts
(e.g., related to gene length or GC-content) and for between-lane
distributional differences in read counts (e.g., sequencing depths).

To illustrate the functionality of the `EDASeq` package, we
make use of the _Saccharomyces cerevisiae_ RNA-Seq data from
[@lee2008novel]. Briefly, a wild-type strain and three mutant
strains were sequenced using the Solexa 1G Genome Analyzer.  For each
strain, there are four technical replicate lanes from the same library
preparation. The reads were aligned using `Bowtie`
[@langmead2009ultrafast], with unique mapping and allowing up to
two mismatches.

The `leeBamViews` package provides a subset of the aligned
reads in BAM format. In particular, only the reads mapped between
bases 800,000 and 900,000 of chromosome XIII are considered. We use
these reads to illustrate read-level EDA.

The `yeastRNASeq` package contains gene-level read counts for
four lanes: two replicates of the wild-type strain ("wt") and
two replicates of one of the mutant strains ("mut").  We use
these data to illustrate gene-level EDA.

```{r setup, echo=FALSE}
library(knitr)
opts_chunk$set(cache=FALSE, message=FALSE, echo=TRUE, results="markup")
options(width=60)
```

```{r data}
library(EDASeq)
library(yeastRNASeq)
library(leeBamViews)
```

# Reading in unaligned and aligned read data {#secRead}

## Unaligned reads

Unaligned (unmapped) reads stored in FASTQ format may be managed via
the class `FastqFileList` imported from `ShortRead`.
Information related to the libraries sequenced in each lane can be
stored in the `elementMetadata` slot of the
`FastqFileList` object.

```{r import-raw}
files <- list.files(file.path(system.file(package = "yeastRNASeq"),
                              "reads"), pattern = "fastq", full.names = TRUE)
names(files) <- gsub("\\.fastq.*", "", basename(files))
met <- DataFrame(conditions=c(rep("mut",2), rep("wt",2)),
                 row.names=names(files))
fastq <- FastqFileList(files)
elementMetadata(fastq) <- met
fastq
```

## Aligned reads

The package can deal with aligned (mapped) reads in BAM format, using
the class `BamFileList` from `Rsamtools`.  Again, the
`elementMetadata` slot can be used to store lane-level sample
information.

```{r import-aligned}
files <- list.files(file.path(system.file(package = "leeBamViews"), "bam"),
                    pattern = "bam$", full.names = TRUE)
names(files) <- gsub("\\.bam", "", basename(files))

gt <- gsub(".*/", "", files)
gt <- gsub("_.*", "", gt)
lane <- gsub(".*(.)$", "\\1", gt)
geno <- gsub(".$", "", gt)

pd <- DataFrame(geno=geno, lane=lane,
                row.names=paste(geno,lane,sep="."))

bfs <- BamFileList(files)
elementMetadata(bfs) <- pd
bfs
```


# Read-level EDA


## Numbers of unaligned and aligned reads

One important check for quality control is to look at the total number
of reads produced in each lane, the number and the percentage of reads mapped to a
reference genome. A low total
number of reads might be a symptom of low quality of the input RNA,
while a low mapping percentage might indicate poor quality of the
reads (low complexity), problems with the reference genome, or
mislabeled lanes.

```{r plot-total}
colors <- c(rep(rgb(1,0,0,alpha=0.7),2),
            rep(rgb(0,0,1,alpha=0.7),2),
            rep(rgb(0,1,0,alpha=0.7),2),
            rep(rgb(0,1,1,alpha=0.7),2))
barplot(bfs,las=2,col=colors)
```

The figure, produced using the `barplot` method for the
`BamFileList` class, displays the number of mapped reads for
the subset of the yeast dataset included in the package
`leeBamViews`.  Unfortunately, `leeBamViews` does
not provide unaligned reads, but barplots of the total number of reads
can be obtained using the `barplot` method for the
`FastqFileList` class. Analogously, one can plot the percentage
of mapped reads with the `plot` method with signature
`c(x="BamFileList", y="FastqFileList")`. See the manual pages for
details.

## Read quality scores

As an additional quality check, one can plot the mean per-base (i.e.,
per-cycle) quality of the unmapped or mapped reads in every lane.

```{r plot-total-2}
plotQuality(bfs,col=colors,lty=1)
legend("topright",unique(elementMetadata(bfs)[,1]), fill=unique(colors))
```

## Individual lane summaries

If one is interested in looking more thoroughly at one lane, it is
possible to display the per-base distribution of quality scores for
each lane and the number of mapped reads
stratified by chromosome or strand. As expected,
all the reads are mapped to chromosome XIII.

```{r plot-qual}
plotQuality(bfs[[1]],cex.axis=.8)
barplot(bfs[[1]],las=2)
```

## Read nucleotide distributions

A potential source of bias is related to the sequence composition of
the reads. The function `plotNtFrequency` plots the
per-base nucleotide frequencies for all the reads in a given
lane.

```{r plot-nt}
plotNtFrequency(bfs[[1]])
```


# Gene-level EDA

Examining statistics and quality metrics at a read level can help in
discovering problematic libraries or systematic biases in one or more
lanes. Nevertheless, some biases can be difficult to detect at this
scale and gene-level EDA is equally important.

## Classes and methods for gene-level counts

There are several Bioconductor packages for aggregating reads over
genes (or other genomic regions, such as, transcripts and exons) given
a particular genome annotation, e.g., `IRanges`,
`ShortRead`, `Genominator`, `Rsubread`. See
their respective vignettes for details.

Here, we consider this step done and load the object
`geneLevelData` from `yeastRNASeq`, which provides
gene-level counts for 2 wild-type and 2 mutant lanes from the yeast
dataset of `lee2008novel` (see the `Genominator`
vignette for an example on the same dataset).

```{r load-gene-level}
data(geneLevelData)
head(geneLevelData)
```

Since it is useful to explore biases related to length and GC-content,
the `EDASeq` package provides, for illustration purposes,
length and GC-content for _S. cerevisiae_ genes (based on SGD
annotation, version r64 [@sgd]).

Functionality for automated retrieval of gene length and GC-content
is introduced in the last section of the vignette.


```{r load-lgc}
data(yeastGC)
head(yeastGC)
data(yeastLength)
head(yeastLength)
```

First, we filter the non-expressed genes, i.e., we consider only the
genes with an average read count greater than 10 across the four lanes
and for which we have length and GC-content information.

```{r filter}
filter <- apply(geneLevelData,1,function(x) mean(x)>10)
table(filter)
common <- intersect(names(yeastGC),
                    rownames(geneLevelData[filter,]))
length(common)
```

This leaves us with `r length(common)` genes.

The `EDASeq` package provides the `SeqExpressionSet`
class to store gene counts, (lane-level) information on the sequenced
libraries, and (gene-level) feature information.  We use the data
frame `met` created in Section `secRead` for the
lane-level data.  As for the feature data, we use gene length and
GC-content.

```{r create-object}
feature <- data.frame(gc=yeastGC,length=yeastLength)
data <- newSeqExpressionSet(counts=as.matrix(geneLevelData[common,]),
                            featureData=feature[common,],
                            phenoData=data.frame(
                              conditions=factor(c(rep("mut",2),rep("wt",2))),
                              row.names=colnames(geneLevelData)))
data
```

Note that the row names of `counts` and `featureData`
must be the same; likewise for the row names of `phenoData`
and the column names of `counts`.  The expression values can be accessed with `counts`, the lane information with `pData`,
and the feature information with `fData`.

```{r show-data}
head(counts(data))
pData(data)
head(fData(data))
```

The `SeqExpressionSet` class has two additional slots:
`normalizedCounts` and `offset` (matrices of the same dimension as `counts`),
which may be used to store a matrix of normalized counts and of
normalization offsets, respectively, to be used for subsequent analyses (see Section
\ref{secDE} and the `edgeR` vignette for details on the
role of offsets). If not specified, the offset is initialized as a matrix
of zeros.

```{r show-offset}
head(offst(data))
```

## Between-lane distribution of gene-level counts

One of the main considerations when dealing with gene-level counts is
the difference in count distributions between lanes. The
`boxplot` method provides an easy way to produce boxplots of
the logarithms of the gene counts in each lane.

```{r boxplot-genelevel}
boxplot(data,col=colors[1:4])
```

The `MDPlot` method produces a mean-difference plot (MD-plot)
of read counts for two lanes.

```{r md-plot}
MDPlot(data,c(1,3))
```

## Over-dispersion

Although the Poisson distribution is a natural and simple way to model
count data, it has the limitation of assuming equality of the mean and
variance. For this reason, the negative binomial distribution has been
proposed as an alternative when the data show over-dispersion. The
function `meanVarPlot` can be used to check whether the
count data are over-dispersed (for the Poisson distribution, one would
expect the points in the following Figures to be evenly
scattered around the black line).

```{r plot-mean-var}
meanVarPlot(data[,1:2], log=TRUE, ylim=c(0,16))
meanVarPlot(data, log=TRUE, ylim=c(0,16))
```

Note that the mean-variance relationship should be examined within
replicate lanes only (i.e., conditional on variables expected to
contribute to differential expression). For the yeast dataset, it is
not surprising to see no evidence of over-dispersion for the two
mutant technical replicate lanes; likewise for
the two wild-type lanes. However, one expects over-dispersion in the
presence of biological variability, when considering at once all four mutant and wild-type lanes
[@anders2010differential,@bullard2010evaluation,@robinson2010edger].


## Gene-specific effects on read counts

Several authors have reported selection biases related to sequence
features such as gene length, GC-content, and mappability
[@bullard2010evaluation,@hansen2011removing,@oshlack2009transcript,@risso2011gc].

In the following figure, obtained using `biasPlot`, one can
see the dependence of gene-level counts on GC-content. The same plot
could be created for gene length or mappability instead of GC-content.

```{r plot-gc}
biasPlot(data, "gc", log=TRUE, ylim=c(1,5))
```

To show that GC-content dependence can bias differential expression
analysis, one can produce stratified boxplots of the log-fold-change
of read counts from two lanes using the `biasBoxplot` method.
Again, the same type of plots can be created
for gene length or mappability.

```{r boxplot-gc}
lfc <- log(counts(data)[,3]+0.1) - log(counts(data)[,1]+0.1)
biasBoxplot(lfc, fData(data)$gc)
```


# Normalization

Following [@risso2011gc], we consider two main types of effects
on gene-level counts: (1) within-lane gene-specific (and possibly
lane-specific) effects, e.g., related to gene length or GC-content,
and (2) effects related to between-lane distributional differences,
e.g., sequencing depth. Accordingly,
`withinLaneNormalization` and
`betweenLaneNormalization` adjust for the first and second
type of effects, respectively.  We recommend to normalize for
within-lane effects prior to between-lane normalization.

We implemented four within-lane normalization methods, namely: loess
robust local regression of read counts (log) on a gene feature such as
GC-content (`loess`), global-scaling between feature strata
using the median (`median`), global-scaling between feature
strata using the upper-quartile (`upper`), and full-quantile
normalization between feature strata (`full`). For a discussion
of these methods in context of GC-content normalization see
[@risso2011gc].

```{r normalization}
dataWithin <- withinLaneNormalization(data,"gc", which="full")
dataNorm <- betweenLaneNormalization(dataWithin, which="full")
```

Regarding between-lane normalization, the package implements three of
the methods introduced in [@bullard2010evaluation]:
global-scaling using the median (`median`), global-scaling using
the upper-quartile (`upper`), and full-quantile normalization
(`full`).

The next figure shows how after full-quantile within- and
between-lane normalization, the GC-content bias is reduced and the
distribution of the counts is the same in each lane.

```{r plot-gc-norm}
biasPlot(dataNorm, "gc", log=TRUE, ylim=c(1,5))
boxplot(dataNorm, col=colors)
```

## Offset 

Some authors have argued that it is better to leave the count data
unchanged to preserve their sampling properties and instead use an
offset for normalization purposes in the statistical model for read
counts
[@anders2010differential,@hansen2011removing,@robinson2010edger]. This
can be achieved easily using the argument `offset` in both
normalization functions.

```{r norm-offset}
dataOffset <- withinLaneNormalization(data,"gc",
                                      which="full",offset=TRUE)
dataOffset <- betweenLaneNormalization(dataOffset,
                                       which="full",offset=TRUE)
```

Note that the `dataOffset` object will have both normalized
counts and offset stored in their respective slots.


# Differential expression analysis

One of the main applications of RNA-Seq is differential expression
analysis.  The normalized counts (or the original counts and the
offset) obtained using the `EDASeq` package can be supplied
to packages such as `edgeR` [@robinson2010edger] or
`DESeq2` [@deseq2] to find differentially
expressed genes. This section should be considered only as an
illustration of the compatibility of the results of `EDASeq`
with two of the most widely used packages for differential expression;
we refer ther reader to the edgeR's user guide and to the DESeq2 vignettes for more details on the methods implemented there.

## edgeR

We can perform a differential expression analysis with
`edgeR` based on the original counts by passing an offset to
the generalized linear model.  See the `edgeR` vignette for details
about how to perform a differential expression analysis with more complex designs or more robust approaches.

```{r edger}
library(edgeR)
design <- model.matrix(~conditions, data=pData(dataOffset))

y <- DGEList(counts=counts(dataOffset),
             group=pData(dataOffset)$conditions)
y$offset <- -offst(dataOffset)
y <- estimateDisp(y, design)

fit <- glmFit(y, design)
lrt <- glmLRT(fit, coef=2)
topTags(lrt)
```

## DESeq2

We can perform the same differential expression analysis with
`DESeq2`.

```{r deseq}
library(DESeq2)
dds <- DESeqDataSetFromMatrix(countData = counts(dataOffset),
                              colData = pData(dataOffset),
                              design = ~ conditions)

normFactors <- exp(-1 * offst(dataOffset))
normFactors <- normFactors / exp(rowMeans(log(normFactors)))
normalizationFactors(dds) <- normFactors

dds <- DESeq(dds)
res <- results(dds)
res
```

# Definitions and conventions

## Rounding

After either within-lane or between-lane normalization, the expression
values are not counts
anymore. However, their distribution still shows some typical features
of counts distribution (e.g., the variance depends on the mean).
Hence, for most applications, it is useful to round the normalized
values to recover count-like values, which we refer to as "pseudo-counts".

By default, both
`withinLaneNormalization` and
`betweenLaneNormalization` round the normalized values to
the closest integer. This behavior can be changed by specifying
`round=FALSE`. This gives the user more flexibility and assures
that rounding approximations do not affect subsequent computations
(e.g., recovering the offset from the normalized counts).

## Zero counts

To avoid problems in the computation of logarithms (e.g. in log-fold-changes), we add a small positive constant (namely $0.1$) to the
counts. For instance, the log-fold-change between $y_1$ and $y_2$ is
defined as
\begin{equation*}
  \frac{\log(y_1 + 0.1)}{\log(y_2 + 0.1)}.
\end{equation*}

## Offset
We define an offset in the normalization as
\begin{equation*}
  o = \log(y_{norm} + 0.1) - \log(y_{raw} + 0.1),
\end{equation*}
where $y_{norm}$ and $y_{raw}$ are the normalized and raw counts, respectively.

One can easily recover the normalized data from the raw counts and
offset, as shown here:

```{r unrounded}
dataNorm <- betweenLaneNormalization(data, round=FALSE, offset=TRUE)

norm1 <- normCounts(dataNorm)
norm2 <- exp(log(counts(dataNorm) + 0.1 ) + offst(dataNorm)) - 0.1

head(norm1 - norm2)
```

Note that the small constant added in the definition of offset does
not matter when pseudo-counts are considered, i.e.,

```{r rounded}
head(round(normCounts(dataNorm)) - round(counts(dataNorm) * exp(offst(dataNorm))))
```

We defined the offset as the log-ratio between normalized and raw
counts. However, the `edgeR` functions expect as offset
argument the log-ratio between raw and normalized counts. One must use
`-offst(offsetData)` as the offset argument of `edgeR`.


# Retrieving gene length and GC-content

Two essential features the gene-level EDA normalizes for are gene length and
GC-content. As users might wish to automatically retrieve this information, we
provide the function `getGeneLengthAndGCContent`. Given selected
ENTREZ or ENSEMBL gene IDs and the organism under investigation, this can be
done either based on BioMart (default) or using BioC annotation utilities.

```{r getLengthAndGC, eval=FALSE}
getGeneLengthAndGCContent(id=c("ENSG00000012048", "ENSG00000139618"), org="hsa")
```

Accordingly, we can retrieve the precalculated yeast data that has been used
throughout the vignette via

```{r getLengthAndGC-full, eval=FALSE}
fData(data) <- getGeneLengthAndGCContent(featureNames(data),
                                              org="sacCer3", mode="org.db")
```

# SessionInfo

```{r sessionInfo}
sessionInfo()
```

# References