% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/test_predictions.R
\name{test_predictions}
\alias{test_predictions}
\alias{hypothesis_test}
\alias{test_predictions.default}
\alias{test_predictions.ggeffects}
\title{(Pairwise) comparisons between predictions (marginal effects)}
\usage{
test_predictions(object, ...)

hypothesis_test(object, ...)

\method{test_predictions}{default}(
  object,
  terms = NULL,
  by = NULL,
  test = "pairwise",
  test_args = NULL,
  equivalence = NULL,
  scale = "response",
  p_adjust = NULL,
  df = NULL,
  ci_level = 0.95,
  margin = "mean_reference",
  condition = NULL,
  collapse_levels = FALSE,
  engine = "marginaleffects",
  verbose = TRUE,
  ...
)

\method{test_predictions}{ggeffects}(
  object,
  by = NULL,
  test = "pairwise",
  equivalence = NULL,
  scale = "response",
  p_adjust = NULL,
  df = NULL,
  collapse_levels = FALSE,
  engine = "marginaleffects",
  verbose = TRUE,
  ...
)
}
\arguments{
\item{object}{A fitted model object, or an object of class \code{ggeffects}. If
\code{object} is of class \code{ggeffects}, arguments \code{terms}, \code{margin} and \code{ci_level}
are taken from the \code{ggeffects} object and don't need to be specified.}

\item{...}{Arguments passed down to \code{\link[=data_grid]{data_grid()}} when creating the reference
grid and to \code{\link[marginaleffects:predictions]{marginaleffects::predictions()}} resp. \code{\link[marginaleffects:slopes]{marginaleffects::slopes()}}.
For instance, arguments \code{type} or \code{transform} can be used to back-transform
comparisons and contrasts to different scales. \code{vcov} can be used to
calculate heteroscedasticity-consistent standard errors for contrasts.

To define a heteroscedasticity-consistent variance-covariance matrix, you can
either use the same arguments as for \code{predict_response()} etc., namely \code{vcov}
and \code{vcov_args}. These are then transformed into a matrix and passed down to
the \code{vcov} argument in \strong{marginaleffects}. Or you directly use the \code{vcov}
argument. See \code{?marginaleffects::slopes} for further details.}

\item{terms}{If \code{object} is an object of class \code{ggeffects}, the same \code{terms}
argument is used as for the predictions, i.e. \code{terms} can be ignored. Else,
if \code{object} is a model object, \code{terms} must be a character vector with the
names of the focal terms from \code{object}, for which contrasts or comparisons
should be displayed. At least one term is required, maximum length is three
terms. If the first focal term is numeric, contrasts or comparisons for the
\emph{slopes} of this numeric predictor are computed (possibly grouped by the
levels of further categorical focal predictors).}

\item{by}{Character vector specifying the names of predictors to condition on.
Hypothesis test is then carried out for focal terms by each level of \code{by}
variables. This is useful especially for interaction terms, where we want
to test the interaction within "groups". \code{by} is only relevant for
categorical predictors.}

\item{test}{Hypothesis to test, defined as character string, formula, or
data frame. Can be one of:
\itemize{
\item String:
\itemize{
\item \code{"pairwise"} (default), to test pairwise comparisons.
\item \code{"trend"} (or \code{"slope"}) to test for the linear trend/slope of (usually)
continuous predictors. These options are just aliases for setting
\code{trend = NULL}.
\item \code{"contrast"} to test simple contrasts (i.e. each level is tested against
the average over \emph{all} levels).
\item \code{"exclude"} to test simple contrasts (i.e. each level is tested against
the average over \emph{all other} levels, excluding the contrast that is being
tested).
\item \code{"interaction"} to test interaction contrasts (difference-in-difference
contrasts). More flexible interaction contrasts can be calcualted using
the \code{test_args} argument.
\item \code{"consecutive"} to test contrasts between consecutive levels of a predictor.
\item \code{"polynomial"} to test orthogonal polynomial contrasts, assuming
equally-spaced factor levels.
}
\item String equation:

A character string with a custom hypothesis, e.g. \code{"b2 = b1"}. This would
test if the second level of a predictor is different from the first level.
Custom hypotheses are very flexible. It is also possible to test interaction
contrasts (difference-in-difference contrasts) with custom hypotheses, e.g.
\code{"(b2 - b1) = (b4 - b3)"}. See also section \emph{Introduction into contrasts
and pairwise comparisons}.
\item Formula:

A formula, where the left-hand side indicates the type of comparison and
the right-hand side which pairs to compare. Optionally, grouping variables
can be specified after a vertical bar. See also 'Examples'.
\itemize{
\item For the left-hand side, comparisons can be \code{difference} or \code{ratio}.
\item For the right-hand side, pairs can be \code{reference}, \code{sequential}, or
\code{meandev}. For \code{reference}, all factor levels are compared to the
reference level. \code{sequential} compares consecutive levels of a predictor.
\code{meandev} compares each factor level against the "average" factor level.
\item If a variable is specified after \code{|}, comparisons will be grouped by
that variable.
}
\item A data frame with custom contrasts. See 'Examples'.
\item \code{NULL}, in which case simple contrasts are computed.
}

Technical details about the packages used as back-end to calculate contrasts
and pairwise comparisons are provided in the section \emph{Packages used as back-end
to calculate contrasts and pairwise comparisons} below.}

\item{test_args}{Optional arguments passed to \code{test}, typically provided as
named list. Only applies to those options that use the \strong{emmeans} package
as backend, e.g. if \code{test = "interaction"}, \code{test_args} will be passed to
\code{emmeans::contrast(interaction = test_args)}. For other \emph{emmeans} options
(like \code{"contrast"}, \code{"exclude"}, \code{"consecutive"} and so on), \code{test_args} will
be passed to the \code{option} argument in \code{emmeans::contrast()}.}

\item{equivalence}{ROPE's lower and higher bounds. Should be \code{"default"} or
a vector of length two (e.g., \code{c(-0.1, 0.1)}). If \code{"default"},
\code{\link[bayestestR:rope_range]{bayestestR::rope_range()}} is used. Instead of using the \code{equivalence}
argument, it is also possible to call the \code{equivalence_test()} method
directly. This requires the \strong{parameters} package to be loaded. When
using \code{equivalence_test()}, two more columns with information about the
ROPE coverage and decision on H0 are added. Furthermore, it is possible
to \code{plot()} the results from \code{equivalence_test()}. See
\code{\link[bayestestR:equivalence_test]{bayestestR::equivalence_test()}} resp. \code{\link[parameters:equivalence_test.lm]{parameters::equivalence_test.lm()}}
for details.}

\item{scale}{Character string, indicating the scale on which the contrasts
or comparisons are represented. Can be one of:
\itemize{
\item \code{"response"} (default), which would return contrasts on the response
scale (e.g. for logistic regression, as probabilities);
\item \code{"link"} to return contrasts on scale of the linear predictors
(e.g. for logistic regression, as log-odds);
\item \code{"probability"} (or \code{"probs"}) returns contrasts on the probability scale,
which is required for some model classes, like \code{MASS::polr()};
\item \code{"oddsratios"} to return contrasts on the odds ratio scale (only applies
to logistic regression models);
\item \code{"irr"} to return contrasts on the odds ratio scale (only applies to
count models);
\item or a transformation function like \code{"exp"} or \code{"log"}, to return transformed
(exponentiated respectively logarithmic) contrasts; note that these
transformations are applied to the \emph{response scale}.
}

\strong{Note:} If the \code{scale} argument is not supported by the provided \code{object},
it is automatically changed to a supported scale-type (a message is printed
when \code{verbose = TRUE}).}

\item{p_adjust}{Character vector, if not \code{NULL}, indicates the method to
adjust p-values. See \code{\link[stats:p.adjust]{stats::p.adjust()}} or \code{\link[stats:p.adjust]{stats::p.adjust.methods}}
for details. Further possible adjustment methods are \code{"tukey"} or \code{"sidak"},
and for \code{johnson_neyman()}, \code{"fdr"} (or \code{"bh"}) and \code{"esarey"} (or its
short-cut \code{"es"}) are available options. Some caution is necessary when
adjusting p-value for multiple comparisons. See also section \emph{P-value adjustment}
below.}

\item{df}{Degrees of freedom that will be used to compute the p-values and
confidence intervals. If \code{NULL}, degrees of freedom will be extracted from
the model using \code{\link[insight:get_df]{insight::get_df()}} with \code{type = "wald"}.}

\item{ci_level}{Numeric, the level of the confidence intervals. If \code{object}
is an object of class \code{ggeffects}, the same \code{ci_level} argument is used as
for the predictions, i.e. \code{ci_level} can be ignored.}

\item{margin}{Character string, indicates the method how to marginalize over
non-focal terms. See \code{\link[=predict_response]{predict_response()}} for details. If \code{object} is an
object of class \code{ggeffects}, the same \code{margin} argument is used as for the
predictions, i.e. \code{margin} can be ignored.}

\item{condition}{Named character vector, which indicates covariates that
should be held constant at specific values, for instance
\code{condition = c(covariate1 = 20, covariate2 = 5)}.}

\item{collapse_levels}{Logical, if \code{TRUE}, term labels that refer to identical
levels are no longer separated by "-", but instead collapsed into a unique
term label (e.g., \code{"level a-level a"} becomes \code{"level a"}). See 'Examples'.}

\item{engine}{Character string, indicates the package to use for computing
contrasts and comparisons. Usually, this argument can be ignored, unless you
want to explicitly use another package than \emph{marginaleffects} to calculate
contrasts and pairwise comparisons. \code{engine} can be either \code{"marginaleffects"}
(default) or \code{"emmeans"}. The latter is useful when the \strong{marginaleffects}
package is not available, or when the \strong{emmeans} package is preferred. Note
that using \strong{emmeans} as back-end is currently not as feature rich as the default
(\strong{marginaleffects}). Setting \code{engine = "emmeans"} provides some additional
test options: \code{"interaction"} to calculate interaction contrasts,
\code{"consecutive"} to calculate contrasts between consecutive levels of a
predictor, or a data frame with custom contrasts (see also \code{test}). There is
a third option as well, \code{engine = "ggeffects"}. However, this option offers
less features as the default engine, \code{"marginaleffects"}. It can be faster in
some cases, though, and works for comparing predicted random effects in mixed
models, or predicted probabilities of the zero-inflation component. If the
\strong{marginaleffects} package is not installed, the \strong{emmeans} package is used
automatically. If this package is not installed as well, \code{engine = "ggeffects"}
is used.}

\item{verbose}{Toggle messages and warnings.}
}
\value{
A data frame containing predictions (e.g. for \code{test = NULL}),
contrasts or pairwise comparisons of adjusted predictions or estimated
marginal means.
}
\description{
Function to test differences of adjusted predictions for
statistical significance. This is usually called contrasts or (pairwise)
comparisons, or "marginal effects". \code{hypothesis_test()} is an alias.
}
\section{Simple workflow for pairwise comparisons}{


A simple workflow includes calculating adjusted predictions and passing the
results directly to \code{test_predictions()}, e.g.:

\if{html}{\out{<div class="sourceCode">}}\preformatted{# 1. fit your model
model <- lm(mpg ~ hp + wt + am, data = mtcars)
# 2. calculate adjusted predictions
pr <- predict_response(model, "am")
pr
# 3. test pairwise comparisons
test_predictions(pr)
}\if{html}{\out{</div>}}

See also \href{https://strengejacke.github.io/ggeffects/articles/practical_glm_workflow.html}{this vignette}.
}

\section{Packages used as back-end to calculate contrasts and pairwise comparisons}{


The \code{test} argument is used to define which kind of contrast or comparison
should be calculated. The default is to use the \strong{marginaleffects} package.
Here are some technical details about the packages used as back-end. When
\code{test} is...
\itemize{
\item \code{"pairwise"} (default), pairwise comparisons are based on the \strong{marginaleffects}
package.
\item \code{"trend"} or \code{"slope"} also uses the \strong{marginaleffects} package.
\item \code{"contrast"} uses the \strong{emmeans} package, i.e. \code{emmeans::contrast(method = "eff")}
is called.
\item \code{"exclude"} relies on the \strong{emmeans} package, i.e. \code{emmeans::contrast(method = "del.eff")}
is called.
\item \code{"polynomial"} relies on the \strong{emmeans} package, i.e. \code{emmeans::contrast(method = "poly")}
is called.
\item \code{"interaction"} uses the \strong{emmeans} package, i.e. \code{emmeans::contrast(interaction = ...)}
is called.
\item \code{"consecutive"} also relies on the \strong{emmeans} package, i.e.
\code{emmeans::contrast(method = "consec")} is called.
\item a character string with a custom hypothesis, the \strong{marginaleffects}
package is used.
\item a data frame with custom contrasts, \strong{emmeans} is used again.
\item for formulas, the \strong{marginaleffects} package is used.
\item \code{NULL} calls functions from the \strong{marginaleffects} package with
\code{hypothesis = NULL}.
\item If all focal terms are only present as random effects in a mixed model,
or if predicted probabilities for the zero-inflation component of a model
should be tested, functions from the \strong{ggeffects} package are used. There
is an example for pairwise comparisons of random effects in
\href{https://strengejacke.github.io/ggeffects/articles/practical_intersectionality.html}{this vignette}.
}
}

\section{P-value adjustment for multiple comparisons}{


Note that p-value adjustment for methods supported by \code{p.adjust()} (see also
\code{p.adjust.methods}), each row is considered as one set of comparisons, no
matter which \code{test} was specified. That is, for instance, when \code{test_predictions()}
returns eight rows of predictions (when \code{test = NULL}), and \code{p_adjust = "bonferroni"},
the p-values are adjusted in the same way as if we had a test of pairwise
comparisons (\code{test = "pairwise"}) where eight rows of comparisons are
returned. For methods \code{"tukey"} or \code{"sidak"}, a rank adjustment is done
based on the number of combinations of levels from the focal predictors
in \code{terms}. Thus, the latter two methods may be useful for certain tests
only, in particular pairwise comparisons.

For \code{johnson_neyman()}, the only available adjustment methods are \code{"fdr"}
(or \code{"bh"}) (\emph{Benjamini & Hochberg (1995)}) and \code{"esarey"} (or \code{"es"})
(\emph{Esarey and Sumner 2017}). These usually return similar results. The major
difference is that \code{"fdr"} can be slightly faster and more stable in edge
cases, however, confidence intervals are not updated. Only the p-values are
adjusted. \code{"esarey"} is slower, but confidence intervals are updated as well.
}

\section{Global options to choose package for calculating comparisons}{


\code{ggeffects_test_engine} can be used as option to either use the \strong{marginaleffects}
package for computing contrasts and comparisons (default), or the \strong{emmeans}
package (e.g. \code{options(ggeffects_test_engine = "emmeans")}). The latter is
useful when the \strong{marginaleffects} package is not available, or when the
\strong{emmeans} package is preferred. You can also provide the engine directly, e.g.
\code{test_predictions(..., engine = "emmeans")}. Note that using \strong{emmeans} as
backend is currently not as feature rich as the default (\strong{marginaleffects}).

If \code{engine = "emmeans"}, the \code{test} argument can also be \code{"interaction"}
to calculate interaction contrasts (difference-in-difference contrasts),
\code{"consecutive"} to calculate contrasts between consecutive levels of a predictor,
or a data frame with custom contrasts. If \code{test} is one of the latter options,
and \code{engine} is not specified, the \code{engine} is automatically set to \code{"emmeans"}.
Additionally, the \code{test_args} argument can be used to specify further options
for those contrasts. See 'Examples' and documentation of \code{test_args}.

If the \strong{marginaleffects} package is not installed, the \strong{emmeans} package is
used automatically. If this package is not installed as well,
\code{engine = "ggeffects"} is used.
}

\section{Global Options to Customize Tables when Printing}{

The \code{verbose} argument can be used to display or silence messages and
warnings. Furthermore, \code{options()} can be used to set defaults for the
\code{print()} and \code{print_html()} method. The following options are available,
which can simply be run in the console:
\itemize{
\item \code{ggeffects_ci_brackets}: Define a character vector of length two, indicating
the opening and closing parentheses that encompass the confidence intervals
values, e.g. \code{options(ggeffects_ci_brackets = c("[", "]"))}.
\item \code{ggeffects_collapse_ci}: Logical, if \code{TRUE}, the columns with predicted
values (or contrasts) and confidence intervals are collapsed into one
column, e.g. \code{options(ggeffects_collapse_ci = TRUE)}.
\item \code{ggeffects_collapse_p}: Logical, if \code{TRUE}, the columns with predicted
values (or contrasts) and p-values are collapsed into one column, e.g.
\code{options(ggeffects_collapse_p = TRUE)}. Note that p-values are replaced
by asterisk-symbols (stars) or empty strings when \code{ggeffects_collapse_p = TRUE},
depending on the significance level.
\item \code{ggeffects_collapse_tables}: Logical, if \code{TRUE}, multiple tables for
subgroups are combined into one table. Only works when there is more than
one focal term, e.g. \code{options(ggeffects_collapse_tables = TRUE)}.
\item \code{ggeffects_output_format}: String, either \code{"text"}, \code{"markdown"} or \code{"html"}.
Defines the default output format from \code{predict_response()}. If \code{"html"}, a
formatted HTML table is created and printed to the view pane. \code{"markdown"}
creates a markdown-formatted table inside Rmarkdown documents, and prints
a text-format table to the console when used interactively. If \code{"text"} or
\code{NULL}, a formatted table is printed to the console, e.g.
\code{options(ggeffects_output_format = "html")}.
\item \code{ggeffects_html_engine}: String, either \code{"tt"} or \code{"gt"}. Defines the default
engine to use for printing HTML tables. If \code{"tt"}, the \emph{tinytable} package
is used, if \code{"gt"}, the \emph{gt} package is used, e.g.
\code{options(ggeffects_html_engine = "gt")}.
}

Use \verb{options(<option_name> = NULL)} to remove the option.
}

\examples{
\dontshow{if (all(insight::check_if_installed(c("parameters", "marginaleffects", "modelbased"), quietly = TRUE)) && interactive()) (if (getRversion() >= "3.4") withAutoprint else force)(\{ # examplesIf}
\donttest{
data(efc)
efc$c172code <- as.factor(efc$c172code)
efc$c161sex <- as.factor(efc$c161sex)
levels(efc$c161sex) <- c("male", "female")
m <- lm(barthtot ~ c12hour + neg_c_7 + c161sex + c172code, data = efc)

# direct computation of comparisons
test_predictions(m, "c172code")

# passing a `ggeffects` object
pred <- predict_response(m, "c172code")
test_predictions(pred)

# test for slope
test_predictions(m, "c12hour")

# interaction - contrasts by groups
m <- lm(barthtot ~ c12hour + c161sex * c172code + neg_c_7, data = efc)
test_predictions(m, c("c161sex", "c172code"), test = NULL)

# interaction - pairwise comparisons by groups
test_predictions(m, c("c161sex", "c172code"))

# equivalence testing
test_predictions(m, c("c161sex", "c172code"), equivalence = c(-2.96, 2.96))

# equivalence testing, using the parameters package
pr <- predict_response(m, c("c161sex", "c172code"))
parameters::equivalence_test(pr)

# interaction - collapse unique levels
test_predictions(m, c("c161sex", "c172code"), collapse_levels = TRUE)

# p-value adjustment
test_predictions(m, c("c161sex", "c172code"), p_adjust = "tukey")

# not all comparisons, only by specific group levels
test_predictions(m, "c172code", by = "c161sex")

# specific comparisons
test_predictions(m, c("c161sex", "c172code"), test = "b2 = b1")

# interaction - slope by groups
m <- lm(barthtot ~ c12hour + neg_c_7 * c172code + c161sex, data = efc)
test_predictions(m, c("neg_c_7", "c172code"))

# Interaction and consecutive contrasts -----------------
# -------------------------------------------------------
data(coffee_data, package = "ggeffects")
m <- lm(alertness ~ time * coffee + sex, data = coffee_data)

# consecutive contrasts
test_predictions(m, "time", by = "coffee", test = "consecutive")

# same as (using formula):
pr <- predict_response(m, c("time", "coffee"))
test_predictions(pr, test = difference ~ sequential | coffee)

# interaction contrasts - difference-in-difference comparisons
pr <- predict_response(m, c("time", "coffee"), margin = "marginalmeans")
test_predictions(pr, test = "interaction")

# Ratio contrasts ---------------------------------------
# -------------------------------------------------------
test_predictions(test = ratio ~ reference | coffee)

# Custom contrasts --------------------------------------
# -------------------------------------------------------
wakeup_time <- data.frame(
  "wakeup vs later" = c(-2, 1, 1) / 2, # make sure each "side" sums to (+/-)1!
  "start vs end of day" = c(-1, 0, 1)
)
test_predictions(m, "time", by = "coffee", test = wakeup_time)

# Example: marginal effects -----------------------------
# -------------------------------------------------------
data(iris)
m <- lm(Petal.Width ~ Petal.Length + Species, data = iris)

# we now want the marginal effects for "Species". We can calculate
# the marginal effect using the "marginaleffects" package
marginaleffects::avg_slopes(m, variables = "Species")

# finally, test_predictions() returns the same. while the previous results
# report the marginal effect compared to the reference level "setosa",
# test_predictions() returns the marginal effects for all pairwise comparisons
test_predictions(m, "Species")
}
\dontshow{\}) # examplesIf}
}
\references{
Esarey, J., & Sumner, J. L. (2017). Marginal effects in interaction models:
Determining and controlling the false positive rate. Comparative Political
Studies, 1–33. Advance online publication. doi: 10.1177/0010414017730080
}
\seealso{
There is also an \code{equivalence_test()} method in the \strong{parameters}
package (\code{\link[parameters:equivalence_test.lm]{parameters::equivalence_test.lm()}}), which can be used to
test contrasts or comparisons for practical equivalence. This method also
has a \code{plot()} method, hence it is possible to do something like:

\if{html}{\out{<div class="sourceCode">}}\preformatted{library(parameters)
predict_response(model, focal_terms) |>
  equivalence_test() |>
  plot()
}\if{html}{\out{</div>}}
}