<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
  <!ENTITY drbdsetup_options SYSTEM "./drbdsetup-options.xml">
]>
<refentry id="re-drbdsetup">
  &drbdsetup_options;

  <refentryinfo>
    <date>17 January 2018</date>

    <productname>DRBD</productname>

    <productnumber>9.0.x</productnumber>
  </refentryinfo>

  <refmeta>
    <refentrytitle>drbdsetup</refentrytitle>

    <manvolnum>8</manvolnum>

    <refmiscinfo class="manual">System Administration</refmiscinfo>
  </refmeta>

  <refnamediv>
    <refname>drbdsetup</refname>

    <refpurpose>Configure the DRBD kernel module<indexterm significance="normal">
        <primary>drbdsetup</primary>
      </indexterm></refpurpose>
  </refnamediv>

  <refsynopsisdiv>
    <cmdsynopsis sepchar=" ">
      <command moreinfo="none">drbdsetup</command>
      <arg choice="plain" rep="norepeat">command</arg>
      <arg choice="req" rep="repeat">argument</arg>
      <arg choice="opt" rep="repeat">option</arg>
    </cmdsynopsis>
  </refsynopsisdiv>

  <refsect1>
    <title>Description</title>

    <para>The <command moreinfo="none">drbdsetup</command> utility serves to
      configure the DRBD kernel module and to show its current configuration.
      Users usually interact with the <command
	moreinfo="none">drbdadm</command> utility, which provides a more
      high-level interface to DRBD than <command
	moreinfo="none">drbdsetup</command>.  (See <option>drbdadm</option>'s
      <option>--dry-run</option> option to see how <option>drbdadm</option>
      uses <option>drbdsetup</option>.)</para>

    <para>Some option arguments have a default scale which applies when a plain
    number is specified (for example Kilo, or 1024 times the numeric value).
    Such default scales can be overridden by using a suffix (for example, M
    for Mega). The common suffixes K = 2^10 = 1024, M = 1024 K, and G = 1024
    M are supported.</para>
  </refsect1>

  <refsect1>
    <title>Commands</title>

    <variablelist>
      <varlistentry>
	<xi:include href="drbdsetup_attach.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Attach a lower-level device to an existing replicated device. -->
	<xi:include href="drbdsetup_disk-options.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Change the disk options of an attached lower-level device. -->

        <listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>disk</secondary>
	  </indexterm>

	  <para>The <option>attach</option> command attaches a lower-level
	    device to an existing replicated device.  The
	    <option>disk-options</option> command changes the disk options of
	    an attached lower-level device.  In either case, the replicated
	    device must have been created with <command
	    moreinfo="none">drbdsetup new-minor</command>.</para>

	    <para>Both commands refer to the replicated device by its
	    <replaceable>minor</replaceable> number.
	    <replaceable>lower_dev</replaceable> is the name of the lower-level
	    device.  <replaceable>meta_data_dev</replaceable> is the name of
	    the device containing the metadata, and may be the same as
	    <replaceable>lower_dev</replaceable>.
	    <replaceable>meta_data_index</replaceable> is either a numeric
	    metadata index, or the keyword <option>internal</option> for
	    internal metadata, or the keyword <option>flexible</option> for
	    variable-size external metadata. Available
	    options:</para>


	  <variablelist>
	    <pick_drbdsetup_option name="al-extents"/>
	    <pick_drbdsetup_option name="al-updates"/>
	    <pick_drbdsetup_option name="disk-barrier"/>
	    <pick_drbdsetup_option name="disk-timeout"/>
	    <pick_drbdsetup_option name="md-flushes"/>
	    <pick_drbdsetup_option name="on-io-error"/>
	    <pick_drbdsetup_option name="read-balancing"/>
	    <pick_drbdsetup_option name="resync-after"/>
	    <pick_drbdsetup_option name="size"/> <!-- only for attach, not for disk-options? -->
	    <pick_drbdsetup_option name="discard-zeroes-if-aligned"/>
	    <pick_drbdsetup_option name="rs-discard-granularity"/>
          </variablelist>
        </listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_peer-device-options.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Remember the current size of a lower-level device. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>peer-device-options</secondary>
	  </indexterm>

	  <para>These are options that affect the <replaceable>peer</replaceable>'s device.
	  </para>


	  <variablelist>
	    <pick_drbdsetup_option name="c-delay-target"/>
	    <pick_drbdsetup_option name="c-fill-target"/>
	    <pick_drbdsetup_option name="c-max-rate"/>
	    <pick_drbdsetup_option name="c-min-rate"/>
	    <pick_drbdsetup_option name="c-plan-ahead"/>
	    <pick_drbdsetup_option name="resync-rate"/>
	 </variablelist>

	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_check-resize.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Remember the current size of a lower-level device. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>check-resize</secondary>
	  </indexterm>

	  <para>Remember the current size of the lower-level device of the
	  specified replicated device.  Used by drbdadm.  The size information
	  is stored in file
	  /var/lib/drbd/drbd-minor-<replaceable>minor</replaceable>.lkbd.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_new-peer.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Create a connection/peer within a resource. -->
	<xi:include href="drbdsetup_net-options.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Change the network options of a connection. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>net</secondary>
	  </indexterm>

	  <para>The <option>new-peer</option> command creates a connection within
          a <replaceable>resource</replaceable>. The resource
	  must have been created with <command moreinfo="none">drbdsetup
	  new-resource</command>.  The <option>net-options</option> command
	  changes the network options of an existing connection. Before a
	  connection can be activated with the <option>connect</option>
	  command, at least one path need to added with the <option>new-path</option>
	  command. Available options:</para>

	  <variablelist>

	    <pick_drbdsetup_option name="after-sb-0pri"/>
	    <pick_drbdsetup_option name="after-sb-1pri"/>
	    <pick_drbdsetup_option name="after-sb-2pri"/>
	    <pick_drbdsetup_option name="allow-two-primaries"/>
	    <pick_drbdsetup_option name="always-asbp"/>
	    <pick_drbdsetup_option name="connect-int"/>
	    <pick_drbdsetup_option name="cram-hmac-alg"/>
	    <pick_drbdsetup_option name="csums-alg"/>
	    <pick_drbdsetup_option name="csums-after-crash-only"/>
	    <pick_drbdsetup_option name="data-integrity-alg"/>
	    <pick_drbdsetup_option name="fencing"/>
	    <pick_drbdsetup_option name="ko-count"/>
	    <pick_drbdsetup_option name="max-buffers"/>
	    <pick_drbdsetup_option name="max-epoch-size"/>
	    <pick_drbdsetup_option name="on-congestion"/>
	    <pick_drbdsetup_option name="ping-int"/>
	    <pick_drbdsetup_option name="ping-timeout"/>
	    <pick_drbdsetup_option name="socket-check-timeout"/>
	    <pick_drbdsetup_option name="protocol"/>
	    <pick_drbdsetup_option name="rcvbuf-size"/>
	    <pick_drbdsetup_option name="rr-conflict"/>
	    <pick_drbdsetup_option name="shared-secret"/>
	    <pick_drbdsetup_option name="sndbuf-size"/>
	    <pick_drbdsetup_option name="tcp-cork"/>
	    <pick_drbdsetup_option name="timeout"/>
	    <pick_drbdsetup_option name="use-rle"/>
	    <pick_drbdsetup_option name="verify-alg"/>
	  </variablelist>
	</listitem>
      </varlistentry>


      <varlistentry>
	<xi:include href="drbdsetup_new-path.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Create a path to a peer (within a connection). -->
	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>net</secondary>
	  </indexterm>

	  <para>The <option>new-path</option> command creates a path within
          a <replaceable>connection</replaceable>. The connection
	  must have been created with <command moreinfo="none">drbdsetup
	  new-peer</command>. <replaceable>Local_addr</replaceable> and
	  <replaceable>remote_addr</replaceable> refer to the local and remote
	  protocol, network address, and port in the format
	  <optional><replaceable>address-family</replaceable>:</optional><replaceable>address</replaceable><optional>:<replaceable>port</replaceable></optional>.
	  The address families <option>ipv4</option>,
	  <option>ipv6</option>, <option>ssocks</option> (Dolphin Interconnect
	  Solutions' "super sockets"), <option>sdp</option> (Infiniband Sockets
	  Direct Protocol), and <option>sci</option> are supported
	  (<option>sci</option> is an alias for <option>ssocks</option>).  If
	  no address family is specified, <option>ipv4</option> is assumed. For
	  all address families except <option>ipv6</option>, the
	  <replaceable>address</replaceable> uses IPv4 address notation (for
	  example, 1.2.3.4). For <option>ipv6</option>, the address is enclosed
	  in brackets and uses IPv6 address notation (for example,
	  [fd01:2345:6789:abcd::1]).  The <replaceable>port</replaceable>
	  defaults to 7788.</para>

	</listitem>
      </varlistentry>


      <varlistentry>
	<xi:include href="drbdsetup_connect.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Activate a connection. -->
	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>net</secondary>
	  </indexterm>

	  <para>The <option>connect</option> command activates a connection.
	  That means that the DRBD driver will bind and listen on all local
	  addresses of the connection-'s paths. It will begin to try to establish
	  one or more paths of the connection.  Available options: </para>

	  <variablelist>
	    <pick_drbdsetup_option name="tentative"/>
	    <pick_drbdsetup_option name="discard-my-data"/>
	  </variablelist>

	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_del-peer.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Remove a connection/peer within a resource. -->
	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>net</secondary>
	  </indexterm>

	  <para>The <option>del-peer</option> command removes a connection from
          a <replaceable>resource</replaceable>.</para>

	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_del-path.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Remove a connection/peer within a resource. -->
	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>net</secondary>
	  </indexterm>

	  <para>The <option>del-path</option> command removes a path from
          a <replaceable>connection</replaceable>. Please note that it fails if
	  the path is necessary to keep a connected connection in tact. In order
	  to remove all paths, disconnect the connection first.</para>

	</listitem>
      </varlistentry>


      <varlistentry>
	<xi:include href="drbdsetup_cstate.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Show the current state of a connection. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>cstate</secondary>
	  </indexterm>

	  <para>Show the current state of a connection.  The connection is
	  identified by the node-id of the peer; see the <command
	  moreinfo="none">drbdsetup connect</command> command.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_del-minor.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Remove a replicated device. -->

	<listitem>
	  <para>Remove a replicated device.  No lower-level device may be
	  attached; see <command moreinfo="none">drbdsetup detach</command>.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_del-resource.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Remove a resource. -->

	<listitem>
	  <para>Remove a resource.  All volumes and connections must be removed
	  first (<command moreinfo="none">drbdsetup del-minor</command>,
	  <command moreinfo="none">drbdsetup disconnect</command>).
	  Alternatively, <command moreinfo="none">drbdsetup down</command> can
	  be used to remove a resource together with all its volumes and
	  connections.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_detach.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Detach the lower-level device of a replicated device. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>detach</secondary>
	  </indexterm>
	  <para>Detach the lower-level device of a replicated device.  Available options:
	  <variablelist>
	    <varlistentry>
	      <term><option>--force</option></term>
	      <listitem>
		<para>Force the detach and return immediately.  This puts the
		lower-level device into failed state until all pending I/O
		has completed, and then detaches the device.  Any I/O not yet
		submitted to the lower-level device (for example, because I/O
		on the device was suspended) is assumed to have failed.</para>
	      </listitem>
	    </varlistentry>
	  </variablelist>
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_disconnect.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Disconnect a resource from a peer host. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>disconnect</secondary>
	  </indexterm>

	  <para>Remove a connection to a peer host.  The connection is
	  identified by the node-id of the peer; see the <command
	  moreinfo="none">drbdsetup connect</command> command.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_down.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Take a resource down. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>down</secondary>
	  </indexterm>

	  <para>Take a resource down by removing all volumes, connections, and
	    the resource itself.
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_dstate.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Show the current disk state of a lower-level device. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>dstate</secondary>
	  </indexterm>

	  <para>Show the current disk state of a lower-level device.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_events2.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Show the current state and all state changes of all resources. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>events2</secondary>
	  </indexterm>

	  <para>Show the current state of all configured DRBD objects, followed
	    by all changes to the state.</para>

	  <para>The output format is meant to be human as well as machine
	    readable. The line starts with a word that indicates the kind of event:
	    <option>exists</option> for an existing object;
	    <option>create</option>, <option>destroy</option>, and
	    <option>change</option> if an object is created, destroyed, or
	    changed; or <option>call</option> or <option>response</option> if
	    an event handler is called or it returns.  The second word indicates
	    the object the event applies to: <option>resource</option>,
	    <option>device</option>, <option>connection</option>,
	    <option>peer-device</option>, <option>helper</option>, or a dash
	    (<option>-</option>) to indicate that the current state has been
	    dumped completely.</para>

	  <para>The remaining words identify the object and describe the state
	    that he object is in.  Available options:

	    <variablelist>
	      <varlistentry>
		<term><option>--now</option></term>

		<listitem>
		  <para>Terminate after reporting the current state.  The
		    default is to continuously listen and report state
		    changes.</para>
		</listitem>
	      </varlistentry>

	      <varlistentry>
		<term><option>--statistics</option></term>

		<listitem>
		  <para>Include statistics in the output.</para>
		</listitem>
	      </varlistentry>
	    </variablelist>
	  </para>
	</listitem>
    </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_get-gi.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Show the data generation identifiers for a device on a particular connection. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>get-gi</secondary>
	  </indexterm>

	  <para>Show the data generation identifiers for a device on a
	  particular connection.  The device is identified by its volume
	  number.  The connection is identified by its endpoints; see the
	  <command moreinfo="none">drbdsetup connect</command> command.</para>

	  <para>The output consists of the current UUID, bitmap UUID, and the
	  first two history UUIDS, folowed by a set of flags.  The current UUID
	  and history UUIDs are device specific; the bitmap UUID and flags are
	  peer device specific.  This command only shows the first two history
	  UUIDs.  Internally, DRBD maintains one history UUID for each possible
	  peer device.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_invalidate.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Replace the local data of a resource with that of a peer. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>invalidate</secondary>
	  </indexterm>

	  <para>Replace the local data of a device with that of a peer.  All
	    the local data will be marked out-of-sync, and a resync with the
	    specified peer device will be initialted.</para>
	  <!-- "This command will fail if the device is not part of a connected device pair." -->
	  <!-- FIXME: What if the peer is not connected? -->
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_invalidate-remote.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Replace a peer's data of a resource with the local data. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>invalidate-remote</secondary>
	  </indexterm>

	  <para>Replace a peer device's data of a resource with the local data.
	    The peer device's data will be marked out-of-sync, and a resync
	    from the local node to the specified peer will be initiated.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_new-current-uuid.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Generate a new current UUID. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>new-current-uuid</secondary>
	  </indexterm>

	  <!-- FIXME: The description does not really explain what is going on
	  and why; this needs to be completely analyzed and revised.  Also, we
	  should have separate commands for creating a new current uuid and for
	  clearing unused bitmap slots. -->

	  <para>Generate a new current UUID and rotates all other UUID values. This has at least two
	  use cases, namely to skip the initial sync, and to reduce network bandwidth when starting in
	  a single node configuration and then later (re-)integrating a remote site.</para>

	  <para>Available option: <variablelist>
	      <varlistentry>
		<term><option>--clear-bitmap</option></term>

		<listitem>
		  <para>Clears the sync bitmap in addition to generating a new current UUID.</para>
		</listitem>
	      </varlistentry>
	    </variablelist></para>

	  <para>This can be used to skip the initial sync, if you want to start from scratch. This
	  use-case does only work on "Just Created" meta data. Necessary steps: <orderedlist
	      continuation="restarts" inheritnum="ignore" numeration="arabic">
	      <listitem>
		<simpara>On <emphasis>both</emphasis> nodes, initialize meta data and configure the
		device.</simpara>

		<simpara><command moreinfo="none">drbdadm create-md --force
		<replaceable>res</replaceable></command></simpara>
	      </listitem>

	      <listitem>
		<simpara>They need to do the initial handshake, so they know their sizes.</simpara>

		<simpara><command moreinfo="none">drbdadm up
		<replaceable>res</replaceable></command></simpara>
	      </listitem>

	      <listitem>
		<simpara>They are now Connected Secondary/Secondary Inconsistent/Inconsistent.
		Generate a new current-uuid and clear the dirty bitmap.</simpara>

		<simpara><command moreinfo="none">drbdadm --clear-bitmap new-current-uuid
		<replaceable>res</replaceable></command></simpara>
	      </listitem>

	      <listitem>
		<simpara>They are now Connected Secondary/Secondary UpToDate/UpToDate. Make one side
		primary and create a file system.</simpara>

		<simpara><command moreinfo="none">drbdadm primary
		<replaceable>res</replaceable></command></simpara>

		<simpara><command moreinfo="none">mkfs -t <replaceable>fs-type</replaceable> $(drbdadm
		sh-dev <replaceable>res</replaceable>)</command></simpara>
		<!-- FIXME: sh-dev can probably return more than one device; we can no longer use it like this. -->
	      </listitem>
	    </orderedlist></para>

	  <para>One obvious side-effect is that the replica is full of old garbage (unless you made
	  them identical using other means), so any online-verify is expected to find any number of
	  out-of-sync blocks.</para>

	  <para><emphasis>You must not use this on pre-existing data!</emphasis> Even though it may
	  appear to work at first glance, once you switch to the other node, your data is toast, as it
	  never got replicated. So <emphasis>do not leave out the mkfs</emphasis> (or
	  equivalent).</para>

	  <para>This can also be used to shorten the initial resync of a cluster where the second node
	  is added after the first node is gone into production, by means of disk shipping. This
	  use-case works on disconnected devices only, the device may be in primary or secondary
	  role.</para>

	  <para>The necessary steps on the current active server are: <orderedlist
	      continuation="restarts" inheritnum="ignore" numeration="arabic">
	      <listitem>
		<simpara><command moreinfo="none">drbdsetup new-current-uuid --clear-bitmap <replaceable>minor</replaceable>
		</command></simpara>
	      </listitem>

	      <listitem>
		<simpara>Take the copy of the current active server. E.g. by pulling a disk out of the
		RAID1 controller, or by copying with dd. You need to copy the actual data, and the
		meta data.</simpara>
	      </listitem>

	      <listitem>
		<simpara><command moreinfo="none">drbdsetup new-current-uuid <replaceable>minor</replaceable>
		</command></simpara>
	      </listitem>
	    </orderedlist> Now add the disk to the new secondary node, and join it to the cluster. You
	  will get a resync of that parts that were changed since the first call to <command
	  moreinfo="none">drbdsetup</command> in step 1.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_new-minor.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Create a new replicated device within a resource. -->

	<listitem>
	  <para>Create a new replicated device within a resource.  The command
	    creates a block device inode for the replicated device (by default,
	    /dev/drbd<replaceable>minor</replaceable>).  The
	    <replaceable>volume</replaceable> number identifies the device
	    within the <replaceable>resource</replaceable>.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_new-resource.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Create a new resource. -->
	<xi:include href="drbdsetup_resource-options.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Change the resource options of an existing resource. -->

	<listitem>
	  <para>The <option>new-resource</option> command creates a new
	    resource.  The <option>resource-options</option> command changes
	    the resource options of an existing resource.  Available options:

	  <variablelist>
	    <pick_drbdsetup_option name="auto-promote"/>
	    <pick_drbdsetup_option name="cpu-mask"/>
	    <pick_drbdsetup_option name="on-no-data-accessible"/>
	    <pick_drbdsetup_option name="peer-ack-window"/>
	    <pick_drbdsetup_option name="peer-ack-delay"/>
	    <pick_drbdsetup_option name="quorum"/>
	    <pick_drbdsetup_option name="quorum-minimum-redundancy"/>
	    <pick_drbdsetup_option name="on-no-quorum"/>
	  </variablelist>
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_outdate.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Mark the data on a lower-level device as outdated. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>outdate</secondary>
	  </indexterm>

	  <para>Mark the data on a lower-level device as outdated.  This
	    is used for fencing, and prevents the resource the device is part
	    of from becoming primary in the future.  See the
	    <option>--fencing</option> disk option.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_pause-sync.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Stop resynchronizing between a local and a peer device. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>pause-sync</secondary>
	  </indexterm>

	  <para>Stop resynchronizing between a local and a peer device by
	    setting the local pause flag.  The resync can only resume if the
	    pause flags on both sides of a connection are cleared.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_primary.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Change the role of a node in a resource to primary. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>primary</secondary>
	  </indexterm>

	  <para>Change the role of a node in a resource to primary.  This
	  allows the replicated devices in this resource to be mounted or
	  opened for writing.  Available options:</para>

	  <variablelist>
	    <varlistentry>
	      <term><option>--overwrite-data-of-peer</option></term>

	      <listitem>
		<para>This option is an alias for the <option>--force</option> option.</para>
	      </listitem>
	    </varlistentry>
	  </variablelist>

	  <variablelist>
	    <varlistentry>
	      <term><option>--force</option></term>

	      <listitem>
		<para>Force the resource to become primary even if some devices
		  are not guaranteed to have up-to-date data.  This option is
		  used to turn one of the nodes in a newly created cluster into
		  the primary node, or when manually recovering from a
		  disaster.</para>
		<para>Note that this can lead to split-brain scenarios.  Also,
		  when forcefully turning an inconsistent device into an
		  up-to-date device, it is highly recommended to use any
		  integrity checks available (such as a filesystem check) to
		  make sure that the device can at least be used without
		  crashing the system.</para>
	      </listitem>
	    </varlistentry>
	  </variablelist>

	  <para>Note that DRBD usually only allows one node in a cluster to be
	    in primary role at any time; this allows DRBD to coordinate access
	    to the devices in a resource across nodes.  The
	    <option>--allow-two-primaries</option> network option changes this;
	    in that case, a mechanism outside of DRBD needs to coordinate
	    device access.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_resize.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Reexamine the lower-level device sizes to resize a replicated device. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>resize</secondary>
	  </indexterm>

	  <para>Reexamine the size of the lower-level devices of a replicated
	  device on all nodes.  This command is called after the lower-level
	  devices on all nodes have been grown to adjust the size of the
	  replicated device.  Available options:

	    <variablelist>
	      <varlistentry>
		<term><option>--assume-peer-has-space</option></term>

		<listitem>
		  <para>Resize the device even if some of the peer devices are
		  not connected at the moment.  DRBD will try to resize the peer
		  devices when they next connect.  It will refuse to connect to a
		  peer device which is too small.</para>
		</listitem>
	      </varlistentry>

	      <varlistentry>
		<term><option>--assume-clean</option></term>
		<listitem>
		  <para>Do not resynchronize the added disk space; instead,
		  assume that it is identical on all nodes.  This option can be
		  used when the disk space is uninitialized and differences do
		  not matter, or when it is known to be identical on all nodes.
		  See the <command moreinfo="none">drbdsetup verify</command>
		  command.</para>
		</listitem>
	      </varlistentry>

	      <varlistentry>
		<term><option>--size <replaceable>val</replaceable></option></term>
		<listitem>
		  <para>This option can be used to online shrink the usable
		  size of a drbd device. It's the users responsibility to
		  make sure that a file system on the device is not
		  truncated by that operation.</para>
		</listitem>
	      </varlistentry>

	      <varlistentry>
		<term><option>--al-stripes <replaceable>val</replaceable></option>
		<option>--al-stripes <replaceable>val</replaceable></option></term>
		<listitem>
		  <para>These options may be used to change the layout of
		  the activity log online. In case of internal meta data
		  this may invovle shrinking the user visible size at the
		  same time (unsing the <option>--size</option>) or
		  increasing the avalable space on the backing
		  devices.</para>
		</listitem>
	      </varlistentry>

	    </variablelist>
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_resume-io.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Resume I/O on a replicated device. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>resume-io</secondary>
	  </indexterm>

	  <para>Resume I/O on a replicated device.  See the
	  <option>--fencing</option> net option.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_resume-sync.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Resume the resynchronization between a local and a peer device. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>resume-sync</secondary>
	  </indexterm>

	  <para>Allow resynchronization to resume by clearing the local sync pause flag.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_role.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Show the current role of a resource. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>role</secondary>
	  </indexterm>

	  <para>Show the current role of a resource.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_secondary.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Change the role of a node in a resource to secondary. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>secondary</secondary>
	  </indexterm>

	  <para>Change the role of a node in a resource to secondary.  This
	  command fails if the replicated device is in use.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_show.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Show the current configuration of a resource, or of all resources. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>show</secondary>
	  </indexterm>

	  <para>Show the current configuration of a resource, or of all
	  resources.  Available options:

	    <variablelist>
	      <varlistentry>
		<term><option>--show-defaults</option></term>

		<listitem>
		  <para>Show all configuration parameters, even the ones with
		  default values.  Normally, parameters with default values are
		  not shown.</para>
		</listitem>
	      </varlistentry>
	    </variablelist>
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_show-gi.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Show data generation identifiers for a device and peer device, with explanations. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>show-gi</secondary>
	  </indexterm>

	  <para>Show the data generation identifiers for a device on a
	  particular connection.  In addition, explain the output.  The output
	  otherwise is the same as in the <command moreinfo="none">drbdsetup
	  get-gi</command> command.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<term><command moreinfo="none">drbdsetup</command><arg choice="plain" rep="norepeat">state</arg></term>

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>state</secondary>
	  </indexterm>

	  <para>This is an alias for <command moreinfo="none">drbdsetup
	  role</command>. Deprecated.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_status.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Show the status of a resource, or of all resources. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>status</secondary>
	  </indexterm>

	  <para>Show the status of a resource, or of all resources.  The
	  output consists of one paragraph for each configured resource.  Each
	  paragraph contains one line for each resource, followed by one line
	  for each device, and one line for each connection.  The device and
	  connection lines are indented.  The connection lines are followed by
	  one line for each peer device; these lines are indented against the
	  connection line.</para>

	  <para>Long lines are wrapped around at terminal width, and indented
	  to indicate how the lines belongs together.  Available options:

	    <variablelist>
	      <varlistentry>
		<term><option>--verbose</option></term>

		<listitem>
		  <para>Include more information in the output even when it is
		  likely redundant or irrelevant.</para>
		</listitem>
	      </varlistentry>

	      <varlistentry>
		<term><option>--statistics</option></term>

		<listitem>
		  <para>Include data transfer statistics in the output.</para>
		</listitem>
	      </varlistentry>

	      <varlistentry>
		<term>
		  <option>--color=<group choice="req" rep="norepeat">
		      <arg choice="plain" rep="norepeat">always</arg>
		      <arg choice="plain" rep="norepeat">auto</arg>
		      <arg choice="plain" rep="norepeat">never</arg>
		    </group>
		  </option>
		</term>

		<listitem>
		  <para>Colorize the output.  With
		    <option>--color=auto</option>, <option>drbdsetup</option>
		    emits color codes only when standard output is connected to
		    a terminal.
		  </para>
		</listitem>
	      </varlistentry>
	    </variablelist>
	  </para>

	  <para>For example, the non-verbose output for a resource with only
	  one connection and only one volume could look like this:
	      <programlisting format="linespecific">
drbd0 role:Primary
  disk:UpToDate
  host2.example.com role:Secondary
    disk:UpToDate
	      </programlisting>
	    </para>

	    <para>With the <option>--verbose</option> option, the same resource
	    could be reported as:
	      <programlisting format="linespecific">
drbd0 node-id:1 role:Primary suspended:no
  volume:0 minor:1 disk:UpToDate blocked:no
  host2.example.com local:ipv4:192.168.123.4:7788
      peer:ipv4:192.168.123.2:7788 node-id:0 connection:WFReportParams
      role:Secondary congested:no
    volume:0 replication:Connected disk:UpToDate resync-suspended:no
	      </programlisting>
	    </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_suspend-io.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Suspend I/O on a replicated device. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>suspend-io</secondary>
	  </indexterm>

	  <para>Suspend I/O on a replicated device.  It is not usually
	  necessary to use this command.</para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_verify.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Verify the data on a lower-level device against a peer device. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>verify</secondary>
	  </indexterm>

	  <para>Start online verification, change which part of the device will
	  be verified, or stop online verification.  The command requires the
	  specified peer to be connected.</para>

	  <para>Online verification compares each disk block on the local and
	  peer node.  Blocks which differ between the nodes are marked
	  as out-of-sync, but they are <emphasis>not</emphasis> automatically
	  brought back into sync.  To bring them into sync, the resource must
	  be disconnected and reconnected.  Progress can be monitored in the
	  output of <command moreinfo="none">drbdsetup status
	    --statistics</command>.  Available options:</para>

	  <variablelist>
	    <varlistentry>
	      <term><option>--start <replaceable>position</replaceable></option></term>

	      <listitem>
		<para>Define where online verification should start.  This
		parameter is ignored if online verification is already in
		progress.  If the start parameter is not specified, online
		verification will continue where it was interrupted (if the
		connection to the peer was lost while verifying), after the
		previous stop sector (if the previous online verification has
		finished), or at the beginning of the device (if the end of the
		device was reached, or online verify has not run
		before).</para>

		<para>The position on disk is specified in disk sectors (512
		bytes) by default.</para>
	      </listitem>
	    </varlistentry>
	    <varlistentry>
	      <term><option>--stop <replaceable>position</replaceable></option></term>
	      <listitem>
		<para>Define where online verification should stop.  If online
		verification is already in progress, the stop position of the
		active online verification process is changed.  Use this to
		stop online verification.</para>

		<para>The position on disk is specified in disk sectors (512
		bytes) by default.</para>
	      </listitem>
	    </varlistentry>
	  </variablelist>

	  <para>Also see the notes on data integrity in the <citerefentry>
	  <refentrytitle>drbd.conf</refentrytitle> <manvolnum>5</manvolnum>
	  </citerefentry> manual page.</para>
	</listitem>
      </varlistentry>

      <varlistentry>

	<xi:include href="drbdsetup_wait-connect-volume.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Wait until a device on a peer is visible. -->
	<xi:include href="drbdsetup_wait-connect-connection.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Wait until all peer volumes of connection are visible. -->
	<xi:include href="drbdsetup_wait-connect-resource.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Wait until all connections are establised. -->

	<xi:include href="drbdsetup_wait-sync-volume.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Wait until a device on a peer is up to date. -->
	<xi:include href="drbdsetup_wait-sync-connection.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Wait until resync finished on all volumes of a connection. -->
	<xi:include href="drbdsetup_wait-sync-resource.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Wait until resync finished on all volumes. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>wait-connect-volume</secondary>
	    <secondary>wait-connect-connection</secondary>
	    <secondary>wait-connect-resource</secondary>
	  </indexterm>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>wait-sync-volume</secondary>
	    <secondary>wait-sync-connection</secondary>
	    <secondary>wait-sync-resource</secondary>
	  </indexterm>

	  <para>The <option>wait-connect-*</option> commands waits until a device
	  on a peer is visible.  The <option>wait-sync-*</option> commands waits
	  until a device on a peer is up to date.  Available options for both
	  commands:

	    <variablelist>
	      <pick_drbdsetup_option name="degr-wfc-timeout"/>
	      <pick_drbdsetup_option name="outdated-wfc-timeout"/>
	      <pick_drbdsetup_option name="wait-after-sb"/>
	      <pick_drbdsetup_option name="wfc-timeout"/>
	    </variablelist>
	  </para>
	</listitem>
      </varlistentry>

      <varlistentry>
	<xi:include href="drbdsetup_forget-peer.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
	<!-- Completely remove any reference to a unconnected peer from meta-data. -->

	<listitem>
	  <indexterm significance="normal">
	    <primary>drbdsetup</primary>

	    <secondary>forget-peer</secondary>
	  </indexterm>

	  <para>The <option>forget-peer</option> command removes all traces of
	  a peer node from the meta-data. It frees a bitmap slot in the meta-data
	  and make it avalable for futher bitmap slot allocation in case a
	  so-far never seen node connects.</para>
	  <para>The connection must be taken down before this command may be used.
	  In case the peer re-connects at a later point a bit-map based resync
	  will be turned into a full-sync.</para>
	</listitem>
      </varlistentry>

    </variablelist>
  </refsect1>

  <refsect1>
    <title>Examples</title>

    <para>Please see the <ulink
	url="http://www.drbd.org/users-guide/"><citetitle>DRBD User's Guide</citetitle></ulink>
      for examples.</para>

  </refsect1>

  <refsect1>
    <title>Version</title>

    <simpara>This document was revised for version 9.0.0 of the DRBD distribution.</simpara>
  </refsect1>

  <refsect1>
    <title>Author</title>

    <simpara>Written by Philipp Reisner <email>philipp.reisner@linbit.com</email> and Lars
    Ellenberg <email>lars.ellenberg@linbit.com</email>.</simpara>
  </refsect1>

  <refsect1>
    <title>Reporting Bugs</title>

    <simpara>Report bugs to <email>drbd-user@lists.linbit.com</email>.</simpara>
  </refsect1>

  <refsect1>
    <title>Copyright</title>

    <simpara>Copyright 2001-2018 LINBIT Information Technologies, Philipp Reisner, Lars Ellenberg.
    This is free software; see the source for copying conditions. There is NO warranty; not even
    for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.</simpara>
  </refsect1>

  <refsect1>
    <title>See Also</title>

    <para><citerefentry>
        <refentrytitle>drbd.conf</refentrytitle>

        <manvolnum>5</manvolnum>
      </citerefentry>, <citerefentry>
        <refentrytitle>drbd</refentrytitle>

        <manvolnum>8</manvolnum>
      </citerefentry>, <citerefentry>
        <refentrytitle>drbdadm</refentrytitle>

        <manvolnum>8</manvolnum>
      </citerefentry>,
      <ulink url="http://www.drbd.org/users-guide/"><citetitle>DRBD User's Guide</citetitle></ulink>,
       <ulink url="http://www.drbd.org/"><citetitle>DRBD Web Site</citetitle></ulink></para>

  </refsect1>
</refentry>