=head1 NAME

libnbd - network block device (NBD) client library in userspace

=head1 SYNOPSIS

 #include <libnbd.h>

 struct nbd_handle *nbd;
 char buf[512];

 if ((nbd = nbd_create ()) == NULL ||
     nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1 ||
     nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1)
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }
 nbd_close (nbd);

=for paragraph

 cc prog.c -o prog -lnbd
or:
 cc prog.c -o prog `pkg-config libnbd --cflags --libs`

=head1 DESCRIPTION

Network Block Device (NBD) is a network protocol for accessing block
devices over the network.  Block devices are hard disks and things
that behave like hard disks such as disk images and virtual machines.

Libnbd is a client library for the NBD protocol which can access most
of the features of NBD while being simple to use and powerful.

This manual page gives an overview of libnbd, using C as an example,
but the library is available from other programming languages.

=over 4

=item L<nbd_create(3)>, L<nbd_pread(3)>, etc.

Each manual page covers one function from the C API in detail.  There
is a full list in section L</C API> below.

=item L<libnbd-ocaml(3)>

Using the API from OCaml.

=item L<libnbd-golang(3)>

Using the API from Go.

=item L<libnbd-rust(3)>

Using the API from Rust.

=item L<nbdsh(1)>

Using the NBD shell (nbdsh) for command line and Python scripting.

=back

=head1 HANDLES

To use the API at all you must first open a handle by calling
L<nbd_create(3)> (or its equivalent in other languages):

 struct nbd_handle *nbd;

 nbd = nbd_create ();

This creates and returns a handle, which is associated with
one connection to an NBD server, initially not connected.

Each handle is a complex state machine which can be in
states such as created, connected to a remote server, handshaking,
idle and ready to issue commands, or busy sending or receiving
commands.

Handles have a name used in debugging messages.  The name is normally
generated (C<nbd1>, C<nbd2> etc) but you can set a friendly name with
L<nbd_set_handle_name(3)>.  Also there is a private field in the
handle for use by the application, see L<nbd_set_private_data(3)>.

When you have finished with the handle you must call L<nbd_close(3)>
which closes the underlying socket (if necessary) and frees up all
associated resources.

=head1 SYNCHRONOUS VS ASYNCHRONOUS API

There are two levels of API available.  A simple high level
synchronous API lets you give the handle high level instructions like
“connect to the server”, “read a block”, “write a block”, etc.  Each
of these functions will run to completion, blocking the current thread
before returning.  A more complicated low level non-blocking
asynchronous API is also available where you can integrate with
L<poll(2)> or another main loop.

You can freely mix the two APIs on the same handle.  You can also call
APIs on a single handle from multiple threads.  Single API calls on
the handle are atomic — they either take a lock on the handle while
they run or are careful to access handle fields atomically.

Libnbd does B<not> create its own threads.

=head1 USING THE SYNCHRONOUS (“HIGH LEVEL”) API

This is the simplest way to use the API, with the possible drawback
that each libnbd function blocks until it is finished.

Create a handle and connect to the server:

 struct nbd_handle *nbd;

 nbd = nbd_create ();
 if (!nbd) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }
 if (nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }

Read the first sector (512 bytes) from the NBD export:

 char buf[512];

 if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }

Close the handle:

 nbd_close (nbd);

You can call the high level API from multiple threads, but each libnbd
API call takes a lock on the handle and so commands will not run in
parallel.

=head1 USING THE ASYNCHRONOUS (“LOW LEVEL”) API

The low level API is useful if you want to use libnbd in non-blocking
code; or if you want to issue commands in parallel from multiple
threads; or if you need more control especially over having multiple
commands in-flight on a single connection.

To use the low level API you will need to integrate with L<poll(2)> or
another “main loop” such as the GLib main event loop.

=head2 Issuing asynchronous commands

Use the C<nbd_aio_*> variants to issue commands asynchronously
(without waiting for the command to complete before returning).  For
example the asynchronous variant of L<nbd_pread(3)> is:

 int64_t cookie;

 cookie = nbd_aio_pread (nbd, buf, sizeof buf,
                         NBD_NULL_COMPLETION, 0);
 if (cookie == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }

There are several things to note here:

=over 4

=item *

This only starts the command.  The command is (usually) still in
flight when the call returns success, where you must rely on
subsequent API calls for learning the final command outcome and
trigger any remaining callbacks.  However, you must also be able to
handle the case where system load allows the state machine to advance
far enough to invoke callbacks before the asynchronous API returns.

=item *

A buffer (C<buf>) has been assigned to collect the result of the read,
but it is not guaranteed to be filled with data until the command has
completed (see examples below).  The buffer must not be freed until
the command has finished running.

=item *

You can issue multiple commands on the same handle at the same time.

=item *

A cookie is returned which identifies this command in subsequent
calls.  The cookie is unique (per libnbd handle) and E<ge> 1.

=item *

You may register a function which is called when the command
completes, see L</Completion callbacks> below.  In this case we have
specified a null completion callback.  If a completion callback is
specified, it will only be called if the asynchronous command was
successfully submitted (if the asynchronous API itself returns an
error, there is nothing further to be completed).

=back

=head2 Socket and direction

Each libnbd handle has an associated socket (once it has started
connecting).  You can read the file descriptor of the socket using:

 int fd = nbd_aio_get_fd (nbd);

The socket is non-blocking.  Between calls into libnbd it is in the
"would block" condition.  You can find out if libnbd is expecting to
read or write from the socket next by calling:

 int dir = nbd_aio_get_direction (nbd);

which returns one of C<LIBNBD_AIO_DIRECTION_READ>,
C<LIBNBD_AIO_DIRECTION_WRITE> or C<LIBNBD_AIO_DIRECTION_BOTH> (=
C<READ|WRITE>).  And so to set up the next call to L<poll(2)> or other
main loop you must translate this to C<POLLIN>, C<POLLOUT> or
C<POLLIN|POLLOUT> (or whatever mechanism your main loop uses).

=head2 Notifying libnbd when an event happens

When you detect (eg. using L<poll(2)>) that a read or write event has
happened on the socket, you must then tell libnbd about it.  You have
to check the direction I<again> (since it may have been changed by
another thread), and notify libnbd:

 int r = 0;

 dir = nbd_aio_get_direction (nbd);

 if ((dir & LIBNBD_AIO_DIRECTION_READ) &&
                 a_read_event_occurred ())
   r = nbd_aio_notify_read (nbd);
 else if ((dir & LIBNBD_AIO_DIRECTION_WRITE) &&
                 a_write_event_occurred ())
   r = nbd_aio_notify_write (nbd);

 if (r == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   // ...
 }

The notify calls move the state machine along, reading and writing
from the socket possibly multiple times, until the socket would block
again, at which point they return control to the caller.

=head2 Simple implementation with L<nbd_poll(3)>

In fact if you want to use L<poll(2)> on a single handle, a simple
implementation has already been written called L<nbd_poll(3)>.  It is
also useful to examine how this is implemented (F<lib/poll.c> in the
libnbd source code) because that will tell you how to integrate libnbd
with more complex main loops.

Some examples of using L<nbd_poll(3)> follow.

As with the high level API, it all starts by creating a handle:

 struct nbd_handle *nbd;

 nbd = nbd_create ();
 if (nbd == NULL) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }

To connect to the server asynchronously, we start the connection using
L<nbd_aio_connect(3)> and then enter our main loop to check for events
until the connection becomes ready:

 int fd;
 struct sockaddr_un addr;
 socklen_t len;

 /* some code to set up addr,
    then ... */
 if (nbd_aio_connect (nbd, &addr, len) == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }
 while (! nbd_aio_is_ready (nbd)) {
   if (nbd_poll (nbd, -1) == -1) {
     fprintf (stderr, "%s\n", nbd_get_error ());
     nbd_close (nbd);
     exit (EXIT_FAILURE);
   }
 }

To read data asynchronously, start an asynchronous read command, which
returns a 64 bit command cookie, and enter the main loop until the
command has completed:

 int64_t cookie;
 char buf[512];

 cookie = nbd_aio_pread (nbd, buf, sizeof buf, offset,
                         NBD_NULL_COMPLETION, 0);
 if (cookie == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }
 while (! nbd_aio_command_completed (nbd, cookie)) {
   if (nbd_poll (nbd, -1) == -1) {
     fprintf (stderr, "%s\n", nbd_get_error ());
     nbd_close (nbd);
     exit (EXIT_FAILURE);
   }
 }

For almost all high level synchronous calls (eg. L<nbd_pread(3)>) there
is a low level asynchronous equivalent (eg. L<nbd_aio_pread(3)>) for
starting a command.

=head2 glib2 integration

See
L<https://gitlab.com/nbdkit/libnbd/blob/master/examples/glib-main-loop.c>

=head2 libev integration

See
L<https://gitlab.com/nbdkit/libnbd/blob/master/examples/copy-libev.c>

=head1 ERROR HANDLING

When any API call returns an error (C<-1> or C<NULL> depending on the
API), an error message and sometimes an errno value are available.
You can retrieve the error message and/or errno of the most recently
failed call using L<nbd_get_error(3)> and L<nbd_get_errno(3)>.  For example:

 if (nbd_connect_tcp (nbd, "remote", "nbd") == -1) {
   fprintf (stderr,
            "failed to connect to remote server: %s (errno = %d)\n",
            nbd_get_error (), nbd_get_errno ());
 }

These functions use thread-local storage to return the most recent
error in the current thread.  This is why you don't need to pass the
handle to these calls.  They even work if L<nbd_create(3)> returns
C<NULL> when there is no handle at all.

For this reason you cannot call them from a different thread.  You
should call them immediately after the failed API call, from the same
thread.  Furthermore the error string returned by L<nbd_get_error(3)> is
only valid until the next libnbd API call in the current thread, so if
you need to keep the string you must copy it (eg. using L<strdup(3)>).

=head2 Errno

For some errors, a system call error number (see L<errno(3)>) is
available.  You can find the error number by calling
L<nbd_get_errno(3)>.  It works the same way as L<nbd_get_error(3)>
with respect to threads.

Even when a call returns an error, L<nbd_get_errno(3)> might return
C<0>.  This does I<not> mean there was no error.  It means no
additional errno information is available for this error.

The error number is often the raw error returned by a system call that
failed.

It can also be used to indicate special conditions.  The most common
cases are:

=over 4

=item C<EINVAL>

Invalid parameters or state for the current libnbd call.  (This can
also indicate that requests are not aligned to
L</Block size constraints>).

=item C<ENOTSUP>

The libnbd call is not available in this build of libnbd (eg. when
using a TLS API if the library was compiled without TLS support).

=item C<ENOMEM>

The library ran out of memory while performing some operation.

=item C<ERANGE>

A request is too large, for example if you try to read too many bytes
in a single L<nbd_pread(3)> call.

=item C<EFAULT>

A pointer parameter was C<NULL> when it should be non-NULL.
See the section below.

=back

=head2 Non-NULL parameters

Almost all libnbd functions when called from C take one or more
pointer parameters that must not be C<NULL>.  For example, the handle
parameter, strings and buffers should usually not be C<NULL>.

If a C<NULL> is passed as one of these parameters, libnbd attempts to
return an error with L<nbd_get_errno(3)> returning C<EFAULT>.

However it may cause other compiler-related warnings and even
undefined behaviour, so you should try to avoid this programming
mistake.

=head1 DEBUGGING MESSAGES

Libnbd can print lots of debugging messages, useful if you have a
problem with the library.  Either enable debugging after creating the
handle:

 nbd = nbd_create ();
 nbd_set_debug (nbd, true);

or set the C<LIBNBD_DEBUG=1> environment variable which will enable
debugging by default on all new handles.

Debugging messages are sent to stderr by default, but you can redirect
them to a logging system using L<nbd_set_debug_callback(3)>.

=head1 CONNECTING TO LOCAL OR REMOTE NBD SERVERS

There are several ways to connect to NBD servers, and you can even run
a server from libnbd.  Normally you would connect to a server which is
already running, over a local Unix domain socket or a remote TCP
connection.  The high level API calls are:

 nbd_connect_unix (nbd, "socket");
 nbd_connect_tcp (nbd, "localhost", "nbd");

For L<nbd_connect_tcp(3)> the third parameter is the port name or number,
which can either be a name from F</etc/services> or the port number as
a string (eg. C<"10809">).

=head2 Connecting to an NBD URI

libnbd supports the
L<NBD URI specification|https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md>.
The format of URIs is documented in L<nbd_connect_uri(3)>.

You can connect to a URI as in these examples (using the high level
API):

 nbd_connect_uri (nbd, "nbd://example.com/");

=for paragraph

 nbd_connect_uri (nbd, "nbds+unix:///export?socket=/tmp/nbd.sock");

This feature is implemented by calling other libnbd APIs to set up the
export name, TLS parameters, and finally connect over a Unix domain
socket or TCP.

URI support is an optional feature of the library, requiring libxml2
at compile time.  The L<nbd_connect_uri(3)> and
L<nbd_aio_connect_uri(3)> calls will raise an error (with
L<nbd_get_errno(3)> returning C<ENOTSUP>) if it was not built with
this feature, and you can also test for it explicitly using
L<nbd_supports_uri(3)>.

L<nbd_is_uri(3)> uses a heuristic to detect if the parameter could be
an NBD URI or something else (like a filename).  It can be useful for
program parameters that can be a URI or a file.

=head2 Connecting to a subprocess

Some NBD servers — notably L<nbdkit(1)> with the I<-s> parameter, and
L<nbd-server(1)> with the port parameter set to 0 — can also accept a
single NBD connection on stdin/stdout.  You can run these servers as a
subprocess of your main program using L<nbd_connect_command(3)>.  This
example creates a 1G writable RAM disk:

 char *argv[] = { "nbdkit", "-s", "--exit-with-parent",
                            "memory", "1G", NULL };
 nbd_connect_command (nbd, argv);

When the handle is closed the nbdkit subprocess is killed, which in
this case means the RAM disk is discarded, so this is useful for
testing.

=head2 Connecting to a subprocess using systemd socket activation

Some NBD servers — notably L<nbdkit(1)> and L<qemu-nbd(1)> — support
systemd socket activation allowing libnbd to pass a socket to the
subprocess.  This works very similarly to L<nbd_connect_command(3)>
described above, but you must use
L<nbd_connect_systemd_socket_activation(3)> instead.

=head2 Connecting to any socket

If none of the other nbd_connect* methods are suitable you can create
a connected socket yourself and pass it to L<nbd_connect_socket(3)>.

One use for this is in fuzzing where we use L<socketpair(2)> to create
the socket, then fork, then have the test harness in the child process
connected to libnbd over the socket pair (see:
L<https://gitlab.com/nbdkit/libnbd/-/blob/master/fuzzing/libnbd-fuzz-wrapper.c>).

Another use is to connect libnbd to an address family that it does not
support natively, such as XDP or IB.

=head1 CONTROLLING NEGOTIATION

By default, when beginning a connection, libnbd will handle all
negotiation with the server, using only the configuration
(eg. L<nbd_set_export_name(3)> or L<nbd_add_meta_context(3)>) that was
requested before the connection attempt; this phase continues until
L<nbd_aio_is_connecting(3)> no longer returns true, at which point,
either data commands are ready to use or else the connection has
failed with an error.

But there are scenarios in which it is useful to also control the
handshaking commands sent during negotiation, such as asking the
server for a list of available exports prior to selecting which one to
use.  This is done by calling L<nbd_set_opt_mode(3)> before
connecting; then after requesting a connection, the state machine will
pause at L<nbd_aio_is_negotiating(3)> at any point that the user can
decide which handshake command to send next.  Note that the
negotiation state is only reachable from newstyle servers; oldstyle
servers cannot negotiate and will progress all the way to the ready
state.

When the negotiating state is reached, you can initiate option
commands such as L<nbd_opt_list(3)> or their asynchronous equivalents,
as well as alter configuration such as export name that previously had
to be set before connection.  Since the NBD protocol does not allow
parallel negotiating commands, no cookie is involved, and you can
track completion of each command when the state is no longer
L<nbd_aio_is_connecting(3)>.  If L<nbd_opt_go(3)> fails but the
connection is still live, you will be back in negotiation state, where
you can request a different export name and try again.  Exiting the
negotiation state is only possible with a successful L<nbd_opt_go(3)>
which moves to the data phase, or L<nbd_opt_abort(3)> which performs a
clean shutdown of the connection by skipping the data phase.

=head1 EXPORTS AND FLAGS

It is possible for NBD servers to serve different content on different
“exports”.  For this you must pass the right export name to the
server.  Call this API before connecting:

 nbd_set_export_name (nbd, "export");

Note that there are some servers (like L<nbdkit(1)> E<le> 1.14) which
ignore this, and other servers (like L<qemu-nbd(8)>) which require it
to be set correctly but cannot serve different content.

These APIs are also available after a successful L<nbd_opt_info(3)>
during the negotiation phase, if you used L<nbd_set_opt_mode(3)> prior
to connecting.

=head2 Flag calls

After connecting the server will send back a set of flags describing
the export, such as whether it is writable and if it can support flush
to permanent storage.  These flags can be accessed from libnbd using
APIs such as:

 int is_read_only = nbd_is_read_only (nbd);
 int can_flush = nbd_can_flush (nbd);

Flag calls are: __API_FLAG_LINKS__

=head2 Size of the export

To get the size of the export in bytes, use L<nbd_get_size(3)>:

 int64_t size = nbd_get_size (nbd);

=head2 Block size constraints

Some NBD servers cannot handle requests at any byte boundary.  They
might, for example, require all requests to be aligned to 512 byte
sectors.

Also some servers advertise a preferred block size.  This is not a
requirement, but is the minimum block size that can be accessed
efficiently (usually without triggering expensive read-modify-write
cycles inside the server).

These are referred to as block size constraints and can be queried by
calling L<nbd_get_block_size(3)>.  Pay attention in particular to the
C<LIBNBD_SIZE_MINIMUM> constraint as some servers will fail requests
which are smaller or not aligned to this block size with C<EINVAL>
("Invalid argument") errors.

For information on the server side, see
L<nbdkit-plugin(3)/.block_size>.

For definitive information about block size constraints, read
L<the NBD protocol specification|https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md>.

=head2 Newstyle and oldstyle servers

Libnbd can connect to both the simpler, old NBD protocol (before 2011)
referred to now as C<"oldstyle">, or modern C<"newstyle-fixed">
servers.  ("newstyle" was only briefly available.)  To find out the
protocol used by the server, call L<nbd_get_protocol(3)>.

=head1 DATA COMMANDS

You can read and write data from the NBD server using L<nbd_pread(3)>
and L<nbd_pwrite(3)> or their asynchronous equivalents.

All data commands support a C<flags> argument (mandatory in C, but
optional in languages where it can default to 0).  For convenience,
the constant C<LIBNBD_CMD_FLAG_MASK> is defined with the set of flags
currently recognized by libnbd, where future NBD protocol extensions
may result in additional flags being supported; but in general,
specific data commands only accept a subset of known flags.

Libnbd defaults to performing some client-side sanity checking in each
of its data commands; for example, attempts to write to a server that
has advertised a read-only connection are rejected.  It is possible to
override aspects of this checking by using L<nbd_set_strict_mode(3)>.

Some servers also support:

=over 4

=item trim/discard

If L<nbd_can_trim(3)> returns true, L<nbd_trim(3)> can be used to “punch
holes” in the backing storage of the disk on the server.  Normally
(although not in every case) the holes read back as zeroes but take up
no space.

=item zeroing

If L<nbd_can_zero(3)> returns true, L<nbd_zero(3)> can be used to
efficiently zero parts of the disk without having to send large
amounts of zero bytes over the network (as would be necessary if using
L<nbd_pwrite(3)>).

This is slightly different from trimming because the backing storage
is still allocated.  For some storage types this can make future
writes more efficient and/or less likely to fail because of out of
space errors.

=item flushing

Some servers can commit data to permanent storage and tell you that
this has happened reliably.  There are two export flags associated
with this: L<nbd_can_flush(3)> and L<nbd_can_fua(3)>.

The L<nbd_flush(3)> call (available if L<nbd_can_flush(3)> returns true)
flushes all pending writes to disk and does not complete until that
operation has finished.  It is similar to using L<sync(2)> on POSIX
systems.

A more efficient way to achieve this is to set the flag
C<LIBNBD_CMD_FLAG_FUA> on write-like calls (like write, trim and
zero).  This flag means the call will not complete until committed to
permanent storage, but it does not involve flushing the entire disk.

=item prefetching

Some servers can prefetch data, making subsequent reads faster.  The
L<nbd_cache(3)> call (available if L<nbd_can_cache(3)> returns true) is used
to prefetch.

=item block status

Some servers are able to provide information about the various extents
within the image, via the notion of one or more meta contexts.  The
most common meta context is "base:allocation" (available in libnbd.h
as C<LIBNBD_CONTEXT_BASE_ALLOCATION>), which can be used to learn
which portions of a file are allocated or read as zero.  Other
contexts may be available; for example, L<qemu-nbd(8)> can expose a
meta context "qemu:dirty-bitmap:NAME" for tracking which portions of a
file are tracked by a qcow2 dirty bitmap.

In order to utilize block status, the client must call
L<nbd_add_meta_context(3)> prior to connecting, for each meta context
in which it is interested, then check L<nbd_can_meta_context(3)> after
connection to see which contexts the server actually supports.  If a
context is supported, the client can then use
L<nbd_block_status_64(3)> with a callback function that will receive
an array of structs describing consecutive extents within a context.
Each struct gives the length of the extent, then a bitmask description
of that extent (for the "base:allocation" context, the bitmask may
include C<LIBNBD_STATE_HOLE> for unallocated portions of the file,
and/or C<LIBNBD_STATE_ZERO> for portions of the file known to read as
zero).

There is a full example of requesting meta context and using block
status available at
L<https://gitlab.com/nbdkit/libnbd/blob/master/interop/dirty-bitmap.c>

=back

=head1 PERFORMANCE

=head2 Issuing multiple in-flight requests

NBD servers which properly implement the specification can handle
multiple data requests in flight over the same connection at the same
time.  Libnbd supports this when using the low level API.

To use it you simply issue more requests as needed (eg. using calls
like L<nbd_aio_pread(3)>, L<nbd_aio_pwrite(3)>) without waiting for previous
commands to complete.  You need to be careful that requests in flight
do not overlap with disk offsets of other write-like commands in
flight — an overlapping read may see indeterminate data, and an
overlapping write may even cause disk corruption where the resulting
disk contents do not match either of the two writes.

Each request is identified by a unique 64 bit cookie (assigned by
libnbd), allowing libnbd and callers to match replies to requests.
Replies may arrive out of order.  A request that is rejected
client-side for failing a sanity check (such as attempting to write to
a read-only server, see L<nbd_set_strict_mode(3)>) will fail rather
than returning a cookie, although closure cleanup is still performed.

Although in theory you can have an indefinite number of requests in
flight at the same time, in practice it's a good idea to limit them to
some number.  Libnbd will queue commands in the handle even if it
cannot write them to the server, so this limit is largely to prevent a
backlog of commands from consuming too much memory.  It is suggested
to start with a limit of 64 requests in flight (per NBD connection),
and measure how adjusting the limit up and down affects performance
for your local configuration.

There is a full example using multiple in-flight requests available at
L<https://gitlab.com/nbdkit/libnbd/blob/master/examples/threaded-reads-and-writes.c>

=head2 Multi-conn

Some NBD servers advertise “multi-conn” which means that it is safe to
make multiple connections to the server and load-balance commands
across all of the connections.

To do this you should open a single connection first and test for this
feature using L<nbd_can_multi_conn(3)>.  Without error handling it
would look like this:

 struct nbd_handle *nbd[4];
 size_t i;
 bool supports_multi_conn;

 nbd[0] = nbd_create ();
 nbd_connect_tcp (nbd[0], "server", "10809");
 supports_multi_conn = nbd_can_multi_conn (nbd[0]) > 0;

If multi-conn is supported then you can open further connections:

 if (supports_multi_conn) {
   for (i = 1; i <= 3; ++i) {
     nbd[i] = nbd_create ();
     nbd_connect_tcp (nbd[i], "server", "10809");
   }
 }

If you are issuing multiple in-flight requests (see above) and
limiting the number, then the limit should be applied to each
individual NBD connection.

=head1 ENCRYPTION AND AUTHENTICATION

The NBD protocol and libnbd supports TLS (sometimes incorrectly called
“SSL”) for encryption of the data stream and authentication of clients
and servers.  Libnbd defaults to TLS I<disabled> for maximum
interoperability.  To enable it on a handle you must call
L<nbd_set_tls(3)> before connecting.

To allow TLS, but fall back to unencrypted:

 nbd_set_tls (nbd, LIBNBD_TLS_ALLOW);

Use L<nbd_get_tls_negotiated(3)> to find out if TLS negotiation was
successful.  Avoid C<LIBNBD_TLS_ALLOW> if man-in-the-middle attacks
are a concern.

The most secure mode is to require TLS and fail to connect if the
server does not support it:

 nbd_set_tls (nbd, LIBNBD_TLS_REQUIRE);

It may also be necessary to verify that the server’s identity is
correct.  For some servers it may be necessary to verify to the server
that the client is permitted to connect.  This can be done using
either X.509 certificates, or TLS Pre-Shared Keys (PSK).  Certificates
are more secure.  PSK is far more convenient, but you must have an
existing secure channel to distribute the keys.

=head2 Setting up X.509 using system certificate authorities (CAs)

This is the default if you don’t call any other C<nbd_set_tls_*>
functions.  In this case the server must have a public (eg. HTTPS)
certificate which can be verified against the CAs registered on your
system (eg. under F</etc/pki>).

To disable server name verification — which opens you up to a potential
Man-In-The-Middle (MITM) attack — use:

 nbd_set_tls_verify_peer (nbd, false);

=head2 Setting up an X.509 certificate authority (CA)

You can set up your own CA and register clients and servers with it,
issuing client and server certificates which will reliably
authenticate your clients and servers to each other.

Doing this is described in detail in the L<nbdkit-tls(1)> manual.  The
only differences for libnbd are:

=over 4

=item *

Non-root certificates must be placed in C<$HOME/.pki/libnbd/> or
C<$HOME/.config/pki/libnbd/>

=item *

Libnbd reads F<client-cert.pem> and F<client-key.pem> (instead of
F<server-cert.pem> and F<server-key.pem>).

=back

Once you have set up the directory containing the certificates, call:

 nbd_set_tls_certificates (nbd, "/path/to/directory");

=head2 Setting up Pre-Shared Keys (PSK)

TLS Pre-Shared Keys are a much more convenient method of setting up
TLS, and more appropriate for NBD, but you should have an existing
secure method available to distribute the keys.  They are therefore
ideal if you want to set up an NBD service as an adjunct to an
existing secure REST API.

Use L<psktool(1)> to create a file of C<username:key> pairs:

 psktool -u username -p keys.psk

and pass this path to libnbd:

 nbd_set_tls_psk_file (nbd, "keys.psk");

If necessary you may need to set the client username (otherwise libnbd
will use your login name):

 nbd_set_tls_username (nbd, "username");

=head1 CALLBACKS

Some libnbd calls take callbacks (eg. L<nbd_set_debug_callback(3)>,
L<nbd_aio_pread(3)>).  Libnbd can call these functions while processing.

In the C API these libnbd calls take a structure which contains the
function pointer and an optional opaque C<void *user_data> pointer:

 nbd_aio_pread (nbd, buf, sizeof buf, offset,
                (nbd_completion_callback) { .callback = my_fn,
                                            .user_data = my_data },
                0);

For optional callbacks, if you don't want the callback, either set
C<.callback> to C<NULL> or use the equivalent macros (such as
C<NBD_NULL_COMPLETION>) defined in C<libnbd.h>:

 nbd_aio_pread (nbd, buf, sizeof buf, offset,
                NBD_NULL_COMPLETION, 0);

From other languages the structure and opaque pointer are not needed
because you can use closures to achieve the same effect.

=head2 Callback lifetimes

You can associate an optional free function with callbacks.  Libnbd
will call this function when the callback will not be called again by
libnbd, including in the case where the API fails.

This can be used to free associated C<user_data>.  For example:

 void *my_data = malloc (...);

 nbd_aio_pread_structured (nbd, buf, sizeof buf, offset,
                (nbd_chunk_callback) { .callback = my_fn,
                                       .user_data = my_data,
                                       .free = free },
                NBD_NULL_COMPLETION,
                0);

will call L<free(3)> once on C<my_data> after the point where it is
known that the S<C<chunk.callback = my_fn>> function can no longer be
called, regardless of how many times C<my_fn> was actually called.  If
both a mid-command and completion callback are supplied, the functions
will be reached in this order: mid-function callbacks, completion
callback, mid-function free, and finally completion free.

The free function is only accessible in the C API as it is not needed
in garbage collected programming languages.

=head2 Callbacks with C<.callback=NULL> and C<.free!=NULL>

It is possible to register a callback like this:

  ...
    (nbd_completion_callback) { .callback = NULL,
                                .user_data = my_data,
                                .free = free },
  ...

The meaning of this is that the callback is never called, but the free
function is still called after the last time the callback would have
been called.  This is useful for applying generic freeing actions when
asynchronous commands are retired.

=head2 Callbacks and locking

The callbacks are invoked at a point where the libnbd lock is held,
typically during a call to C<nbd_aio_notify_read>,
C<nbd_aio_notify_write>, C<nbd_aio_poll>, or other call that can
advance libnbd's state machine.  Depending on system load, it is even
possible for a callback to be reached before completion of the
C<nbd_aio_*> call that specified the callback.  As such, it is unsafe
for the callback to call any C<nbd_*> APIs on the same nbd object, as
it would cause deadlock.

=head2 Completion callbacks

All of the asynchronous commands have an optional completion callback
function that is used if the call to the asynchronous API reports
success.  The completion callback is invoked when the submitted
command is eventually marked complete, after any mid-command callbacks
have finished, and before any free functions.  The completion callback
is not reached if the asynchronous API itself fails, while free
callbacks are reached regardless of the result of the initial
asynchronous API.

When the completion callback returns C<1>, the command is
automatically retired (there is no need to call
L<nbd_aio_command_completed(3)>); for any other return value, the
command still needs to be manually retired (otherwise, the command
will tie up resources until L<nbd_close(3)> is eventually reached).

=head2 Callbacks with C<int *error> parameter

Some of the high-level commands (L<nbd_pread_structured(3)>,
L<nbd_block_status_64(3)>) involve the use of a callback function invoked
by the state machine at appropriate points in the server's reply
before the overall command is complete.  These callback functions,
along with all of the completion callbacks, include a parameter
C<error> which is a pointer containing the value of any error detected
so far.  If a callback function fails and wants to change the
resulting error of the overall command visible later in the API
sequence, it should assign back into C<error> and return C<-1> in the
C API.  Assignments into C<error> are ignored for any other return
value; similarly, assigning C<0> into C<error> does not have an
effect.  In other language bindings, reporting callback errors is
generally done by raising an exception rather than by return value.

Note that a mid-command callback might never be reached, such as if
libnbd detects that the command was invalid to send (see
L<nbd_set_strict_mode(3)>) or if the server reports a failure that
concludes the command.  It is safe for a mid-command callback to
ignore non-zero C<error>: all the other parameters to the mid-command
callback will still be valid (corresponding to the current portion of
the server's reply), and the overall command will still fail (at the
completion callback or L<nbd_aio_command_completed(3)> for an
asynchronous command, or as the result of the overall synchronous
command).  Returning C<-1> from a mid-command callback does not prevent
that callback from being reached again, if the server sends more
mid-command replies that warrant another use of that callback.  A
mid-command callback may be reached more times than expected if the
server is non-compliant.

On the other hand, if a completion callback is supplied (only possible
with asynchronous commands), it will not be called if the initial API
call fails (such as attempting an asynchronous command in the wrong
state - there is nothing to be completed since the command was not
queued), but will otherwise be called exactly once, and the completion
callback must not ignore the value pointed to by C<error>.  In
particular, the content of a buffer passed to L<nbd_aio_pread(3)> or
L<nbd_aio_pread_structured(3)> is undefined if C<*error> is non-zero
on entry to the completion callback.  It is recommended that if you
choose to use automatic command retirement (where the completion
callback returns C<1> to avoid needing to call
L<nbd_aio_command_completed(3)> later), your completion function
should return C<1> on all control paths, even when handling errors
(note that with automatic retirement, assigning into C<error> is
pointless as there is no later API to see that value).

=head1 STATISTICS COUNTERS

Libnbd tracks several statistics counters, useful for tracking how
much traffic was sent over the connection.  The counters track the
number of plaintext bytes sent and received by the NBD protocol (not
necessarily the number of bytes sent over the socket, particularly
when TLS is enabled), as well as the number of protocol chunks (a
group of bytes delineated by a magic number, and not the same as the
number of TCP packets).

 printf ("bytes: sent=%" PRIu64 " received=%" PRIu64,
          nbd_stats_bytes_sent (nbd), nbd_stats_bytes_received (nbd));
 printf ("chunks: sent=%" PRIu64 " received=%" PRIu64,
          nbd_stats_chunks_sent (nbd), nbd_stats_chunks_received (nbd));

=head1 SIGNALS

Libnbd does not install signal handlers.  It attempts to disable
C<SIGPIPE> when writing to the NBD socket using the C<MSG_NOSIGNAL>
flag of L<send(2)>, or the C<SO_NOSIGPIPE> socket option, on platforms
that support those.

On some old Linux or newer non-Linux platforms the main program may
wish to register a signal handler to ignore SIGPIPE:

 signal (SIGPIPE, SIG_IGN);

=head1 COMPILING YOUR PROGRAM

On most systems, C programs that use libnbd can be compiled like this:

 cc prog.c -o prog -lnbd

To detect if the libnbd library and header file is installed, the
preferred method is to use L<pkg-config(1)> or L<pkgconf(1)>:

 pkg-config libnbd --exists || fail libnbd is required

In case the library or header file are not installed in the usual
system locations, you can compile your program like this, using
pkg-config to detect the proper location of libnbd:

 cc prog.c -o prog `pkg-config libnbd --cflags --libs`

To compile an external project against a built copy of the libnbd
source tree which hasn't been installed, see the F<./run> script.

=head2 Autoconf projects

External projects which use autoconf and need to check if libnbd is
installed should use the C<PKG_CHECK_MODULES> macro in F<configure.ac>
like this:

 PKG_CHECK_MODULES([LIBNBD], [libnbd])

This will define C<@LIBNBD_CFLAGS@> and C<@LIBNBD_LIBS@> which you
will need to add to your F<Makefile.am>.

=head2 CMake projects

For CMake projects use:

 find_package(PkgConfig REQUIRED)
 pkg_check_modules(LIBNBD REQUIRED libnbd)
 target_link_libraries(prog ${LIBNBD_LIBRARIES})
 target_include_directories(prog PUBLIC ${LIBNBD_INCLUDE_DIRS})
 target_compile_options(prog PUBLIC ${LIBNBD_CFLAGS_OTHER})

=head2 Meson projects

For meson projects use:

 nbd_dep = dependency('libnbd')
 executable('prog', 'prog.c', dependencies : [nbd_dep])

=head1 ENVIRONMENT VARIABLES

=over 4

=item C<HOME>

Used in some situations to find TLS certificates.  See
L<nbd_set_tls_certificates(3)>.

=item C<LIBNBD_DEBUG>

If this is set to the exact string C<1> when the handle is created
then debugging is enabled.  See L</DEBUGGING MESSAGES> above.

=item C<LOGNAME>

The default TLS username.  See L<nbd_set_tls_username(3)>.

=back

=head1 SEE ALSO

=head2 C API

__API_LINKS__

=head2 Servers

L<nbdkit(1)>,
L<nbd-server(1)>,
L<qemu-nbd(8)>.

=head2 Encryption tools

L<certtool(1)>,
L<nbdkit-tls(1)>,
L<psktool(1)>.

=head2 Standards

L<https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md>,
L<https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md>.

=head2 Release notes

L<libnbd-release-notes-1.22(1)>,
L<libnbd-release-notes-1.20(1)>,
L<libnbd-release-notes-1.18(1)>,
L<libnbd-release-notes-1.16(1)>,
L<libnbd-release-notes-1.14(1)>,
L<libnbd-release-notes-1.12(1)>,
L<libnbd-release-notes-1.10(1)>,
L<libnbd-release-notes-1.8(1)>,
L<libnbd-release-notes-1.6(1)>,
L<libnbd-release-notes-1.4(1)>,
L<libnbd-release-notes-1.2(1)>.

=head2 Other

L<libnbd-security(3)>,
L<nbdcopy(1)>,
L<nbddump(1)>,
L<nbdfuse(1)>,
L<nbdinfo(1)>,
L<nbdsh(1)>,
L<nbdublk(1)>,
L<qemu(1)>.

=head1 AUTHORS

Eric Blake

Richard W.M. Jones

=head1 COPYRIGHT

Copyright Red Hat