1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791
|
.. _rest_api:
Patroni REST API
================
Patroni has a rich REST API, which is used by Patroni itself during the leader race, by the :ref:`patronictl` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring. Below you will find the list of Patroni REST API endpoints.
Health check endpoints
----------------------
For all health check ``GET`` requests Patroni returns a JSON document with the status of the node, along with the HTTP status code. If you don't want or don't need the JSON document, you might consider using the ``HEAD`` or ``OPTIONS`` method instead of ``GET``.
- The following requests to Patroni REST API will return HTTP status code **200** only when the Patroni node is running as the primary with leader lock:
- ``GET /``
- ``GET /primary``
- ``GET /read-write``
- ``GET /standby-leader``: returns HTTP status code **200** only when the Patroni node is running as the leader in a :ref:`standby cluster <standby_cluster>`.
- ``GET /leader``: returns HTTP status code **200** when the Patroni node has the leader lock. The major difference from the two previous endpoints is that it doesn't take into account whether PostgreSQL is running as the ``primary`` or the ``standby_leader``.
- ``GET /replica``: replica health check endpoint. It returns HTTP status code **200** only when the Patroni node is in the state ``running``, the role is ``replica`` and ``noloadbalance`` tag is not set.
- ``GET /replica?lag=<max-lag>``: replica check endpoint. In addition to checks from ``replica``, it also checks replication latency and returns status code **200** only when it is below specified value. The key cluster.last_leader_operation from DCS is used for Leader wal position and compute latency on replica for performance reasons. max-lag can be specified in bytes (integer) or in human readable values, for e.g. 16kB, 64MB, 1GB.
- ``GET /replica?lag=1048576``
- ``GET /replica?lag=1024kB``
- ``GET /replica?lag=10MB``
- ``GET /replica?lag=1GB``
- ``GET /replica?tag_key1=value1&tag_key2=value2``: replica check endpoint. In addition, It will also check for user defined tags ``key1`` and ``key2`` and their respective values in the **tags** section of the yaml configuration management. If the tag isn't defined for an instance, or if the value in the yaml configuration doesn't match the querying value, it will return HTTP Status Code 503.
In the following requests, since we are checking for the leader or standby-leader status, Patroni doesn't apply any of the user defined tags and they will be ignored.
- ``GET /?tag_key1=value1&tag_key2=value2``
- ``GET /leader?tag_key1=value1&tag_key2=value2``
- ``GET /primary?tag_key1=value1&tag_key2=value2``
- ``GET /read-write?tag_key1=value1&tag_key2=value2``
- ``GET /standby_leader?tag_key1=value1&tag_key2=value2``
- ``GET /standby-leader?tag_key1=value1&tag_key2=value2``
- ``GET /read-only``: like the above endpoint, but also includes the primary.
- ``GET /synchronous`` or ``GET /sync``: returns HTTP status code **200** only when the Patroni node is running as a synchronous standby.
- ``GET /read-only-sync``: like the above endpoint, but also includes the primary.
- ``GET /quorum``: returns HTTP status code **200** only when this Patroni node is listed as a quorum node in ``synchronous_standby_names`` on the primary.
- ``GET /read-only-quorum``: like the above endpoint, but also includes the primary.
- ``GET /asynchronous`` or ``GET /async``: returns HTTP status code **200** only when the Patroni node is running as an asynchronous standby.
- ``GET /asynchronous?lag=<max-lag>`` or ``GET /async?lag=<max-lag>``: asynchronous standby check endpoint. In addition to checks from ``asynchronous`` or ``async``, it also checks replication latency and returns status code **200** only when it is below specified value. The key cluster.last_leader_operation from DCS is used for Leader wal position and compute latency on replica for performance reasons. max-lag can be specified in bytes (integer) or in human readable values, for e.g. 16kB, 64MB, 1GB.
- ``GET /async?lag=1048576``
- ``GET /async?lag=1024kB``
- ``GET /async?lag=10MB``
- ``GET /async?lag=1GB``
- ``GET /health``: returns HTTP status code **200** only when PostgreSQL is up and running.
- ``GET /liveness``: returns HTTP status code **200** if Patroni heartbeat loop is properly running and **503** if the last run was more than ``ttl`` seconds ago on the primary or ``2*ttl`` on the replica. Could be used for ``livenessProbe``.
- ``GET /readiness?lag=<max-lag>&mode=apply|write``: returns HTTP status code **200** when the Patroni node is running as the leader or when PostgreSQL is up, replicating and not too far behind the leader. The lag parameter sets how far a standby is allowed to be behind, it defaults to ``maximum_lag_on_failover``. Lag can be specified in bytes or in human readable values, for e.g. 16kB, 64MB, 1GB. Mode sets whether the WAL needs to be replayed (apply) or just received (write). The default is apply.
When used as Kubernetes ``readinessProbe`` it will make sure freshly started pods only become ready when they have caught up to the leader. This combined with a PodDisruptionBudget will protect against leader being terminated too early during a rolling restart of nodes. It will also make sure that replicas that cannot keep up with replication do not service read-only traffic. The endpoint could be used for ``readinessProbe`` when it is not possible to use Kubernetes endpoints for leader elections (OpenShift).
The ``liveness`` endpoint is very light-weight and not executing any SQL. Probes should be configured in such a way that they start failing about time when the leader key is expiring. With the default value of ``ttl``, which is ``30s`` example probes would look like:
.. code-block:: yaml
readinessProbe:
httpGet:
scheme: HTTP
path: /readiness
port: 8008
initialDelaySeconds: 3
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
livenessProbe:
httpGet:
scheme: HTTP
path: /liveness
port: 8008
initialDelaySeconds: 3
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
Monitoring endpoint
-------------------
The ``GET /patroni`` is used by Patroni during the leader race. It also could be used by your monitoring system. The JSON document produced by this endpoint has the same structure as the JSON produced by the health check endpoints.
**Example:** A healthy cluster
.. code-block:: bash
$ curl -s http://localhost:8008/patroni | jq .
{
"state": "running",
"postmaster_start_time": "2024-08-28 19:39:26.352526+00:00",
"role": "primary",
"server_version": 160004,
"xlog": {
"location": 67395656
},
"timeline": 1,
"replication": [
{
"usename": "replicator",
"application_name": "patroni2",
"client_addr": "10.89.0.6",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
},
{
"usename": "replicator",
"application_name": "patroni3",
"client_addr": "10.89.0.2",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
}
],
"dcs_last_seen": 1692356718,
"tags": {
"clonefrom": true
},
"database_system_identifier": "7268616322854375442",
"patroni": {
"version": "4.0.0",
"scope": "demo",
"name": "patroni1"
}
}
**Example:** An unlocked cluster
.. code-block:: bash
$ curl -s http://localhost:8008/patroni | jq .
{
"state": "running",
"postmaster_start_time": "2024-08-28 19:39:26.352526+00:00",
"role": "replica",
"server_version": 160004,
"xlog": {
"received_location": 67419744,
"replayed_location": 67419744,
"replayed_timestamp": null,
"paused": false
},
"timeline": 1,
"replication": [
{
"usename": "replicator",
"application_name": "patroni2",
"client_addr": "10.89.0.6",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
},
{
"usename": "replicator",
"application_name": "patroni3",
"client_addr": "10.89.0.2",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
}
],
"cluster_unlocked": true,
"dcs_last_seen": 1692356928,
"tags": {
"clonefrom": true
},
"database_system_identifier": "7268616322854375442",
"patroni": {
"version": "4.0.0",
"scope": "demo",
"name": "patroni1"
}
}
**Example:** An unlocked cluster with :ref:`DCS failsafe mode <dcs_failsafe_mode>` enabled
.. code-block:: bash
$ curl -s http://localhost:8008/patroni | jq .
{
"state": "running",
"postmaster_start_time": "2024-08-28 19:39:26.352526+00:00",
"role": "replica",
"server_version": 160004,
"xlog": {
"location": 67420024
},
"timeline": 1,
"replication": [
{
"usename": "replicator",
"application_name": "patroni2",
"client_addr": "10.89.0.6",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
},
{
"usename": "replicator",
"application_name": "patroni3",
"client_addr": "10.89.0.2",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
}
],
"cluster_unlocked": true,
"failsafe_mode_is_active": true,
"dcs_last_seen": 1692356928,
"tags": {
"clonefrom": true
},
"database_system_identifier": "7268616322854375442",
"patroni": {
"version": "4.0.0",
"scope": "demo",
"name": "patroni1"
}
}
**Example:** A cluster with the :ref:`pause mode <pause>` enabled
.. code-block:: bash
$ curl -s http://localhost:8008/patroni | jq .
{
"state": "running",
"postmaster_start_time": "2024-08-28 19:39:26.352526+00:00",
"role": "replica",
"server_version": 160004,
"xlog": {
"location": 67420024
},
"timeline": 1,
"replication": [
{
"usename": "replicator",
"application_name": "patroni2",
"client_addr": "10.89.0.6",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
},
{
"usename": "replicator",
"application_name": "patroni3",
"client_addr": "10.89.0.2",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
}
],
"pause": true,
"dcs_last_seen": 1724874295,
"tags": {
"clonefrom": true
},
"database_system_identifier": "7268616322854375442",
"patroni": {
"version": "4.0.0",
"scope": "demo",
"name": "patroni1"
}
}
Retrieve the Patroni metrics in Prometheus format through the ``GET /metrics`` endpoint.
.. code-block:: bash
$ curl http://localhost:8008/metrics
# HELP patroni_version Patroni semver without periods. \
# TYPE patroni_version gauge
patroni_version{scope="batman",name="patroni1"} 040000
# HELP patroni_postgres_running Value is 1 if Postgres is running, 0 otherwise.
# TYPE patroni_postgres_running gauge
patroni_postgres_running{scope="batman",name="patroni1"} 1
# HELP patroni_postmaster_start_time Epoch seconds since Postgres started.
# TYPE patroni_postmaster_start_time gauge
patroni_postmaster_start_time{scope="batman",name="patroni1"} 1724873966.352526
# HELP patroni_primary Value is 1 if this node is the leader, 0 otherwise.
# TYPE patroni_primary gauge
patroni_primary{scope="batman",name="patroni1"} 1
# HELP patroni_xlog_location Current location of the Postgres transaction log, 0 if this node is not the leader.
# TYPE patroni_xlog_location counter
patroni_xlog_location{scope="batman",name="patroni1"} 22320573386952
# HELP patroni_standby_leader Value is 1 if this node is the standby_leader, 0 otherwise.
# TYPE patroni_standby_leader gauge
patroni_standby_leader{scope="batman",name="patroni1"} 0
# HELP patroni_replica Value is 1 if this node is a replica, 0 otherwise.
# TYPE patroni_replica gauge
patroni_replica{scope="batman",name="patroni1"} 0
# HELP patroni_sync_standby Value is 1 if this node is a sync standby replica, 0 otherwise.
# TYPE patroni_sync_standby gauge
patroni_sync_standby{scope="batman",name="patroni1"} 0
# HELP patroni_quorum_standby Value is 1 if this node is a quorum standby replica, 0 otherwise.
# TYPE patroni_quorum_standby gauge
patroni_quorum_standby{scope="batman",name="patroni1"} 0
# HELP patroni_xlog_received_location Current location of the received Postgres transaction log, 0 if this node is not a replica.
# TYPE patroni_xlog_received_location counter
patroni_xlog_received_location{scope="batman",name="patroni1"} 0
# HELP patroni_xlog_replayed_location Current location of the replayed Postgres transaction log, 0 if this node is not a replica.
# TYPE patroni_xlog_replayed_location counter
patroni_xlog_replayed_location{scope="batman",name="patroni1"} 0
# HELP patroni_xlog_replayed_timestamp Current timestamp of the replayed Postgres transaction log, 0 if null.
# TYPE patroni_xlog_replayed_timestamp gauge
patroni_xlog_replayed_timestamp{scope="batman",name="patroni1"} 0
# HELP patroni_xlog_paused Value is 1 if the Postgres xlog is paused, 0 otherwise.
# TYPE patroni_xlog_paused gauge
patroni_xlog_paused{scope="batman",name="patroni1"} 0
# HELP patroni_postgres_streaming Value is 1 if Postgres is streaming, 0 otherwise.
# TYPE patroni_postgres_streaming gauge
patroni_postgres_streaming{scope="batman",name="patroni1"} 1
# HELP patroni_postgres_in_archive_recovery Value is 1 if Postgres is replicating from archive, 0 otherwise.
# TYPE patroni_postgres_in_archive_recovery gauge
patroni_postgres_in_archive_recovery{scope="batman",name="patroni1"} 0
# HELP patroni_postgres_server_version Version of Postgres (if running), 0 otherwise.
# TYPE patroni_postgres_server_version gauge
patroni_postgres_server_version{scope="batman",name="patroni1"} 160004
# HELP patroni_cluster_unlocked Value is 1 if the cluster is unlocked, 0 if locked.
# TYPE patroni_cluster_unlocked gauge
patroni_cluster_unlocked{scope="batman",name="patroni1"} 0
# HELP patroni_postgres_timeline Postgres timeline of this node (if running), 0 otherwise.
# TYPE patroni_postgres_timeline counter
patroni_failsafe_mode_is_active{scope="batman",name="patroni1"} 0
# HELP patroni_postgres_timeline Postgres timeline of this node (if running), 0 otherwise.
# TYPE patroni_postgres_timeline counter
patroni_postgres_timeline{scope="batman",name="patroni1"} 24
# HELP patroni_dcs_last_seen Epoch timestamp when DCS was last contacted successfully by Patroni.
# TYPE patroni_dcs_last_seen gauge
patroni_dcs_last_seen{scope="batman",name="patroni1"} 1724874235
# HELP patroni_pending_restart Value is 1 if the node needs a restart, 0 otherwise.
# TYPE patroni_pending_restart gauge
patroni_pending_restart{scope="batman",name="patroni1"} 1
# HELP patroni_is_paused Value is 1 if auto failover is disabled, 0 otherwise.
# TYPE patroni_is_paused gauge
patroni_is_paused{scope="batman",name="patroni1"} 1
# HELP patroni_postgres_state Numeric representation of Postgres state.
# Values: 0=initdb, 1=initdb_failed, 2=custom_bootstrap, 3=custom_bootstrap_failed, 4=creating_replica, 5=running, 6=starting, 7=bootstrap_starting, 8=start_failed, 9=restarting, 10=restart_failed, 11=stopping, 12=stopped, 13=stop_failed, 14=crashed
# TYPE patroni_postgres_state gauge
patroni_postgres_state{scope="batman",name="patroni1"} 5
PostgreSQL State Values
^^^^^^^^^^^^^^^^^^^^^^^
The ``patroni_postgres_state`` metric provides a numeric representation of the current PostgreSQL instance state. This is useful for monitoring and alerting systems that need to track state changes over time. The numeric values are generated using the ``PostgresqlState.get_metrics_description()`` static method.
.. list-table:: PostgreSQL State Values
:widths: 10 20 50
:header-rows: 1
* - Value
- State Name
- Description
* - 0
- initdb
- Initializing new cluster
* - 1
- initdb_failed
- Initialization of new cluster failed
* - 2
- custom_bootstrap
- Running custom bootstrap script
* - 3
- custom_bootstrap_failed
- Custom bootstrap script failed
* - 4
- creating_replica
- Creating replica from primary
* - 5
- running
- PostgreSQL is running normally
* - 6
- starting
- PostgreSQL is starting up
* - 7
- bootstrap_starting
- Starting after custom bootstrap
* - 8
- start_failed
- PostgreSQL start failed
* - 9
- restarting
- PostgreSQL is restarting
* - 10
- restart_failed
- PostgreSQL restart failed
* - 11
- stopping
- PostgreSQL is stopping
* - 12
- stopped
- PostgreSQL is stopped
* - 13
- stop_failed
- PostgreSQL stop failed
* - 14
- crashed
- PostgreSQL has crashed
.. note::
These numeric values are fixed and will never change to maintain backward compatibility with existing monitoring systems. If new states are added in the future, they will be assigned new numeric values without changing existing ones.
Cluster status endpoints
------------------------
- The ``GET /cluster`` endpoint generates a JSON document describing the current cluster topology and state:
.. code-block:: bash
$ curl -s http://localhost:8008/cluster | jq .
{
"members": [
{
"name": "patroni1",
"role": "leader",
"state": "running",
"api_url": "http://10.89.0.4:8008/patroni",
"host": "10.89.0.4",
"port": 5432,
"timeline": 5,
"tags": {
"clonefrom": true
}
},
{
"name": "patroni2",
"role": "replica",
"state": "streaming",
"api_url": "http://10.89.0.6:8008/patroni",
"host": "10.89.0.6",
"port": 5433,
"timeline": 5,
"tags": {
"clonefrom": true
},
"receive_lag": 0,
"receive_lsn": "0/4000060",
"replay_lag": 0,
"replay_lsn": "0/4000060",
"lag": 0,
"lsn": "0/4000060"
}
],
"scope": "demo",
"scheduled_switchover": {
"at": "2023-09-24T10:36:00+02:00",
"from": "patroni1",
"to": "patroni3"
}
}
- The ``GET /history`` endpoint provides a view on the history of cluster switchovers/failovers. The format is very similar to the content of history files in the ``pg_wal`` directory. The only difference is the timestamp field showing when the new timeline was created.
.. code-block:: bash
$ curl -s http://localhost:8008/history | jq .
[
[
1,
25623960,
"no recovery target specified",
"2019-09-23T16:57:57+02:00"
],
[
2,
25624344,
"no recovery target specified",
"2019-09-24T09:22:33+02:00"
],
[
3,
25624752,
"no recovery target specified",
"2019-09-24T09:26:15+02:00"
],
[
4,
50331856,
"no recovery target specified",
"2019-09-24T09:35:52+02:00"
]
]
.. _config_endpoint:
Config endpoint
---------------
``GET /config``: Get the current version of the dynamic configuration:
.. code-block:: bash
$ curl -s http://localhost:8008/config | jq .
{
"ttl": 30,
"loop_wait": 10,
"retry_timeout": 10,
"maximum_lag_on_failover": 1048576,
"postgresql": {
"use_slots": true,
"use_pg_rewind": true,
"parameters": {
"hot_standby": "on",
"wal_level": "hot_standby",
"max_wal_senders": 5,
"max_replication_slots": 5,
"max_connections": "100"
}
}
}
``PATCH /config``: Change the existing configuration.
.. code-block:: bash
$ curl -s -XPATCH -d \
'{"loop_wait":5,"ttl":20,"postgresql":{"parameters":{"max_connections":"101"}}}' \
http://localhost:8008/config | jq .
{
"ttl": 20,
"loop_wait": 5,
"maximum_lag_on_failover": 1048576,
"retry_timeout": 10,
"postgresql": {
"use_slots": true,
"use_pg_rewind": true,
"parameters": {
"hot_standby": "on",
"wal_level": "hot_standby",
"max_wal_senders": 5,
"max_replication_slots": 5,
"max_connections": "101"
}
}
}
The above REST API call patches the existing configuration and returns the new configuration.
Let's check that the node processed this configuration. First of all it should start printing log lines every 5 seconds (loop_wait=5). The change of "max_connections" requires a restart, so the "pending_restart" flag should be exposed:
.. code-block:: bash
$ curl -s http://localhost:8008/patroni | jq .
{
"database_system_identifier": "6287881213849985952",
"postmaster_start_time": "2024-08-28 19:39:26.352526+00:00",
"xlog": {
"location": 2197818976
},
"timeline": 1,
"dcs_last_seen": 1724874545,
"database_system_identifier": "7408277255830290455",
"pending_restart": true,
"pending_restart_reason": {
"max_connections": {
"old_value": "100",
"new_value": "101"
}
},
"patroni": {
"version": "4.0.0",
"scope": "batman",
"name": "patroni1"
},
"state": "running",
"role": "primary",
"server_version": 160004
}
Removing parameters:
If you want to remove (reset) some setting just patch it with ``null``:
.. code-block:: bash
$ curl -s -XPATCH -d \
'{"postgresql":{"parameters":{"max_connections":null}}}' \
http://localhost:8008/config | jq .
{
"ttl": 20,
"loop_wait": 5,
"retry_timeout": 10,
"maximum_lag_on_failover": 1048576,
"postgresql": {
"use_slots": true,
"use_pg_rewind": true,
"parameters": {
"hot_standby": "on",
"unix_socket_directories": ".",
"wal_level": "hot_standby",
"max_wal_senders": 5,
"max_replication_slots": 5
}
}
}
The above call removes ``postgresql.parameters.max_connections`` from the dynamic configuration.
``PUT /config``: It's also possible to perform the full rewrite of an existing dynamic configuration unconditionally:
.. code-block:: bash
$ curl -s -XPUT -d \
'{"maximum_lag_on_failover":1048576,"retry_timeout":10,"postgresql":{"use_slots":true,"use_pg_rewind":true,"parameters":{"hot_standby":"on","wal_level":"hot_standby","unix_socket_directories":".","max_wal_senders":5}},"loop_wait":3,"ttl":20}' \
http://localhost:8008/config | jq .
{
"ttl": 20,
"maximum_lag_on_failover": 1048576,
"retry_timeout": 10,
"postgresql": {
"use_slots": true,
"parameters": {
"hot_standby": "on",
"unix_socket_directories": ".",
"wal_level": "hot_standby",
"max_wal_senders": 5
},
"use_pg_rewind": true
},
"loop_wait": 3
}
Switchover and failover endpoints
---------------------------------
.. _switchover_api:
Switchover
^^^^^^^^^^
``/switchover`` endpoint only works when the cluster is healthy (there is a leader). It also allows to schedule a switchover at a given time.
When calling ``/switchover`` endpoint a candidate can be specified but is not required, in contrast to ``/failover`` endpoint. If a candidate is not provided, all the eligible nodes of the cluster will participate in the leader race after the leader stepped down.
In the JSON body of the ``POST`` request you must specify the ``leader`` field. The ``candidate`` and the ``scheduled_at`` fields are optional and can be used to schedule a switchover at a specific time.
Depending on the situation, requests might return different HTTP status codes and bodies. Status code **200** is returned when the switchover or failover successfully completed. If the switchover was successfully scheduled, Patroni will return HTTP status code **202**. In case something went wrong, the error status code (one of **400**, **412**, or **503**) will be returned with some details in the response body.
``DELETE /switchover`` can be used to delete the currently scheduled switchover.
**Example:** perform a switchover to any healthy standby
.. code-block:: bash
$ curl -s http://localhost:8008/switchover -XPOST -d '{"leader":"postgresql1"}'
Successfully switched over to "postgresql2"
**Example:** perform a switchover to a specific node
.. code-block:: bash
$ curl -s http://localhost:8008/switchover -XPOST -d \
'{"leader":"postgresql1","candidate":"postgresql2"}'
Successfully switched over to "postgresql2"
**Example:** schedule a switchover from the leader to any other healthy standby in the cluster at a specific time.
.. code-block:: bash
$ curl -s http://localhost:8008/switchover -XPOST -d \
'{"leader":"postgresql0","scheduled_at":"2019-09-24T12:00+00"}'
Switchover scheduled
Failover
^^^^^^^^
``/failover`` endpoint can be used to perform a manual failover when there are no healthy nodes (e.g. to an asynchronous standby if all synchronous standbys are not healthy enough to promote). However there is no requirement for a cluster not to have leader - failover can also be run on a healthy cluster.
In the JSON body of the ``POST`` request you must specify the ``candidate`` field. If the ``leader`` field is specified, a switchover is triggered instead.
**Example:**
.. code-block:: bash
$ curl -s http://localhost:8008/failover -XPOST -d '{"candidate":"postgresql1"}'
Successfully failed over to "postgresql1"
.. warning::
:ref:`Be very careful <failover_healthcheck>` when using this endpoint, as this can cause data loss in certain situations. In most cases, :ref:`the switchover endpoint <switchover_api>` satisfies the administrator's needs.
``POST /switchover`` and ``POST /failover`` endpoints are used by :ref:`patronictl_switchover` and :ref:`patronictl_failover`, respectively.
``DELETE /switchover`` is used by :ref:`patronictl flush cluster-name switchover <patronictl_flush_parameters>`.
.. list-table:: Failover/Switchover comparison
:widths: 25 25 25
:header-rows: 1
* -
- Failover
- Switchover
* - Requires leader specified
- no
- yes
* - Requires candidate specified
- yes
- no
* - Can be run in pause
- yes
- yes (only to a specific candidate)
* - Can be scheduled
- no
- yes (if not in pause)
.. _failover_healthcheck:
Healthy standby
^^^^^^^^^^^^^^^
There are a couple of checks that a member of a cluster should pass to be able to participate in the leader race during a switchover or to become a leader as a failover/switchover candidate:
- be reachable via Patroni API;
- not have ``nofailover`` tag set to ``true``;
- have watchdog fully functional (if required by the configuration);
- in case of a switchover in a healthy cluster or an automatic failover, not exceed maximum replication lag (``maximum_lag_on_failover`` :ref:`configuration parameter <dynamic_configuration>`);
- in case of a switchover in a healthy cluster or an automatic failover, not have a timeline number smaller than the cluster timeline if ``check_timeline`` :ref:`configuration parameter <dynamic_configuration>` is set to ``true``;
- in :ref:`synchronous mode <synchronous_mode>`:
- In case of a switchover (both with and without a candidate): be listed in the ``/sync`` key members;
- For a failover in both healthy and unhealthy clusters, this check is omitted.
.. warning::
In case of a manual failover in a cluster without a leader, a candidate will be allowed to promote even if:
- it is not in the ``/sync`` key members when synchronous mode is enabled;
- its lag exceeds the maximum replication lag allowed;
- it has the timeline number smaller than the last known cluster timeline.
.. _restart_endpoint:
Restart endpoint
----------------
- ``POST /restart``: You can restart Postgres on the specific node by performing the ``POST /restart`` call. In the JSON body of ``POST`` request it is possible to optionally specify some restart conditions:
- **restart_pending**: boolean, if set to ``true`` Patroni will restart PostgreSQL only when restart is pending in order to apply some changes in the PostgreSQL config.
- **role**: perform restart only if the current role of the node matches with the role from the POST request.
- **postgres_version**: perform restart only if the current version of postgres is smaller than specified in the POST request.
- **timeout**: how long we should wait before PostgreSQL starts accepting connections. Overrides ``primary_start_timeout``.
- **schedule**: timestamp with time zone, schedule the restart somewhere in the future.
- ``DELETE /restart``: delete the scheduled restart
``POST /restart`` and ``DELETE /restart`` endpoints are used by :ref:`patronictl_restart` and :ref:`patronictl flush cluster-name restart <patronictl_flush_parameters>` respectively.
.. _reload_endpoint:
Reload endpoint
---------------
The ``POST /reload`` call will order Patroni to re-read and apply the configuration file. This is the equivalent of sending the ``SIGHUP`` signal to the Patroni process. In case you changed some of the Postgres parameters which require a restart (like **shared_buffers**), you still have to explicitly do the restart of Postgres by either calling the ``POST /restart`` endpoint or with the help of :ref:`patronictl_restart`.
The reload endpoint is used by :ref:`patronictl_reload`.
Reinitialize endpoint
---------------------
``POST /reinitialize``: reinitialize the PostgreSQL data directory on the specified node. It is allowed to be executed only on replicas. Once called, it will remove the data directory and start ``pg_basebackup`` or some alternative :ref:`replica creation method <custom_replica_creation>`.
The call might fail if Patroni is in a loop trying to recover (restart) a failed Postgres. In order to overcome this problem one can specify ``{"force":true}`` in the request body.
You can specify {"from-leader":true} in the request body to directly get basebackup from leader node. This is useful when executing reinit during all replica nodes fail.
The reinitialize endpoint is used by :ref:`patronictl_reinit`.
|