1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969
|
# Elasticsearch Puppet Module
[](https://github.com/voxpupuli/puppet-elasticsearch/actions?query=workflow%3ACI)
[](https://github.com/voxpupuli/puppet-elasticsearch/actions/workflows/release.yml)
[](https://forge.puppetlabs.com/puppet/elasticsearch)
[](https://forge.puppetlabs.com/puppet/elasticsearch)
[](https://forge.puppetlabs.com/puppet/elasticsearch)
[](https://forge.puppetlabs.com/puppet/elasticsearch)
[](http://www.puppetmodule.info/m/puppet-elasticsearch)
[](LICENSE)
[](#transfer-notice)
#### Table of Contents
1. [Module description - What the module does and why it is useful](#module-description)
2. [Setup - The basics of getting started with Elasticsearch](#setup)
* [The module manages the following](#the-module-manages-the-following)
* [Requirements](#requirements)
3. [Usage - Configuration options and additional functionality](#usage)
4. [Advanced features - Extra information on advanced usage](#advanced-features)
5. [Reference - An under-the-hood peek at what the module is doing and how](#reference)
6. [Limitations - OS compatibility, etc.](#limitations)
7. [Development - Guide for contributing to the module](#development)
8. [Support - When you need help with this module](#support)
9. [Transfer Notice](#transfer-notice)
## Module description
This module sets up [Elasticsearch](https://www.elastic.co/overview/elasticsearch/) instances with additional resource for plugins, templates, and more.
This module is actively tested against Elasticsearch 2.x, 5.x, and 6.x.
# WARNING: The 7.x major release of this module contains breaking changes!
In order to simplify the management of Elasticsearch moving forward, and add support for both Elasticsearch 6.x and 7.x, support for
running multiple instances of Elasticsearch has been removed.
This module also does not currently handle the migration from the instance based configuration to the new single deployment model.
Therefore in-place upgrades from version 6.x of this module to 7.x, or migrations from multi-instance to single deployment is not currently supported.
We hope to add support for this in a future release.
Therefore please ensure that you test this major release in your environment before using it in production!
## Setup
### The module manages the following
* Elasticsearch repository files.
* Elasticsearch package.
* Elasticsearch configuration file.
* Elasticsearch service.
* Elasticsearch plugins.
* Elasticsearch snapshot repositories.
* Elasticsearch templates.
* Elasticsearch ingest pipelines.
* Elasticsearch index settings.
* Elasticsearch users, roles, and certificates.
* Elasticsearch licenses.
* Elasticsearch keystores.
### Requirements
* The [stdlib](https://forge.puppetlabs.com/puppetlabs/stdlib) Puppet library.
* [Augeas](http://augeas.net/)
* [puppetlabs-java_ks](https://forge.puppetlabs.com/puppetlabs/java_ks) for certificate management (optional).
Beginning with Elasticsearch 7.0.0, a Java JDK has been bundled as part of the elasticsearch package.
However there still needs to be a version of Java present on the system being managed in order for Puppet to be able to run various utilities.
We recommend managing your Java installation with the [puppetlabs-java](https://forge.puppetlabs.com/puppetlabs/java) module.
#### Repository management
When using the repository management, the following module dependencies are required:
* General: [Puppet/elastic_stack](https://forge.puppet.com/puppet/elastic_stack)
* Debian/Ubuntu: [Puppetlabs/apt](https://forge.puppetlabs.com/puppetlabs/apt)
* openSUSE/SLES: [puppet/zypprepo](https://forge.puppetlabs.com/puppet/zypprepo)
### Beginning with Elasticsearch
Declare the top-level `elasticsearch` class (managing repositories) and set up an instance:
```puppet
include ::java
class { 'elasticsearch': }
```
## Usage
### Main class
Most top-level parameters in the `elasticsearch` class are set to reasonable defaults.
The following are some parameters that may be useful to override:
#### Install a specific version
```puppet
class { 'elasticsearch':
version => '7.9.3'
}
```
Note: This will only work when using the repository.
#### Automatically restarting the service (default set to false)
By default, the module will not restart Elasticsearch when the configuration file, package, or plugins change.
This can be overridden globally with the following option:
```puppet
class { 'elasticsearch':
restart_on_change => true
}
```
Or controlled with the more granular options: `restart_config_change`, `restart_package_change`, and `restart_plugin_change.`
#### Automatic upgrades (default set to false)
```puppet
class { 'elasticsearch':
autoupgrade => true
}
```
#### Removal/Decommissioning
```puppet
class { 'elasticsearch':
ensure => 'absent'
}
```
#### Install everything but disable service(s) afterwards
```puppet
class { 'elasticsearch':
status => 'disabled'
}
```
#### API Settings
Some resources, such as `elasticsearch::template`, require communicating with the Elasticsearch REST API.
By default, these API settings are set to:
```puppet
class { 'elasticsearch':
api_protocol => 'http',
api_host => 'localhost',
api_port => 9200,
api_timeout => 10,
api_basic_auth_username => undef,
api_basic_auth_password => undef,
api_ca_file => undef,
api_ca_path => undef,
validate_tls => true,
}
```
Each of these can be set at the top-level `elasticsearch` class and inherited for each resource or overridden on a per-resource basis.
#### Dynamically Created Resources
This module supports managing all of its defined types through top-level parameters to better support Hiera and Puppet Enterprise.
For example, to manage an index template directly from the `elasticsearch` class:
```puppet
class { 'elasticsearch':
templates => {
'logstash' => {
'content' => {
'template' => 'logstash-*',
'settings' => {
'number_of_replicas' => 0
}
}
}
}
}
```
### Plugins
This module can help manage [a variety of plugins](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html#known-plugins).
Note that `module_dir` is where the plugin will install itself to and must match that published by the plugin author; it is not where you would like to install it yourself.
#### From an official repository
```puppet
elasticsearch::plugin { 'x-pack': }
```
#### From a custom url
```puppet
elasticsearch::plugin { 'jetty':
url => 'https://oss-es-plugins.s3.amazonaws.com/elasticsearch-jetty/elasticsearch-jetty-1.2.1.zip'
}
```
#### Using a proxy
You can also use a proxy if required by setting the `proxy_host` and `proxy_port` options:
```puppet
elasticsearch::plugin { 'lmenezes/elasticsearch-kopf',
proxy_host => 'proxy.host.com',
proxy_port => 3128
}
```
Proxies that require usernames and passwords are similarly supported with the `proxy_username` and `proxy_password` parameters.
Plugin name formats that are supported include:
* `elasticsearch/plugin/version` (for official elasticsearch plugins downloaded from download.elastic.co)
* `groupId/artifactId/version` (for community plugins downloaded from maven central or OSS Sonatype)
* `username/repository` (for site plugins downloaded from github master)
#### Upgrading plugins
When you specify a certain plugin version, you can upgrade that plugin by specifying the new version.
```puppet
elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.1.1': }
```
And to upgrade, you would simply change it to
```puppet
elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.4.1': }
```
Please note that this does not work when you specify 'latest' as a version number.
#### ES 6.x and 7.x official plugins
For the Elasticsearch commercial plugins you can refer them to the simple name.
See [Plugin installation](https://www.elastic.co/guide/en/elasticsearch/plugins/current/installation.html) for more details.
### Scripts
Installs [scripts](http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting.html) to be used by Elasticsearch.
These scripts are shared across all defined instances on the same host.
```puppet
elasticsearch::script { 'myscript':
ensure => 'present',
source => 'puppet:///path/to/my/script.groovy'
}
```
Script directories can also be recursively managed for large collections of scripts:
```puppet
elasticsearch::script { 'myscripts_dir':
ensure => 'directory,
source => 'puppet:///path/to/myscripts_dir'
recurse => 'remote',
}
```
### Templates
By default templates use the top-level `elasticsearch::api_*` settings to communicate with Elasticsearch.
The following is an example of how to override these settings:
```puppet
elasticsearch::template { 'templatename':
api_protocol => 'https',
api_host => $::ipaddress,
api_port => 9201,
api_timeout => 60,
api_basic_auth_username => 'admin',
api_basic_auth_password => 'adminpassword',
api_ca_file => '/etc/ssl/certs',
api_ca_path => '/etc/pki/certs',
validate_tls => false,
source => 'puppet:///path/to/template.json',
}
```
From version 7.8 elasticserch provides a new composable templates functionality, both legacy templates api and new composable templates api are supported using different data types.
#### Data types & defines
For legacy templates `template` is used.
For composable templates `component_template` and `index_template` are used.
Both types can be used at the same time
#### Add a new template using a file
This will install and/or replace the template in Elasticsearch:
```puppet
elasticsearch::template { 'templatename':
source => 'puppet:///path/to/template.json',
}
```
#### Add a new template using content
This will install and/or replace the template in Elasticsearch:
```puppet
elasticsearch::template { 'templatename':
content => {
'template' => "*",
'settings' => {
'number_of_replicas' => 0
}
}
}
```
Plain JSON strings are also supported.
```puppet
elasticsearch::template { 'templatename':
content => '{"template":"*","settings":{"number_of_replicas":0}}'
}
```
#### Delete a template
```puppet
elasticsearch::template { 'templatename':
ensure => 'absent'
}
```
### Ingestion Pipelines
Pipelines behave similar to templates in that their contents can be controlled
over the Elasticsearch REST API with a custom Puppet resource.
API parameters follow the same rules as templates (those settings can either be
controlled at the top-level in the `elasticsearch` class or set per-resource).
#### Adding a new pipeline
This will install and/or replace an ingestion pipeline in Elasticsearch
(ingestion settings are compared against the present configuration):
```puppet
elasticsearch::pipeline { 'addfoo':
content => {
'description' => 'Add the foo field',
'processors' => [{
'set' => {
'field' => 'foo',
'value' => 'bar'
}
}]
}
}
```
#### Delete a pipeline
```puppet
elasticsearch::pipeline { 'addfoo':
ensure => 'absent'
}
```
### Index Settings
This module includes basic support for ensuring an index is present or absent
with optional index settings.
API access settings follow the pattern previously mentioned for templates.
#### Creating an index
At the time of this writing, only index settings are supported.
Note that some settings (such as `number_of_shards`) can only be set at index
creation time.
```puppet
elasticsearch::index { 'foo':
settings => {
'index' => {
'number_of_replicas' => 0
}
}
}
```
#### Delete an index
```puppet
elasticsearch::index { 'foo':
ensure => 'absent'
}
```
### Snapshot Repositories
By default snapshot_repositories use the top-level `elasticsearch::api_*` settings to communicate with Elasticsearch.
The following is an example of how to override these settings:
```puppet
elasticsearch::snapshot_repository { 'backups':
api_protocol => 'https',
api_host => $::ipaddress,
api_port => 9201,
api_timeout => 60,
api_basic_auth_username => 'admin',
api_basic_auth_password => 'adminpassword',
api_ca_file => '/etc/ssl/certs',
api_ca_path => '/etc/pki/certs',
validate_tls => false,
location => '/backups',
}
```
#### Delete a snapshot repository
```puppet
elasticsearch::snapshot_repository { 'backups':
ensure => 'absent',
location => '/backup'
}
```
### SLM (Snapshot Lifecycle Management)
By default SLM use the top-level `elasticsearch::api_*` settings to communicate with Elasticsearch.
The following is an example of how to override these settings:
```puppet
elasticsearch::slm_policy { 'policiyname':
api_protocol => 'https',
api_host => $::ipaddress,
api_port => 9201,
api_timeout => 60,
api_basic_auth_username => 'admin',
api_basic_auth_password => 'adminpassword',
api_ca_file => '/etc/ssl/certs',
api_ca_path => '/etc/pki/certs',
validate_tls => false,
source => 'puppet:///path/to/policy.json',
}
```
#### Add a new SLM policy using a file
This will install and/or replace the SLM policy in Elasticsearch:
```puppet
elasticsearch::slm_policy { 'policyname':
source => 'puppet:///path/to/policy.json',
}
```
#### Add a new SLM policy using content
This will install and/or replace ILM policy in Elasticsearch:
```puppet
elasticsearch::slm_policy { 'policyname':
content => {
name => '<backup-{now/d}>',
schedule => '0 30 1 * * ?',
repository => 'backup',
config => { },
retention => {
expire_after => '60d',
min_count => 2,
max_count => 10
}
}
}
```
### ILM (Index Lifecycle Management)
By default ILM use the top-level `elasticsearch::api_*` settings to communicate with Elasticsearch.
The following is an example of how to override these settings:
```puppet
elasticsearch::ilm_policy { 'policiyname':
api_protocol => 'https',
api_host => $::ipaddress,
api_port => 9201,
api_timeout => 60,
api_basic_auth_username => 'admin',
api_basic_auth_password => 'adminpassword',
api_ca_file => '/etc/ssl/certs',
api_ca_path => '/etc/pki/certs',
validate_tls => false,
source => 'puppet:///path/to/policy.json',
}
```
#### Add a new ILM policy using a file
This will install and/or replace the ILM policy in Elasticsearch:
```puppet
elasticsearch::ilm_policy { 'policyname':
source => 'puppet:///path/to/policy.json',
}
```
#### Add a new ILM policy using content
This will install and/or replace ILM policy in Elasticsearch:
```puppet
elasticsearch::ilm_policy { 'policyname':
content => {
policy => {
phases => {
warm => {
min_age => "7d"
actions => {
forcemerge => {
max_num_segments => 1
}
}
}
}
}
}
}
```
#### Delete an ILM policy
This will install and/or replace the ILM policy in Elasticsearch:
```puppet
elasticsearch::ilm_policy { 'policyname':
ensure => 'absent',
}
```
### Connection Validator
This module offers a way to make sure an instance has been started and is up and running before
doing a next action. This is done via the use of the `es_instance_conn_validator` resource.
```puppet
es_instance_conn_validator { 'myinstance' :
server => 'es.example.com',
port => '9200',
}
```
A common use would be for example :
```puppet
class { 'kibana4' :
require => Es_Instance_Conn_Validator['myinstance'],
}
```
### Package installation
There are two different ways of installing Elasticsearch:
#### Repository
##### Choosing an Elasticsearch major version
This module uses the `elastic/elastic_stack` module to manage package repositories. Because there is a separate repository for each major version of the Elastic stack, selecting which version to configure is necessary to change the default repository value, like this:
```puppet
class { 'elastic_stack::repo':
version => 6,
}
class { 'elasticsearch':
version => '6.8.12',
}
```
This module defaults to the upstream package repositories, which as of Elasticsearch 6.3, includes X-Pack. In order to use the purely OSS (open source) package and repository, the appropriate `oss` flag must be set on the `elastic_stack::repo` and `elasticsearch` classes:
```puppet
class { 'elastic_stack::repo':
oss => true,
}
class { 'elasticsearch':
oss => true,
}
```
##### Manual repository management
You may want to manage repositories manually. You can disable automatic repository management like this:
```puppet
class { 'elasticsearch':
manage_repo => false,
}
```
#### Remote package source
When a repository is not available or preferred you can install the packages from a remote source:
##### http/https/ftp
```puppet
class { 'elasticsearch':
package_url => 'https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.deb',
proxy_url => 'http://proxy.example.com:8080/',
}
```
Setting `proxy_url` to a location will enable download using the provided proxy
server.
This parameter is also used by `elasticsearch::plugin`.
Setting the port in the `proxy_url` is mandatory.
`proxy_url` defaults to `undef` (proxy disabled).
##### puppet://
```puppet
class { 'elasticsearch':
package_url => 'puppet:///path/to/elasticsearch-1.4.2.deb'
}
```
##### Local file
```puppet
class { 'elasticsearch':
package_url => 'file:/path/to/elasticsearch-1.4.2.deb'
}
```
### JVM Configuration
When configuring Elasticsearch's memory usage, you can modify it by setting `jvm_options`:
```puppet
class { 'elasticsearch':
jvm_options => [
'-Xms4g',
'-Xmx4g'
]
}
```
### Service management
Currently only the basic SysV-style [init](https://en.wikipedia.org/wiki/Init) and [Systemd](http://en.wikipedia.org/wiki/Systemd) service providers are supported, but other systems could be implemented as necessary (pull requests welcome).
#### Defaults File
The *defaults* file (`/etc/defaults/elasticsearch` or `/etc/sysconfig/elasticsearch`) for the Elasticsearch service can be populated as necessary.
This can either be a static file resource or a simple key value-style [hash](http://docs.puppetlabs.com/puppet/latest/reference/lang_datatypes.html#hashes) object, the latter being particularly well-suited to pulling out of a data source such as Hiera.
##### File source
```puppet
class { 'elasticsearch':
init_defaults_file => 'puppet:///path/to/defaults'
}
```
##### Hash representation
```puppet
$config_hash = {
'ES_HEAP_SIZE' => '30g',
}
class { 'elasticsearch':
init_defaults => $config_hash
}
```
Note: `init_defaults` hash can be passed to the main class and to the instance.
## Advanced features
### Security
File-based users, roles, and certificates can be managed by this module.
**Note**: If you are planning to use these features, it is *highly recommended* you read the following documentation to understand the caveats and extent of the resources available to you.
#### Roles
Roles in the file realm can be managed using the `elasticsearch::role` type.
For example, to create a role called `myrole`, you could use the following resource:
```puppet
elasticsearch::role { 'myrole':
privileges => {
'cluster' => [ 'monitor' ],
'indices' => [{
'names' => [ '*' ],
'privileges' => [ 'read' ],
}]
}
}
```
This role would grant users access to cluster monitoring and read access to all indices.
See the [Security](https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-security.html) documentation for your version to determine what `privileges` to use and how to format them (the Puppet hash representation will simply be translated into yaml.)
**Note**: The Puppet provider for `elasticsearch_user` has fine-grained control over the `roles.yml` file and thus will leave the default roles in-place.
If you would like to explicitly purge the default roles (leaving only roles managed by puppet), you can do so by including the following in your manifest:
```puppet
resources { 'elasticsearch_role':
purge => true,
}
```
##### Mappings
Associating mappings with a role for file-based management is done by passing an array of strings to the `mappings` parameter of the `elasticsearch::role` type.
For example, to define a role with mappings:
```puppet
elasticsearch::role { 'logstash':
mappings => [
'cn=group,ou=devteam',
],
privileges => {
'cluster' => 'manage_index_templates',
'indices' => [{
'names' => ['logstash-*'],
'privileges' => [
'write',
'delete',
'create_index',
],
}],
},
}
```
If you'd like to keep the mappings file purged of entries not under Puppet's control, you should use the following `resources` declaration because mappings are a separate low-level type:
```puppet
resources { 'elasticsearch_role_mapping':
purge => true,
}
```
#### Users
Users can be managed using the `elasticsearch::user` type.
For example, to create a user `mysuser` with membership in `myrole`:
```puppet
elasticsearch::user { 'myuser':
password => 'mypassword',
roles => ['myrole'],
}
```
The `password` parameter will also accept password hashes generated from the `esusers`/`users` utility and ensure the password is kept in-sync with the Shield `users` file for all Elasticsearch instances.
```puppet
elasticsearch::user { 'myuser':
password => '$2a$10$IZMnq6DF4DtQ9c4sVovgDubCbdeH62XncmcyD1sZ4WClzFuAdqspy',
roles => ['myrole'],
}
```
**Note**: When using the `esusers`/`users` provider (the default for plaintext passwords), Puppet has no way to determine whether the given password is in-sync with the password hashed by Elasticsearch.
In order to work around this, the `elasticsearch::user` resource has been designed to accept refresh events in order to update password values.
This is not ideal, but allows you to instruct the resource to change the password when needed.
For example, to update the aforementioned user's password, you could include the following your manifest:
```puppet
notify { 'update password': } ~>
elasticsearch::user { 'myuser':
password => 'mynewpassword',
roles => ['myrole'],
}
```
#### Certificates
SSL/TLS can be enabled by providing the appropriate class params with paths to the certificate and private key files, and a password for the keystore.
```puppet
class { 'elasticsearch' :
ssl => true,
ca_certificate => '/path/to/ca.pem',
certificate => '/path/to/cert.pem',
private_key => '/path/to/key.pem',
keystore_password => 'keystorepassword',
}
```
**Note**: Setting up a proper CA and certificate infrastructure is outside the scope of this documentation, see the aforementioned security guide for more information regarding the generation of these certificate files.
The module will set up a keystore file for the node to use and set the relevant options in `elasticsearch.yml` to enable TLS/SSL using the certificates and key provided.
#### System Keys
System keys can be passed to the module, where they will be placed into individual instance configuration directories.
This can be set at the `elasticsearch` class and inherited across all instances:
```puppet
class { 'elasticsearch':
system_key => 'puppet:///path/to/key',
}
```
### Licensing
If you use the aforementioned security features, you may need to install a user license to leverage particular features outside of a trial license.
This module can handle installation of licenses without the need to write custom `exec` or `curl` code to install license data.
You may instruct the module to install a license through the `elasticsearch::license` parameter:
```puppet
class { 'elasticsearch':
license => $license,
}
```
The `license` parameter will accept either a Puppet hash representation of the license file json or a plain json string that will be parsed into a native Puppet hash.
Although dependencies are automatically created to ensure that the Elasticsearch service is listening and ready before API calls are made, you may need to set the appropriate `api_*` parameters to ensure that the module can interact with the Elasticsearch API over the appropriate port, protocol, and with sufficient user rights to install the license.
The native provider for licenses will _not_ print license signatures as part of Puppet's changelog to ensure that sensitive values are not included in console output or Puppet reports.
Any fields present in the `license` parameter that differ from the license installed in a cluster will trigger a flush of the resource and new `POST` to the Elasticsearch API with the license content, though the sensitive `signature` field is not compared as it is not returned from the Elasticsearch licensing APIs.
### Data directories
There are several different ways of setting data directories for Elasticsearch.
In every case the required configuration options are placed in the `elasticsearch.yml` file.
#### Default
By default we use:
/var/lib/elasticsearch
Which mirrors the upstream defaults.
#### Single global data directory
It is possible to override the default data directory by specifying the `datadir` param:
```puppet
class { 'elasticsearch':
datadir => '/var/lib/elasticsearch-data'
}
```
#### Multiple Global data directories
It's also possible to specify multiple data directories using the `datadir` param:
```puppet
class { 'elasticsearch':
datadir => [ '/var/lib/es-data1', '/var/lib/es-data2']
}
```
See [the Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#max-local-storage-nodes) for additional information regarding this configuration.
### Elasticsearch configuration
The `config` option can be used to provide additional configuration options to Elasticsearch.
#### Configuration writeup
The `config` hash can be written in 2 different ways:
##### Full hash writeup
Instead of writing the full hash representation:
```puppet
class { 'elasticsearch':
config => {
'cluster' => {
'name' => 'ClusterName',
'routing' => {
'allocation' => {
'awareness' => {
'attributes' => 'rack'
}
}
}
}
}
}
```
##### Short hash writeup
```puppet
class { 'elasticsearch':
config => {
'cluster' => {
'name' => 'ClusterName',
'routing.allocation.awareness.attributes' => 'rack'
}
}
}
```
#### Keystore Settings
Recent versions of Elasticsearch include the [elasticsearch-keystore](https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-settings.html) utility to create and manage the `elasticsearch.keystore` file which can store sensitive values for certain settings.
The settings and values for this file can be controlled by this module.
Settings follow the behavior of the `config` parameter for the top-level Elasticsearch class and `elasticsearch::instance` defined types.
That is, you may define keystore settings globally, and all values will be merged with instance-specific settings for final inclusion in the `elasticsearch.keystore` file.
Note that each hash key is passed to the `elasticsearch-keystore` utility in a straightforward manner, so you should specify the hash passed to `secrets` in flattened form (that is, without full nested hash representation).
For example, to define cloud plugin credentials for all instances:
```puppet
class { 'elasticsearch':
secrets => {
'cloud.aws.access_key' => 'AKIA....',
'cloud.aws.secret_key' => 'AKIA....',
}
}
```
##### Purging Secrets
By default, if a secret setting exists on-disk that is not present in the `secrets` hash, this module will leave it intact.
If you prefer to keep only secrets in the keystore that are specified in the `secrets` hash, use the `purge_secrets` boolean parameter either on the `elasticsearch` class to set it globally or per-instance.
##### Notifying Services
Any changes to keystore secrets will notify running elasticsearch services by respecting the `restart_on_change` and `restart_config_change` parameters.
## Reference
Class parameters are available in [the auto-generated documentation
pages](https://elastic.github.io/puppet-elasticsearch/puppet_classes/elasticsearch.html).
Autogenerated documentation for types, providers, and ruby helpers is also
available on the same documentation site.
## Limitations
This module is built upon and tested against the versions of Puppet listed in
the metadata.json file (i.e. the listed compatible versions on the Puppet
Forge).
The module has been tested on:
* Amazon Linux 1/2
* Debian 8/9/10
* CentOS 7/8
* OracleLinux 7/8
* Ubuntu 16.04, 18.04, 20.04
* SLES 12
Testing on other platforms has been light and cannot be guaranteed.
## Development
Please see the [CONTRIBUTING.md](https://github.com/voxpupuli/puppet-elasticsearch/blob/master/.github/CONTRIBUTING.md) file for instructions regarding development environments and testing.
## Support
The Puppet Elasticsearch module is community supported and not officially supported by Elastic Support.
## Transfer Notice
This module was originally authored by [Elastic](https://www.elastic.co).
The maintainer preferred that Vox Pupuli take ownership of the module for future improvement and maintenance.
Existing pull requests and issues were transferred over, please fork and continue to contribute here instead of Elastic.
|