1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
|
[](https://circleci.com/gh/prometheus-community/smartctl_exporter)
[](https://quay.io/repository/prometheuscommunity/smartctl-exporter)
# smartctl_exporter
Export smartctl statistics to prometheus
Example output you can show in [EXAMPLE.md](EXAMPLE.md)
## Need more?
**If you need additional metrics - contact me :)**
**Create a feature request, describe the metric that you would like to have and
attach exported from smartctl json file**
# Requirements
`smartmontools` >= 7.0, because export to json [released in 7.0](https://www.smartmontools.org/browser/tags/RELEASE_7_0/smartmontools/NEWS#L11)
# Configuration
## Command line options
The exporter will scan the system for available devices if no `--smartctl.device`
flags are used.
```
usage: smartctl_exporter [<flags>]
Flags:
-h, --help Show context-sensitive help (also try --help-long and --help-man).
--smartctl.path="/usr/sbin/smartctl"
The path to the smartctl binary
--smartctl.interval=60s The interval between smartctl polls
--smartctl.rescan=10m The interval between rescanning for new/disappeared devices. If the interval is smaller than 1s no
rescanning takes place. If any devices are configured with smartctl.device also no rescanning takes
place.
--smartctl.device=SMARTCTL.DEVICE ...
The device to monitor (repeatable)
--smartctl.device-exclude=""
Regexp of devices to exclude from automatic scanning. (mutually exclusive to
device-include)
--smartctl.device-include=""
Regexp of devices to include in automatic scanning. (mutually exclusive to
device-exclude)
--web.telemetry-path="/metrics"
Path under which to expose metrics
--web.systemd-socket Use systemd socket activation listeners instead of port listeners (Linux only).
--web.listen-address=:9633 ...
Addresses on which to expose metrics and web interface. Repeatable for multiple
addresses.
--web.config.file="" [EXPERIMENTAL] Path to configuration file that can enable TLS or authentication.
--log.level=info Only log messages with the given severity or above. One of: [debug, info, warn,
error]
--log.format=logfmt Output format of log messages. One of: [logfmt, json]
--version Show application version.
```
## TLS and basic authentication
This exporter supports TLS and basic authentication.
To use TLS and/or basic authentication, you need to pass a configuration file
using the `--web.config.file` parameter. The format of the file is described
[in the exporter-toolkit repository](https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md).
## Example of running in Docker
Minimal functional `docker-compose.yml`:
```yaml
version: "3"
services:
smartctl-exporter:
image: prometheuscommunity/smartctl-exporter
privileged: true
user: root
ports:
- "9633:9633"
```
# Troubleshooting
## Troubleshooting data inconsistencies
`smartmon_exporter` uses the JSON output from `smartctl` to provide the data to
Prometheus. If the data is incorrect, look at the data from `smartctl` to
determine if the issue should be reported upstream to smartmontools or to this
repo. In general, the `smartctl_exporter` should not modify the data in flight.
If the data is missing from `smartctl`, it should not be in `smartctl_exporter`.
If the data from `smartctl` is incorrect, it should be reported upstream.
Requests for `smartctl_exporter` to "fix" incorrect data where `smartctl` is
reporting incorrect data will be closed. The grey area is when invalid or
missing data from smartctl is causing multiple invalid or incorrect data
in `smartctl_exporter`. This could happen if the data is used in a calculation
for other data. This will need to be researched on a case by case basis.
| - | smartctl valid | smartctl missing | smartctl invalid/incorrect |
|---------------------------|-----------------------------|-------------------------------------------------|----------------------------------|
| smartctl_exporter valid | all good | N/A | N/A |
| smartctl_exporter missing | issue for smartctl_exporter | report upstream to smartmontools | report upstream to smartmontools |
| smartctl_exporter invalid | issue for smartctl_exporter | issue for smartctl_exporter and report upstream | report upstream to smartmontools |
### smartctl output vs smartctl_exporter output
The S.M.A.R.T. attributes are mapped in
[smartctl.go](https://github.com/prometheus-community/smartctl_exporter/blob/master/smartctl.go).
Each function has a `prometheus.MustNewConstMetric` or similar function with the
first parameter being the metric name. Find the metric name in
[metrics.go](https://github.com/prometheus-community/smartctl_exporter/blob/master/metrics.go)
to see how the exporter displays the information. This may sound technical, but
it's crucial for understanding how data flows from `smartctl` to
`smartctl_exporter` to Prometheus.
If the data looks incorrect, check the
[Smartmontools Frequently Asked Questions (FAQ)](https://www.smartmontools.org/wiki/FAQ).
It's likely your question may already have an answer. If you still have
questions, open an [issue]().
## Gathering smartctl data
Follow these steps to gather smartctl data for troubleshooting purposes. If you
have unique drives/data/edge cases and would like to "donate" the data, open a
PR with the redacted JSON files.
1. Run `scripts/collect-smartctl-json.sh` to export all drives to a
`smartctl-data` directory (created in the current directory).
2. Run `scripts/redact_fake_json.py` to redact sensitive data.
3. Provide the JSON file for the drive in question.
```bash
cd scripts
./collect-smartctl-json.sh
./redact-fake-json.py smartctl-data/*.json
```
## Run smartctl_exporter using JSON data
The `smartctl_exporter` can be run using local JSON data. The device names are
pulled from actual devices in the machine while the data is redirected to the
`debug` directory. Save the JSON data in the `debug` directory using the actual
device names using a 1:1 ratio. If you have 3 devices, `sda`, `sdb` and `sdc`,
the `smartctl_exporter` will expect 3 files: `debug/sda.json`, `debug/sdb.json`
and `debug/sdc.json`.
Once the "fake devices" (JSON files) are in place, run the exporter passing the
hidden `--smartctl.fake-data` switch on the command line. The port is specified
to prevent conflicts with an existing `smartctl_exporter` on the default port.
```bash
smartctl_exporter --web.listen-address 127.0.0.1:19633 --smartctl.fake-data
```
# FAQ
## How do I run `smartctl_exporter` against a JSON file?
If you're helping someone else, request the output of the `smartctl` command
above. Feed this into the `smartctl_exporter` using the
hidden `--smartctl.fake-data` flag. If a `smartctl_exporter` is already running,
use a different port; in this case, it's `19633`. Run `collect_fake_json.sh`
first to collect the JSON files for **your** devices. Copy the requested JSON
file into one of the fake files. After starting the exporter, you can query it
to see the data generated.
```bash
# Dump the JSON files for your devices into debug/
./collect_fake_json.sh
# copy the test JSON into one of the files in debug/
cp extracted-from-above-sda.json debug/sda.json
# Make sure you have the latest version
go build
# Use a different port in case smartctl_exporter is already running
sudo ./smartctl_exporter --web.listen-address=127.0.0.1:19633 --log.level=debug --smartctl.fake-data
# Use curl with grep
curl --silent 127.0.0.1:19633/metrics | grep -i nvme
# Or xh with ripgrep
xh --body :19633/metrics | rg nvme
```
## Why is root required? Can't I add a user to the "disk" group?
A blogger had the same question and opened a ticket on smartmontools. This is
their response. `smartctl` needs to be run as root.
[RFE: add O_RDRW mode for sat/scsi/ata devices](https://www.smartmontools.org/ticket/1064)
> According to function `blk_verify_command()` from current kernel sources
> (see [block/scsi_ioctl.c](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/block/scsi_ioctl.c)),
> O_RDONLY or O_RDWR make no difference if device was opened as root (or with
> CAP_SYS_RAWIO).
>
> The SCSI commands listed in function `blk_set_cmd_filter_defaults()` show
> that some of the `smartctl -d scsi` functionality might work with O_RDONLY
> for non-root users. Some more might work with O_RDWR.
>
> But `smartctl -d sat` (to access SATA devices) won't work at all because the
> SCSI commands ATA_12 and ATA_16
> (see [scsi_proto.h](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/scsi/scsi_proto.h))
> are **always blocked for non-root users**.
## What about my NVMe drive?
From the smartmontools FAQ: [My NVMe drive is not in the smartctl/smartd database](https://www.smartmontools.org/wiki/FAQ#MyNVMedriveisnotinthesmartctlsmartddatabase)
> SCSI/SAS and NVMe drives do not provide ATA/SATA-like SMART Attributes.
> Therefore the drive database does not contain any entries for these drives.
> This may change in the future as some drives provide similar info via vendor
> specific commands (see ticket #870).
smartmontools also has a [wiki page for NVMe](https://www.smartmontools.org/wiki/NVMe_Support) devices.
## How do I report upstream to smartmontools?
Check their FAQ: [How to create a bug report](https://www.smartmontools.org/wiki/FAQ#Howtocreateabugreport).
|
