1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
|
---
stage: Monitor
group: Platform Insights
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Metrics
DETAILS:
**Tier:** Ultimate
**Offering:** GitLab.com, Self-managed
**Status:** Beta
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124966) in GitLab 16.7 [with a flag](../administration/feature_flags.md) named `observability_metrics`. Disabled by default. This feature is an [experiment](../policy/experiment-beta-support.md#experiment).
> - Feature flag [changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/158786) in GitLab 17.3 to the `observability_features` [feature flag](../administration/feature_flags.md), disabled by default. The previous feature flag (`observability_metrics`) was removed.
> - [Introduced](https://gitlab.com/groups/gitlab-org/opstrace/-/epics/100) for self-managed in GitLab 17.3.
FLAG:
The availability of this feature is controlled by a feature flag.
For more information, see the history.
This feature is available for testing, but not ready for production use.
Metrics provide insight about the operational health of monitored systems.
Use metrics to learn more about your systems and applications in a given time range.
Metrics are structured as time series data, and are:
- Indexed by timestamp
- Continuously expanding as additional data is gathered
- Usually aggregated, downsampled, and queried by range
- Have write-intensive requirements
## Configure metrics
Configure metrics to enable them for a project.
Prerequisites:
You must have at least the Maintainer role for the project.
1. Create an access token:
1. On the left sidebar, select **Search or go to** and find your project.
1. Select **Settings > Access tokens**.
1. Create an access token with the `api` scope and **Developer** role or greater.
Save the access token value for later.
1. To configure your application to send GitLab metrics, set the following environment variables:
```shell
OTEL_EXPORTER = "otlphttp"
OTEL_EXPORTER_OTLP_ENDPOINT = "https://gitlab.example.com/api/v4/projects/<gitlab-project-id>/observability/"
OTEL_EXPORTER_OTLP_HEADERS = "PRIVATE-TOKEN=<gitlab-access-token>"
```
Use the following values:
- `gitlab.example.com` - The hostname for your self-managed instance, or `gitlab.com`
- `gitlab-project-id` - The project ID
- `gitlab-access-token` - The access token you created
Metrics are configured for your project.
When you run your application, the OpenTelemetry exporter sends metrics to GitLab.
## View metrics
You can view the metrics for a given project:
1. On the left sidebar, select **Search or go to** and find your project.
1. Select **Monitor > Metrics**.
A list of metrics is displayed.
Select a metric to view its details.
Each metric contains one or more attributes. You can filter
metrics by attribute with the search bar.
### Metric details
Metrics are displayed as either a sum, a gauge, or a histogram.
The metric details page displays a chart depending on the type of metric.
On the metric details page, you can also view metrics for a specific time range, and
aggregate metrics by attribute:
To make data lookups fast, depending on what time period you filter by,
GitLab automatically chooses the proper aggregation.
For example, if you search for more than seven days of data, the API returns only daily aggregates.
### Aggregations by search period
The following table shows what type of aggregation is used for each search period:
|Period|Aggregation used|
|---|---|
| Less than 30 minutes | Raw data as ingested |
| More than 30 minutes and less than one hour | By minute |
| More than one hour and less than 72 hours | Hourly |
| More than 72 hours | Daily |
### Metrics ingestion limits
Metrics ingest a maximum of 102,400 bytes per minute.
When the limit is exceeded, a `429 Too Many Requests` response is returned.
To request a limit increase to 1,048,576 bytes per minute, contact [GitLab support](https://about.gitlab.com/support/).
### Data retention
GitLab has a retention limit of 30 days for all ingested metrics.
### Create an issue for a metric
You can create an issue to track any action taken to resolve or investigate a metric. To create an issue for a metric:
1. On the left sidebar, select **Search or go to** and find your project.
1. Select **Monitor > Metrics**.
1. From the list of metrics, select a metric.
1. Select **Create issue**.
The issue is created in the selected project and pre-filled with information from the metric.
You can edit the issue title and description.
### View issues related to a metric
1. On the left sidebar, select **Search or go to** and find your project.
1. Select **Monitor > Metrics**.
1. From the list of metrics, select a metric.
1. Scroll to **Related issues**.
1. Optional. To view the issue details, select an issue.
|
