1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
|
lnav v0.8.4:
Features:
* Added the ':comment' command that can be used to attach a comment to a
log line. The comment will be displayed below the line, like so:
2017-01-01T15:30:00 error: computer is on fire
+ This is where it all went wrong
The ':clear-comment' command will remove the attached comment. Comments
are searchable with the standard search mechanism and they are available
in SQL through the "log_comment" column.
* Added the ':tag', ':untag', and ':delete-tags' commands that can be used
to attach/detach tags on the top log line and delete all instances of
a tag. Tags are also searchable and are available in SQL as a JSON
array in the "log_tags" column.
* Pressing left-arrow while viewing log messages will reveal the source
file name for each line and the unique parts of the source path.
Pressing again will reveal the full path.
* The file name section of the top status line will show only the unique
parts of the log file path if there is not enough room to show the full
path.
* Added the ":hide-unmarked-lines" and ":show-unmarked-lines" commands
that hide/show lines based on whether they are bookmarked.
* Added the "json_contains()" SQL function to check if a JSON value
contains a number of a string.
* The relative time parser recognizes "next" at the beginning of the
input, for example, "next hour" or "next day". Handy for use in the
":goto" command.
* Added a "text-transform" option for formatting JSON log messages. The
supported options are: none, uppercase, lowercase, and capitalize.
* Added a special "__level__" field name for formatting JSON messages so
that the lnav level name can be used instead of the internal value in
the JSON object.
* Added a log format for journald JSON logs.
Interface Changes:
* When typing in a search, instead of moving the view to the first match
that was found, the first ten matches will be displayed in the preview
window.
* The pretty-print view maintains highlighting from the log view.
* The pretty-print view no longer tries to reverse lookup IP addresses.
* The online help for commands and SQL functions now includes a 'See Also'
section that lists related commands/functions.
Fixes:
* The HOME key should now work in the command-prompt and move the cursor
to the beginning of the line.
* The :delete-filter command should now tab-complete existing filters.
* Milliseconds can now be used in relative times (e.g. 10:00:00.123)
* The J/K hotkeys were not marking lines correctly when the bottom of
the view was reached.
* The level field in JSON logs should now be recognized by the level
patterns in the format.
lnav v0.8.3:
Features:
* Support for the Bro Network Security Monitor (https://www.bro.org) log
file format.
* Added an fstat() table-valued function for querying the local
filesystem.
* Added readlink() and realpath() SQL functions.
* Highlights specified in log formats can now specify the colors to use
for the highlighted parts of the log message.
* Added a ":quit" command.
* Added a "/ui/default-colors" configuration option to specify that the
terminal's default background and foreground colors should be used
instead of black and white.
Interface Changes:
* Pressing delete at a command-prompt will exit the prompt if there is no
other input.
Fixes:
* The help view now includes all the command-help that would pop up as
you entered commands and SQL queries.
* Hidden fields and lines hidden before/after times are now saved in the
current session and restored.
* Unicode characters should now be displayed correctly (make sure you
have LANG set to a UTF-8 locale).
lnav v0.8.2:
Features:
* The timestamp format for JSON log files can be specified with the
"timestamp-format" option in the "line-format" array.
* Added "min-width", "max-width", "align", and "overflow" options to the
"line-format" in format definitions for JSON log files. These options
give you more control over how the displayed line looks.
* Added a "hidden" option to log format values so that you can hide JSON
log fields from being displayed if they are not in the line format.
* Added a "rewriter" field to log format value definitions that is a
command used to rewrite the field in the pretty-printed version of a
log message. For example, the HTTP access log format will rewrite the
status code field to include the textual version (e.g. 200 (OK)).
* Log message fields can now be hidden using the :hide-fields" command or
by setting the 'hidden' property in the log format. When hidden, the
fields will be replaced with a yellow ellipsis when displayed. Hiding
large fields that contain extra details can make the log easier to read.
The 'x' hotkey can be used to quickly toggle whether these fields are
displayed or not.
* Added a ':mark' command to bookmark the top line in the current view.
* Added an ':alt-msg' command that can be used to set the text to be
displayed in the bottom right of the command line. This command is
mostly intended for use by hotkey maps to set the help text.
* In lnav scripts, the first row of a SQL query result will now be turned
into local variables that can be referenced in other commands or
queries. For example, the following script will print the number one:
;SELECT 1 as foobar
:eval :echo ${foobar}
* Added an 'lnav_view_stack' SQL table that gives access to the view
stack.
* Added a 'top_time' column to the lnav_views table so that you can get
the timestamp for the top line in views that are time-based as well as
allowing you to move the view to a given time with an UPDATE statement.
* Added a 'search' column to the lnav_views table so that you can perform
a text search programmatically.
* Added a 'regexp_capture(<string>, <pattern>)' table-valued function for
getting detailed results from matching a regular expression against a
string.
* Added a 'timediff(<time1>, <time2>)' SQL function for computing the
difference between two relative or absolute timestamps.
* Log formats can now define a default set of highlights with the
"highlights" property.
* Added a '|search-for <pattern>' built-in script that can be used to
start a search from the command-line.
* Log format definitions can now specify the expected log level for a
sample line. This check should make it easier to validate the
definition.
Interface Changes:
* Command and SQL documentation is now displayed in a section at the
bottom of the screen when a command or query is being entered. Some
commands will also display a preview of the command results. For
example, the ':open' command will display the first ten lines of the
file to be opened and the ':filter-out' command will highlight text
that matches in the current view. The preview pane can be shown/hidden
by pressing CTRL-P.
* The color used for text colored via ":highlight" is now based on the
the regex instead of randomly picked so that colors are consistent
across invocations.
* The "graph" view has been removed since it's functionality has been
obsoleted by other features, like ":create-search-table".
* When doing a search, if a hit is found within a second after hitting
<ENTER>, the view will move to the matched line. The previous behavior
was to stay on the current line, which tended to be a surprise to new
users.
* Pressing 'n'/'N' to move through the next/previous search hit will now
skip adjacent lines, up to the vertical size of the view. This should
make scanning through clusters of hits much faster. Repeatedly
pressing these keys within a short time will also accelerate scanning
by moving the view at least a full page at a time.
Breaking Changes:
* The captured timestamp text in log files must fully match a known format
or an error will be reported. The previous behavior was to ignore any
text at the end of the line.
Fixes:
* You can now execute commands from the standard input by using a dash (-)
with the '-f' command-line argument. Reading commands from a file
descriptor should also work, for example, with the following bash
syntax:
$ lnav -f <(echo :open the-file-to-open)
* Programming language syntax highlighting should now only be applied to
source code files instead of everywhere.
lnav v0.8.1:
Features:
* Added a spectrogram command and view that displays the values of a
numeric field over time. The view works for log message fields or
for database result columns.
* Log formats can now create SQL views and execute other statements
by adding '.sql' files to their format directories. The SQL scripts
will be executed on startup.
* Added 'json_group_object' and 'json_group_array' aggregate SQL
functions that collects values from a GROUP BY query into a JSON
object or array, respectively.
* The SQL view will now graph values found in JSON objects/arrays in
addition to the regular columns in the result.
* Added an 'regexp_match(<re>, <str>)' SQL function that can be used to
extract values from a string using a regular expression.
* Added an 'extract(<str>)' SQL function that extracts values using the
same data discover/extraction parser used in the 'logline' table.
* Added a "summary" overlay line to the bottom of the log view that
displays how long ago the last message was received, along with the
total number of files and the error rate over the past five minutes.
* Pressing 'V' in the DB view will now check for a column with a
timestamp and move to the corresponding time in the log view.
* Added 'a/A' hotkeys to restore a view previously popped with 'q/Q'.
* Added ":hide-lines-before", ":hide-lines-after", and
":show-lines-before-and-after" commands so that you can filter out
log lines based on time.
* Scripts containing lnav commands/queries can now be executed using
the pipe ('|') hotkey. See the documentation for more information.
* Added an ":eval" command that can be used to execute a command or
query after performing environment variable substitution.
* Added an ":echo" command that can be useful for scripts to message
the user.
* The "log_part" column can now be set with an SQL UPDATE statement.
* Added a "log_body" hidden column that returns the body of the log
message.
* Added ":config", ":reset-config", and ":save-config" commands to change
configuration options, reset to default, and save them for future
executions.
* Added a "/ui/clock-format" configuration option that controls the time
format in the top-left corner.
* Added a "/ui/dim-text" configuration option that controls the brightness
of text in the UI.
* Added support for TAI64 timestamps (http://cr.yp.to/libtai/tai64.html).
* Added a safe execution mode. If the 'LNAVSECURE' environment variable is
set before executing lnav, the following commands are disabled:
- open
- pipe-to
- pipe-line-to
- write-*-to
This makes it easier to run lnav with escalated privileges in restricted
environments, without the risk of users being able to use the above
mentioned commands to gain privileged access.
Interface Changes:
* The 'o/O' hotkeys have been reassigned to navigate through log
messages that have a matching "opid" field. The old action of
moving forward and backward by 60 minutes can be simulated by
using the ':goto' command with a relative time and the 'r/R'
hotkeys.
* Log messages with timestamps that pre-date previous log messages will
have the timestamp highlighted in yellow and underlined. These out-
of-time-order messages will be assigned the time of the previous
message for sorting purposes. You can press the 'p' hotkey to examine
the 'Received Time' of the message as well as the time parsed from the
original message. A "log_actual_time" hidden field has also been
added to the SQLite virtual table so you can operate on the original
message time from the file.
* The 'A/B' hotkeys for moving forward/backward by 10% line increments
have been reassigned to '[' and ']'. The 'a' and 'A' hotkeys are now
used to return to the previously popped view while trying to preserve
the time range. For example, after leaving the spectrogram view with
'q', you can press 'A' return to the view with the top time in the
spectrogram matching the top time in the log view.
* The 'Q' hotkey now pops the current view off of the stack while
maintaining the top time between views.
Fixes:
* Issues with tailing JSON logs have been fixed.
* The jget() SQL function should now work for objects nested in arrays.
lnav v0.8.0:
Features:
* Integration with "papertrailapp.com" for querying and tailing
server log and syslog messages. See the Papertrail section in
the online help for more details.
* Remote files can be opened when lnav is built with libcurl v7.23.0+
* SQL queries can now be done on lines that match a regular expression
using the "log_search" table or by creating custom tables with the
":create-search-table" command.
* Log formats that are "containers" for other log formats, like
syslog, are now supported. See the online help for more
information.
* Formats can be installed from git repositories using the '-i' option.
A standard set of extra formats can be installed by doing
'lnav -i extra'. (You must have git installed for this to work.)
* Added support for 'VMware vSphere Auto Deploy' log format.
* Added a 'sudo' log format.
* Added hotkeys to move left/right by a smaller increment (H/L or
Shift+Left/Shift+Right).
* A color-coded bar has been added to the left side to show where
messages from one file stop and messages from another file start.
* The '-C' option will now try to check any specified log files to
make sure the format(s) match all of the lines.
* Added an "all_logs" SQLite table that contains the message format
extracted from each log line. Also added a ';.msgformat' SQL command
that executes a query that returns the counts for each format and the
first line where the format was seen.
* Added an "lnav_views" SQLite table that can be used to query and
change the lnav view state.
* When typing in a command, the status bar will display a short
summary of the currently entered command.
* Added a "delete-filter" command.
* Added a "log_msg_instance" column to the logline and log_search
tables to make it easier to join tables that are matching log
messages that are ordered.
* Added a "timeslice()" function to SQLite so that it is easier to
group log messages by time buckets.
* The 'goto' command now supports relative time values like
'a minute ago', 'an hour later', and many more.
Interface Changes:
* The 'r/R' hotkeys have been reassigned to navigate through the log
messages by the relative time value that was last used with the
'goto' command.
Fixes:
* The pretty-print view should now work for text files.
* Nested fields in JSON logs are now supported for levels, bodies, etc...
* Tab-completion should work for quoted SQL identifiers.
* 'lo-fi' mode key shortcut changed to CTRL+L.
* 'redraw' shortcut removed. Relegated to just a command.
* Fixed lnav hang in pretty-print mode while doing a dns lookup.
* The generic log message parser used to extract data has been
optimized and should be a bit faster.
lnav v0.7.3:
Features:
* Add 'pipe-to' and 'pipe-line-to' commands that pipe the currently
marked lines or the current log message to a shell command,
respectively.
* Added a "pretty-print" view (P hotkey) that tries to reformat log
messages so that they are easier to read.
* Added a 'redraw' command (CTRL+L hotkey) to redraw the window in
case it has been corrupted.
* Added a 'relative-goto' command to move the current view relative
to its current position.
* Experimental support for linking with jemalloc.
* The plain text view now supports filtering.
* Added 'next-mark' and 'prev-mark' commands to jump to the next or
previous bookmarked line (e.g. error, warning, ...)
* Added a 'zoom-to' command to change the zoom level of the histogram
view.
* Log formats can now define their own timestamp formats with the
'timestamp-format' field.
Fixes:
* Autotools scripts overhaul.
* Added a configure option to disable linking with libtinfo. The newer
versions of ncurses don't require it, however the build silently pulls
it in as a dependency, if it is available on the system. This can be
explicitly disabled using the '--disable-tinfo' option during configure.
* Fixed the configure script behavior to ignore the values specified using
the CFLAGS and LDFLAGS environment variables while searching for sqlite3
when '--with-sqlite3' switch was specified without the prefix.
* The configure script now recognizes libeditline symlink'ed to masquerade
as libreadline. This previously used to cause problems at compile time,
specially on OS X. If you come across this error, use the
'--with-readline=prefix' switch to specify the path to the correct
location of libreadline.
* The order that log formats are tried against a log file is now
automatically determined so that more specific formats are tested
before more general ones. The order is determined on startup based on
how each format matches each other formats sample lines.
* Command files (i.e. those executed via the '-f' flag) now support
commands/queries that span more than one line.
* Added more log levels: stats, debug2 - debug5.
lnav v0.7.2:
* Added log formats for vdsm, openstack, and the vmkernel.
* Added a "lo-fi" mode (L hotkey) that dumps the displayed log lines
to the terminal without any decorations. The write-to, write-json-to,
and write-csv-to commands will also write their output to the terminal
when passed '-' as the file name. This mode can be useful for copying
plain text lines to the clipboard.
* (OS X) Text search strings are copied to the system's "find" clipboard.
Also, when starting a new search, the current value in the "find"
clipboard can be tab-completed.
lnav v0.7.1:
Features:
* Added an 'environ' SQL table that reflects lnav's environment
variables. The table can be read and written to using SQL
SELECT, INSERT, UPDATE, and DELETE statements. Setting variables
can be a way to use SQL query results in lnav commands.
* Added a 'jget' SQLite function that can extract fields from a JSON-
encoded value.
* Added log formats for the OpenAM identity provider.
* Added a 'clear-highlight' command to clear previous calls to the
'highlight' command.
* Fixed some performance bugs in indexing JSON log formats. Loading
times should be at least five times faster.
* Filtering performance should be improved so that enabling/disabling
filters should be almost instantaneous.
* The filter-in, filter-out, and highlight commands now support
tab-completion of text in the current view.
* Add a '-i' flag that installs format files in: ~/.lnav/formats/installed
lnav v0.7.0:
Features:
* Add the '.schema' SQL command to open a view that displays the schema
for the internal tables and any attached databases. If lnav was only
executed with a SQLite database and no text files, this view will open
by default.
* The scroll bar now indicates the location of errors/warnings, search
hits, and bookmarks.
* The xterm title is update to reflect the file name for the top line
in the view.
* Added a "headless" mode so that you can execute commands and run SQL
queries from the command-line without having to do it from the curses
UI.
* When doing a search or SQL query, any text that is currently being
displayed can be tab-completed.
* The '-H' option was added so you can view the internal help text.
* Added the 'g/G' hotkeys to move to the top/bottom of the file.
* Added a 'log_mark' column to the log tables that indicates whether or
not a log message is bookmarked. The field is writable, so you can
bookmark lines using an SQL UPDATE query.
* Added syntax-highlighting when editing SQL queries or search regexes.
* Added a "write-json-to" command that writes the result of a SQL query
to a JSON-formatted file.
* The "elapsed time" column now uses red/green coloring to indicate
sharp changes in the message rate.
* Added a "set-min-log-level" command to filter out log messages that
are below a given level.
Fixes:
* Performance improvements.
* Multi-line filtering has been fixed.
* A collator has been added to the log_level column in the log tables
so that you can write expressions like "log_level > 'warning'".
* The log_time datetime format now matches what is returned by
"datetime('now')" so that collating works correctly.
* If a search string is not valid PCRE syntax, a search is done for
the exact string instead of just returning an error.
* Static-linking has been cleaned up.
* OpenSSL is no longer a requirement.
* Alpha support for Windows/cygwin.
* Environment variables can now be accessed in SQL queries using
the syntax: $VAR_NAME
* An internal log is kept and written out on a crash.
* Partition bookmarks are now tracked separately from regular user
bookmarks. You can start a partition with the 'partition-name'
command and remove it with the 'clear-partition' command.
* Improved display of possible matches during tab-completion in the
command-prompt. The matches are now shown in a separate view and
pressing tab repeatedly will scroll through the view.
* The "open" command now does shell word expansion for file names.
* More config directory paths have been added: /etc/lnav,
$prefix/etc/lnav, and directories passed on the command-line with -I.
lnav v0.6.2:
Features:
* Word-wrap support.
Fixes:
* Fix some OS X Mavericks build/runtime issues.
lnav v0.6.1:
Features:
* Support for JSON-encoded log files.
Fixes:
* Some minor fixes and performance improvements.
lnav v0.6.0:
Features:
* Custom log formats and more builtin formats
* Automatic extraction of data from logs
* UI improvements, support for 256 color terminals
lnav v0.5.1:
Features:
* Added the '-t' and '-w' options which can be used to prepend a
timestamp to any data piped in on stdin and to specify a file to
write the contents of stdin to.
Fixes:
* Cleanup for packaging.
lnav v0.5.0:
Features:
* Files can be specified on the command-line using wildcards so that
new files are automatically loaded. Directories can also be passed
as command-line arguments to read all of the files in the directory.
* Builds on cygwin again.
* Added the 'C' hotkey to clear any existing user bookmarks.
* Added experimental support for accepting input from mice.
Fixes:
* Internal cleanup.
* Copying to the clipboard on OS X is now supported.
* Many bug fixes.
lnav v0.4.0:
Features:
* Files that are not recognized as containing log messages have been
broken out to a separate text files view. You can flip between the
log view and the text file view with the 't' hotkey. When viewing
text files, the 'f' hotkey will switch between files.
* Files compressed with bzip2 are recognized and decompressed on the
fly.
* Added a "session" file and command for storing commands that should
be executed on startup. For example, if you always want some
highlighting to be done, you can add that command to the session
file.
Fixes:
* Add some more log file formats for generic log files.
* Performance improvements for compressed files.
* Works on OS X now.
lnav v0.3.0:
Changes:
* The hotkey for the SQL view was changed to 'v' and 'V' from '.'.
Features:
* You can now switch between the SQL result view and the log view while
keeping the top of the views in sync with the "log_line" column.
Fixes:
* The "log_line" column is no longer included in the SQL result view's
stacked bar graph.
* Added a "warnings" count to the histogram view.
|
