File: README-Memory

package info (click to toggle)
libdmapsharing 2.9.39-4
  • links: PTS, VCS
  • area: main
  • in suites: buster
  • size: 3,640 kB
  • sloc: ansic: 15,629; sh: 4,443; makefile: 444; xml: 53
file content (88 lines) | stat: -rw-r--r-- 3,614 bytes parent folder | download | duplicates (5)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
This documents the effort to make dmapd and libdmapsharing more memory
efficient. This document describes work done in both libdmapsharing and
dmapd, because dmapd is the primary application of libdmapsharing's
server-side functionality.

I found that dmapd used a significant amount of heap space while trying to
port dmapd to OpenWRT on a WRT160NL router with 32 Megabytes of RAM. A
2,419-song library caused dmapd on x86_64 to use 7,413,760 bytes of heap
space after reading the library into memory. Furthermore, connecting to
dmapd using a DAAP client and receiving a list of songs caused heap usage
to increase to over 10 Megabytes. Note that this memory usage might be
smaller on the WRT160NL due to its 32-bit architecture, but it is still
significant in size.

After attempting to optimize memory use, I measured the following on
OpenWrt / WRT160NL (*** BUT WITH NO METADATA READ! ***):

Heap, after music is loaded:		0xf5000  (1,003,520)
Heap, after client received song list:	0x926000 (9,592,832)

SOME METADATA (MP3 ONLY):

Heap, after music is loaded:		0x2d3000 (2,961,408)
Heap, after client received song list:	0xafa000 (11,509,760)

I use the following tools to troubleshoot memory issues:

	grep heap /proc/PID/maps	(current heap size)
	valgrind / memcheck		(memory leaks)
	valgrind / massif		(max heap size and memory over time)
	gdb
	objdump -t OBJECT-FILE | grep -e "\.bss" -e "\.data" | sort -k5 
		| c++filt | tail

I have made the following changes in order to make dmapd more memory
efficient:

1. Replace the hash table-based DMAPDb implementation with a Berkeley
Database implementation. Currently measuring the memory use compared to
the in-memory database.

2. Fixed several memory leaks after analyzing with valgrind.

	Heap without client connection: 3,940,352 bytes

3. Implement database cache so that dmapd does not need to use GStreamer
to read media metadata each time it is run.

	Heap without client connection using DB cache: 3,346,432 bytes

4. Replace the DMAPDb in the DMAPContainerRecord implementation with a
GSList of ID's.

	Heap without client connection using DB cache: 4,759,552 bytes

5. Implement "stringletons" and fix errors in 4) above.

	Heap without client connection using DB cache: 2,854,912 bytes
	Heap without client connection: 3,518,464 bytes
	Heap after client connection and song list: 6,864,896 bytes
	Max heap usages (massif): 8,945,760 bytes

6. Modify libdmapsharing so that it responds to "/1/items" requests in
chunks, avoiding the need to build the entire response in memory.

	The effectiveness of this can be measured by observing the
	maximum heap usage after a client has asked for a full media
	list. Previous to this change, dmapd would use 7,811,072 bytes
	of heap to describe a 2,232-song library. After this change,
	dmapd's usage was 3,665,920 bytes, a very small increase over
	the memory used by dmapd immediately after starting.

7. Write a disk-based database for dmapd.

	In order to minimize the memory used by a newly started dmapd,
	I wrote a disk-based DMAPDb backend. This backend stores all
	database data in files and avoids storing the entire database
	in memory. To measure the effectiveness, I measured the memory
	usage of dmapd after loading a 3,276-photograph library (note
	that thumbnails are especially problematic). The GHashTable-based
	database used 35,885,056 bytes of heap space. The disk-based
	database used 4,763,648 bytes.

8. Decompressing multiple scan JPEG's.

	VIPS must decompress multiple scan JPEG's fully in memory due
	to the way libjpeg works. Either 1) use embedded EXIF thumbnail
	or 2) skip.