1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
|
# Nydus Tarfs Mode
`Nydus Tarfs Mode` or `Tarfs` is a working mode for Nydus Image, which uses tar files as Nydus data blobs instead of generating native Nydus data blobs.
### Enable Tarfs
`Nydus Tarfs Mode` is still an experiment feature, please edit the snapshotter configuration file to enable the feature:
```
[experimental.tarfs]
enable_tarfs = true
```
### Generate Raw Disk Image for Each Layer of a Container Image
`Tarfs` supports generating a raw disk image for each layer of a container image, which can be directly mounted as EROFS filesystem through loopdev. Please edit the snapshotter configuration file to enable this submode:
```
[experimental.tarfs]
enable_tarfs = true
export_mode = "layer_block"
```
This is an example to generate and verify raw disk image for each layer of a container image:
```
$ containerd-nydus-grpc --config /etc/nydus/config.toml &
$ nerdctl run --snapshotter nydus --rm nginx
# Show mounted rootfs a container
$ mount
/dev/loop17 on /var/lib/containerd-nydus/snapshots/7/mnt type erofs (ro,relatime,user_xattr,acl,cache_strategy=readaround)
# Show loop devices used to mount layers and bootstrap for a container image
$ losetup
NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC
/dev/loop11 0 0 0 0 /var/lib/containerd-nydus/cache/fd9f026c631046113bd492f69761c3ba6042c791c35a60e7c7f3b8f254592daa 0 512
/dev/loop12 0 0 0 0 /var/lib/containerd-nydus/cache/055fa98b43638b67d10c58d41094d99c8696cc34b7a960c7a0cc5d9d152d12b3 0 512
/dev/loop13 0 0 0 0 /var/lib/containerd-nydus/cache/96576293dd2954ff84251aa0455687c8643358ba1b190ea1818f56b41884bdbd 0 512
/dev/loop14 0 0 0 0 /var/lib/containerd-nydus/cache/a7c4092be9044bd4eef78f27c95785ef3a9f345d01fd4512bc94ddaaefc359f4 0 512
/dev/loop15 0 0 0 0 /var/lib/containerd-nydus/cache/e3b6889c89547ec9ba653ab44ed32a99370940d51df956968c0d578dd61ab665 0 512
/dev/loop16 0 0 0 0 /var/lib/containerd-nydus/cache/da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75 0 512
/dev/loop17 0 0 0 0 /var/lib/containerd-nydus/snapshots/7/fs/image/image.boot 0 512
# Files without suffix are tar files, files with suffix `layer.disk` are raw disk image for container image layers
$ ls -l /var/lib/containerd-nydus/cache/
total 376800
-rw-r--r-- 1 root root 3584 Aug 30 23:18 055fa98b43638b67d10c58d41094d99c8696cc34b7a960c7a0cc5d9d152d12b3
-rw-r--r-- 1 root root 527872 Aug 30 23:18 055fa98b43638b67d10c58d41094d99c8696cc34b7a960c7a0cc5d9d152d12b3.layer.disk
-rw-r--r-- 1 root root 77814784 Aug 30 23:18 52d2b7f179e32b4cbd579ee3c4958027988f9a8274850ab0c7c24661e3adaac5
-rw-r--r-- 1 root root 78863360 Aug 30 23:18 52d2b7f179e32b4cbd579ee3c4958027988f9a8274850ab0c7c24661e3adaac5.layer.disk
-rw-r--r-- 1 root root 4608 Aug 30 23:18 96576293dd2954ff84251aa0455687c8643358ba1b190ea1818f56b41884bdbd
-rw-r--r-- 1 root root 528896 Aug 30 23:18 96576293dd2954ff84251aa0455687c8643358ba1b190ea1818f56b41884bdbd.layer.disk
-rw-r--r-- 1 root root 2560 Aug 30 23:18 a7c4092be9044bd4eef78f27c95785ef3a9f345d01fd4512bc94ddaaefc359f4
-rw-r--r-- 1 root root 526848 Aug 30 23:18 a7c4092be9044bd4eef78f27c95785ef3a9f345d01fd4512bc94ddaaefc359f4.layer.disk
-rw-r--r-- 1 root root 7168 Aug 30 23:18 da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75
-rw-r--r-- 1 root root 531456 Aug 30 23:18 da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75.layer.disk
-rw-r--r-- 1 root root 5120 Aug 30 23:18 e3b6889c89547ec9ba653ab44ed32a99370940d51df956968c0d578dd61ab665
-rw-r--r-- 1 root root 529408 Aug 30 23:18 e3b6889c89547ec9ba653ab44ed32a99370940d51df956968c0d578dd61ab665.layer.disk
-rw-r--r-- 1 root root 112968704 Aug 30 23:18 fd9f026c631046113bd492f69761c3ba6042c791c35a60e7c7f3b8f254592daa
-rw-r--r-- 1 root root 113492992 Aug 30 23:18 fd9f026c631046113bd492f69761c3ba6042c791c35a60e7c7f3b8f254592daa.layer.disk
$ file /var/lib/containerd-nydus/cache/055fa98b43638b67d10c58d41094d99c8696cc34b7a960c7a0cc5d9d152d12b3
/var/lib/containerd-nydus/cache/055fa98b43638b67d10c58d41094d99c8696cc34b7a960c7a0cc5d9d152d12b3: POSIX tar archive
# Mount the raw disk image for a container image layer
$ losetup /dev/loop100 /var/lib/containerd-nydus/cache/055fa98b43638b67d10c58d41094d99c8696cc34b7a960c7a0cc5d9d152d12b3.layer.disk
$ mount -t erofs /dev/loop100 ./mnt/
$ mount
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=1544836k,nr_inodes=386209,mode=700,inode64)
/dev/loop17 on /var/lib/containerd-nydus/snapshots/7/mnt type erofs (ro,relatime,user_xattr,acl,cache_strategy=readaround)
/dev/loop100 on /root/ws/nydus-snapshotter.git/mnt type erofs (ro,relatime,user_xattr,acl,cache_strategy=readaround)
```
### Generate Raw Disk Image for a Container Image
`Tarfs` supports generating a raw disk image a container image, which can be directly mounted as EROFS filesystem through loopdev. Please edit the snapshotter configuration file to enable this submode:
```
[experimental.tarfs]
enable_tarfs = true
export_mode = "image_block"
```
This is an example to generate and verify raw disk image for a container image:
```
$ containerd-nydus-grpc --config /etc/nydus/config.toml &
$ nerdctl run --snapshotter nydus --rm nginx
# Files without suffix are tar files, files with suffix `image.disk` are raw disk image for a container image
$ ls -l /var/lib/containerd-nydus/cache/
total 376320
-rw-r--r-- 1 root root 3584 Aug 30 23:35 055fa98b43638b67d10c58d41094d99c8696cc34b7a960c7a0cc5d9d152d12b3
-rw-r--r-- 1 root root 77814784 Aug 30 23:35 52d2b7f179e32b4cbd579ee3c4958027988f9a8274850ab0c7c24661e3adaac5
-rw-r--r-- 1 root root 4608 Aug 30 23:35 96576293dd2954ff84251aa0455687c8643358ba1b190ea1818f56b41884bdbd
-rw-r--r-- 1 root root 2560 Aug 30 23:35 a7c4092be9044bd4eef78f27c95785ef3a9f345d01fd4512bc94ddaaefc359f4
-rw-r--r-- 1 root root 7168 Aug 30 23:35 da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75
-rw-r--r-- 1 root root 194518016 Aug 30 23:36 da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75.image.disk
-rw-r--r-- 1 root root 5120 Aug 30 23:35 e3b6889c89547ec9ba653ab44ed32a99370940d51df956968c0d578dd61ab665
-rw-r--r-- 1 root root 112968704 Aug 30 23:36 fd9f026c631046113bd492f69761c3ba6042c791c35a60e7c7f3b8f254592daa
```
### Generate Raw Disk Image with dm-verity Information
`Tarfs` supports generating raw disk images with dm-verity information, to enable runtime data integrity validation. Please change `export_mode` in snapshotter configuration file to `layer_block_with_verity` or `image_block_with_verity`.
```
[experimental.tarfs]
enable_tarfs = true
export_mode = "image_block_with_verity"
```
This is an example to generate and verify raw disk image for a container image with dm-verity information:
```
$ containerd-nydus-grpc --config /etc/nydus/config.toml &
$ nerdctl run --snapshotter nydus --rm nginx
# Files without suffix are tar files, files with suffix `image.disk` are raw disk image for a container image
$ ls -l /var/lib/containerd-nydus/cache/
total 388296
-rw-r--r-- 1 root root 3584 Aug 30 23:45 055fa98b43638b67d10c58d41094d99c8696cc34b7a960c7a0cc5d9d152d12b3
-rw-r--r-- 1 root root 77814784 Aug 30 23:46 52d2b7f179e32b4cbd579ee3c4958027988f9a8274850ab0c7c24661e3adaac5
-rw-r--r-- 1 root root 4608 Aug 30 23:45 96576293dd2954ff84251aa0455687c8643358ba1b190ea1818f56b41884bdbd
-rw-r--r-- 1 root root 2560 Aug 30 23:45 a7c4092be9044bd4eef78f27c95785ef3a9f345d01fd4512bc94ddaaefc359f4
-rw-r--r-- 1 root root 7168 Aug 30 23:45 da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75
-rw-r--r-- 1 root root 206782464 Aug 30 23:46 da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75.image.disk
-rw-r--r-- 1 root root 5120 Aug 30 23:45 e3b6889c89547ec9ba653ab44ed32a99370940d51df956968c0d578dd61ab665
-rw-r--r-- 1 root root 112968704 Aug 30 23:46 fd9f026c631046113bd492f69761c3ba6042c791c35a60e7c7f3b8f254592daa
$ losetup /dev/loop100 /var/lib/containerd-nydus/cache/da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75.image.disk
$ veritysetup open --no-superblock --format=1 -s "" --hash=sha256 --data-block-size=512 --hash-block-size=4096 --data-blocks 379918 --hash-offset 194519040 /dev/loop100 image1 /dev/loop100 8113799aaf9a5d14feca1eadc3b7e6ea98bdaf61e3a2e4a8ef8c24e26a551efd
$ lsblk
loop100 7:100 0 197.2M 0 loop
└─dm-0 252:0 0 185.5M 1 crypt
$ veritysetup status dm-0
/dev/mapper/dm-0 is active and is in use.
type: VERITY
status: verified
hash type: 1
data block: 512
hash block: 4096
hash name: sha256
salt: -
data device: /dev/loop100
data loop: /var/lib/containerd-nydus/cache/da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75.image.disk
size: 379918 sectors
mode: readonly
hash device: /dev/loop100
hash loop: /var/lib/containerd-nydus/cache/da761d9a302b21dc50767b67d46f737f5072fb4490c525b4a7ae6f18e1dbbf75.image.disk
hash offset: 379920 sectors
root hash: 8113799aaf9a5d14feca1eadc3b7e6ea98bdaf61e3a2e4a8ef8c24e26a551efd
$ mount -t erofs /dev/dm-0 ./mnt/
mount: /root/ws/nydus-snapshotter.git/mnt: WARNING: source write-protected, mounted read-only.
$ ls -l mnt/
total 14
lrwxrwxrwx 1 root root 7 Aug 14 08:00 bin -> usr/bin
drwxr-xr-x 2 root root 27 Jul 15 00:00 boot
drwxr-xr-x 2 root root 27 Aug 14 08:00 dev
drwxr-xr-x 2 root root 184 Aug 16 17:50 docker-entrypoint.d
-rwxrwxr-x 1 root root 1620 Aug 16 17:50 docker-entrypoint.sh
drwxr-xr-x 34 root root 1524 Aug 16 17:50 etc
drwxr-xr-x 2 root root 27 Jul 15 00:00 home
lrwxrwxrwx 1 root root 7 Aug 14 08:00 lib -> usr/lib
lrwxrwxrwx 1 root root 9 Aug 14 08:00 lib32 -> usr/lib32
lrwxrwxrwx 1 root root 9 Aug 14 08:00 lib64 -> usr/lib64
lrwxrwxrwx 1 root root 10 Aug 14 08:00 libx32 -> usr/libx32
drwxr-xr-x 2 root root 27 Aug 14 08:00 media
drwxr-xr-x 2 root root 27 Aug 14 08:00 mnt
drwxr-xr-x 2 root root 27 Aug 14 08:00 opt
drwxr-xr-x 2 root root 27 Jul 15 00:00 proc
drwx------ 2 root root 66 Aug 14 08:00 root
drwxr-xr-x 3 root root 43 Aug 14 08:00 run
lrwxrwxrwx 1 root root 8 Aug 14 08:00 sbin -> usr/sbin
drwxr-xr-x 2 root root 27 Aug 14 08:00 srv
drwxr-xr-x 2 root root 27 Jul 15 00:00 sys
drwxrwxrwt 2 root root 27 Aug 16 17:50 tmp
drwxr-xr-x 14 root root 229 Aug 14 08:00 usr
drwxr-xr-x 11 root root 204 Aug 14 08:00 var
```
|
