1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
|
###############################################################################
###############################################################################
###############################################################################
######### #########
######### #########
######### THIS DOCUMENT IS SUPERSEDED BY README.fstransform #########
######### #########
######### #########
######### PLEASE SEE README.fstransform INSTEAD OF THIS FILE #########
######### #########
######### #########
###############################################################################
###############################################################################
###############################################################################
###############################################################################
###############################################################################
###############################################################################
######### #########
######### THIS DOCUMENT EXPLAINS HOW TO USE #########
######### RISKY PROGRAMS AND PROCEDURES #########
######### THAT MAY COMPLETELY AND IRREVERSIBLY RUIN #########
######### THE DATA ON YOUR DISKS #########
######### #########
######### #########
######### THE AUTHOR DECLINES ALL RESPONSIBILITIES #########
######### FOR ANY DAMAGE THAT MAY DERIVE #########
######### FROM USING THE PROGRAMS AND PROCEDURES #########
######### DESCRIBED IN THIS DOCUMENT #########
######### #########
######### #########
######### IF YOU CANNOT COMPLY #########
######### WITH THE TERMS AND DISCLAIMERS STATED ABOVE, #########
######### PLEASE STOP READING NOW #########
######### AND DO NOT TRY TO USE THE PROGRAMS AND PROCEDURES #########
######### EXPLAINED IN THIS DOCUMENT #########
######### #########
######### #########
######### IF YOU KEEP READING, #########
######### YOU MUST UNDERSTAND AND ACCEPT ALL THE RISKS #########
######### BEFORE PROCEEDING #########
######### #########
###############################################################################
###############################################################################
###############################################################################
This document explains how to manually use 'fsremap' and 'fsmove',
bypassing the MUCH simpler and more user-friendly script 'fstransform.sh'
It is useful only if a user does not want or cannot use the script 'fstransform.sh'
for some reason, or for those folks that want to know the internal procedure
used by 'fstransform.sh'.
It explains which steps must be manually performed _as_root_ by a user,
invoking existing tools (mount, umount, dd, losetup, mkfs.*, fsck.*)
as well as 'fsremap' and 'fsmove', to convert a device from a filesystem to another,
WITHOUT using 'fstransform.sh'.
These steps are automated by 'fstransform.sh', but they are also kept here
1) for historical reasons
2) to document the internal procedure used by 'fstransform.sh'
3) in case 'fstransform.sh' cannot be used
So, let's get started.
FOREWORD:
If the original device is almost full, the algorithm used by fsremap
will create a backup file ("secondary storage", it will be up to
half your free RAM) in your $HOME directory.
You can use the option '-s <size>[k|M|G|T|P|E|Y|Z]' to manually set the secondary storage size,
but please understand that using a too small secondary storage can slow down fsremap.
PROCEDURE:
If you are still reading, let's see to how to actually use fsremap and fsmove:
0) compile them. Running "make" should suffice on any recent Linux
machine, as long as g++ is installed.
You will get two executables, fsmove and fsremap.
They will be located at
fstransform-{version}-src/fsmove/Release/fsmove
and
fstransform-{version}-src/fsremap/Release/fsmap
You are suggested to either run "make install" or to copy them to a simpler path.
Below, they will be referred as {fsmove} and {fsremap}
1) mount read-write the device you want to remap to a different file-system
mount {device} {device-mount-point} [your-options]
if the device is already mounted, check that it is mounted read-write
and that no process is using it.
2) create a sparse file inside the device to be used as loop-file,
with the same length as the device itself
dd if=/dev/zero of={device-mount-point}/{loop-file} bs=1 count=1 seek=$(( {device-size-in-bytes} - 1 ))
or, if you have truncate(1)
truncate -s {device-size-in-bytes} {device-mount-point}/{loop-file}
3) format the loop-file with the new file system you want to use
mkfs.[ext2,ext3,ext4,reiserfs,xfs,jfs...] {device-mount-point}/{loop-file}
if you create an ext2/ext3/ext4 file-system, the option "-m 0" will help you
to detect more precisely if/when the file-system inside the loop-file becomes full
4) mount the loop-file read-write
mount {device-mount-point}/{loop-file} {loop-file-mount-point} -o loop {your-options}
this will find a free /dev/loop{n}, setup it and mount it.
if you have an old mount which does not understand "-o loop", you will need
to manually find and setup a free loop-device:
losetup /dev/loop{n} {device-mount-point}/{loop-file} [your-options]
mount /dev/loop{n} {loop-file-mount-point} [your-options]
5) manually and recursively move all files, directories, links and so on
from {device-mount-point} to {loop-file-mount-point},
reproducing inside {loop-file-mount-point} the original tree that was in {device-mount-point}.
Quite obviously, you should not move {loop-file} itself.
This part is the most delicate, because {loop-file} disk usage will grow while
you copy files inside it, and {device} or /dev/loop{n} - or both -
can become full: if it happens, you will be in BIG troubles.
Let's see how to perform this delicate recursive move.
You should have the fsmove program distributed with fstransform.
If you have it in {fsmove}, run:
{fsmove} {device-mount-point} {loop-file-mount-point} --exclude {device-mount-point}/{loop-file}
################## WARNING ####### WARNING ####### WARNING ################
The last command above can take a LONG time to complete, and it CAN fill either filesystem
{device-mount-point} and/or {loop-file-mount-point}.
Even if {fsmove} periodically checks for free space, you still MUST monitor its progress with something like
watch df {device-mount-point} {loop-file-mount-point}
because, if {device-mount-point} becomes full and the copy receives an 'I/O error', you are in trouble:
the filesystem inside {loop-file-mount-point} will become CORRUPTED!
So if you realize {device-mount-point} is going to become full, you MUST interrupt the copy BEFORE it happens
(and give up, sorry).
######################## ALTERNATIVE SOLUTION #############################
If you don't have fsmove, things get a bit more ugly - and slow - and you need find(1), mv(1) and cpio(1)
This is how: first recreate the directory tree inside loop-file with:
( cd {device-mount-point} && find -xdev -type d ) | cpio -o | ( cd {loop-file-mount-point} && cpio -i )
then move all the files one by one, except {loop-file} (note: {loop-file-mount-point} must be an absolute path)
cd {device-mount-point} && find . -xdev \! -type d \! -path ./{loop-file} | while read i; do mv "$i" {loop-file-mount-point}/"$i" || break; done
The same WARNING above still applies: you MUST monitor the free space
on {device-mount-point} and {loop-file-mount-point} and interrupt
the operation if you realize {device-mount-point} is going to become full.
Actually, since these alternative commands do a simple file-by-file
copy and do not check for free space, in this case it is even more
critical to monitor the free space.
6) once all your files are inside the file-system in {loop-file}, things get less delicate.
now you need to umount {loop-file}, 'sync' and check that {loop-file} is not corrupted
umount {loop-file-mount-point}
# if you used losetup before, you will need also:
losetup -d /dev/loop{n}
# then in any case continue with:
sync
fsck -f {device-mount-point}/{loop-file}
you need to be ABSOLUTELY sure that there are no errors in the
filesystem inside {device-mount-point}/{loop-file} before proceeding!
Additionally, it is now the time to check that the kernel did not
report any error while writing to {loop-file-mount-point}.
Please run:
dmesg
and check that there are no recent I/O errors in the produced
messages.
7) the initial setup is almost complete.
now create a file full with zeros (not sparse) inside {device-mount-point}
to help fsremap to locate any free disk space inside {device}:
dd if=/dev/zero of={device-mount-point}/{zero-file} bs=512
the second command will exit with the error "no space left on device",
but that's expected and not a problem.
Note: if you have fallocate(1) and the file-system on {device} supports it,
you can use fallocate instead of dd as it's much faster.
8) remount {device} read-only to be sure no process
will write into it:
mount {device} -o remount,ro
9) choose a folder with some free space (NOT inside {device}) where fsremap
can write its (hopefully small) backup data and log files, and name it {storage-dir}
If you skip this step, fsremap will use your home directory as
default {storage-dir}.
10) start fsremap. If you have the executable in {fsremap}, run:
{fsremap} -t {storage-dir} {device} {device-mount-point}/{loop-file} {device-mount-point}/{zero-file}
if you did not choose a {storage-dir} folder, omit the arguments -t {storage-dir}
and fsremap will use your home directory as default {storage-dir}
if {device} has very little free space, fsremap will create a file
{storage-dir}/.fsremap/job.{x}/storage.bin and use it to store blocks
while moving them around.
11) after an analysis phase, fsremap will ask you to umount {device}. just do as asked:
umount {device}
12) fsremap will now relocate blocks from {loop-file} to {device}
and it will report its progress.
once finished, check that the remapping actually worked:
fsck -f {device}
mount {device} {device-mount-point} [your-options]
if everything worked, you can delete the file {storage-dir}/.fsremap/job.{x}/storage.bin
in case of problems, you can post your problem to the mailing list,
attaching the file {storage-dir}/.fsremap/job.{x}/fsremap.log
and hopefully somebody will help diagnosing the problem...
Good luck!
|
