lookiconsultants.blogg.se - Directions to copy fast

DIRECTIONS TO COPY FAST FULL

Readonly PRIMARY=/export/home/david/dist/primary I need to copy around 400 files in machineA from machineB and machineC and each file size is 1.5 GB.Ĭurrently I have my below shell script which works fine as I am using scp but somehow it takes ~ 2 hours to copy the 400 files in machineA which is too long for me I guess.

DIRECTIONS TO COPY FAST FULL

So suppose if this is the latest date folder 20140317 inside /data/pe_t1_snapshot then this will be the full path for me - /data/pe_t1_snapshot/20140317įrom where I need to start copying the files in machineB and machineC. So whatever date is the latest date in this format YYYYMMDD inside the above folder - I will pick that folder as the full path from where I need to start copying the files. In machineB and machineC there will be a folder like this YYYYMMDD inside this folder - /data/pe_t1_snapshot So I will try to copy from machineB first, if it is not there in machineB then I will go to machineC to copy the same files. If the file is not there in machineB, then it should be there in machineC for sure. Last, if you have the fast mawk installed, make use of it via the -mawk option.I am running my shell script on machineA which copies the files from machineB and machineC to machineA. Your use case presumably does not require generation of restore scripts: use the -noRestore option. If you want it to just generate the cp script (but not execute it, which would require extensive display and interaction), use the -noExec option. It is used as follows: $ Zaloha.sh -sourceDir="test_source" -backupDir="test_backup" It keeps its internal data in files, not in memory. It runs find on both directories and prepares scripts with cp commands. If you are interested in another synchronizer, you might have a look at Fitus/Zaloha.sh. But rsync keeps its internal data in memory, which may cause problems with huge directory trees. Rsync is strong in copying over network (delta transfer of big files). It is not necessary to use a synchronizer if the target directory is empty, but it brings benefits like restartability, possibility to exclude certain files etc. If both storages are local, cp should transfer data near maximum possible speed.

-inplace to avoid file copy (but only if nothing is reading the destination until the whole transfer completes).

Reading half of a partially transferred file is often much quicker than writing it again.

-no-whole-file so that anything that needs to be resent uses delta transfer.

Note: files won't have a temporary name, so ensure that nothing else is expecting to use the destination until the whole copy has completed.

-partial or -P which is -partial -progress: save any partially transferred files for future resuming.

-S/ -sparse: turn sequences of nulls into sparse blocks.

Also, it checksums the whole file at the end, meaning no significant speed up over -no-whole-file while adding a dangerous failure case. This sounds like a good idea, but it has the dangerous failure case: any destination file the same size (or greater) than the source will be IGNORED.

-append-verify: resume an interrupted transfer.

-z/ -compress: compression will only load up the CPU as the transfer isn't over a network but over RAM.

There are some speed-ups which can be applied to rsync: Avoid And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". The simplest way to preserve most things is to use the -a flag – ‘archive.’ So: rsync -a source destĪlthough UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags: rsync -aHAX source dest # Linux As others mention, it can exclude files easily. And being rsync, it can even restart part way through a large file.

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost.