Recently I had the problem of transferring several files from several locations. I wanted to it in parallel to speed up the process. I had all URL‘s saved in a file, one URL per line. The content of the file looks like this
http://server1/file1 http://server2/file2
I found that wget is capable of reading the file but does not support parallel execution of the downloads. My solution was to use lftp with the following command
$ cat inputFile | while read line; do echo "${line}"; done | xargs -i -t lftp -c 'pget -n 5 {} &'
It transfers every file in parallel and opens 5 connections to the server per file. There is a dedicated process for every lftp command (2 processes in this example) running in the background. It outputs the command befor executing it (-t option)
You could also use the following command, which leaves you with only one process.
$ cat inputFile | while read line; do echo "pget -n 5 ${line} &"; done | exec lftp
This is just a variant of the above, using exec instead of xargs. I think its a rather odd command to solve my problem, but at least it works.
Of course, I could write a script which somehow constructs me the appropriate strings to execute. But again, this would just be another approach (and not much of a difference).
I’m almost sure there must be an easier solution for that, but I just haven’t found it.