Multi-Threaded Download Manager

The Multi-Threaded Download Manager (MTDM) is a linux console application similar to wget that can download remote files by specifying the remote file's URL. With MTDM you can:

• download a remote file by opening multiple connections to the remote server; using this MTDM feature the remote file will be downloaded faster;
• download a web page (html) and later recursively download all hyperlinks from within the already downloaded file; with this feature MTDM can bu used as a web robot, that downloads all the pages referred among each other for further processing.

MTDM is implemented in C++ programming language, it uses POSIX threads (pthread library) and its implementation strictly abides to the Object Oriented software design techniques. In socket programming, MTDM would serve as a good example about:

• the simple socket operations - open socket, close socket, send and receive data;
• using pthread primitives in Object Oriented C++ applications, by wrapping them in C++ classes;
• using the client side HTTP protocol to inspect the header of the remote file; and
• how to download specific parts of a file from HTTP server.

User Manual

This section describes how to download, compile, run and test the program.

Download

You can download the most current MTDM release by clicking here.

Install

To install MTDM you don't need to do anything. You can download the binary and use it.

Compile

After obtaining the source codes of MTDM change the current directory to be the one that contains Makefile file. Then invoke the make utility by typing make in the terminal. As an output two binary executable files will be created - downloadmanager.exe and comparefiles.exe. downloadmanager.exe is the download program and comparefiles.exe compares two files.

Run

Name of the executable file that downloads remote files is downloadmanager.exe. Its usage is:

$./downloadmanager.exe -d | -n [positive number] -u <URL>

Parameters that the program takes are:

• -d - downloads all hyperlinks in depth 1 found in <URL>, using this parameter MTDM acts as a web robot.
• -n - specifies the number of threads to use during the download process. By default n is equal to 5. The use of this parameter together with -d is not supported. For now you can use it with -u only.
• -u - specifies the target URL file that has to be downloaded.
• Parameters -d and -u are mutually exclusive, meaning that you can't use them at the same time.

Example Usage -u

$./downloadmanager.exe -n 12 -u http://localhost/myfile.zip

The above command will download file named myfile.zip from host localhost using 12 threads binding to the default port 80.

$./downloadmanager.exe -u http://www.sc.deu.edu.tr:8080/test.exe

The above command will download file named test.exe from host named www.cs.deu.edu.tr using threads 5 binding to port number 8080.

Example Usage -d

$./downloadmanager.exe -d http://www.linux.org:1010/index.html

The above command downloads all links found in file index.html hosted at server named www.linux.org binding to port 1010.

Test

You can test the download manager using the provided executable comparefiles.exe. Its usage is:

$./comparefiles.exe <filename1> <filename2>

• filename1 could be the original file hosted in the server.
• filename2 could be the file downloaded from the server.

comparefiles.exe performs binary comparison on both files and prints a report - equal or not equal.

Resources