Aria2c download

In today's digital age, downloading files is a routine task for many of us. Whether it's downloading a large file, such as a video or a software package, or multiple files from different websites, we all have to deal with downloads. Fortunately, there are tools available that can help make task of downloading files easier and more efficient. One such tool is Aria2, a multi-protocol command-line download tool for Linux.What is Aria2?Aria2 is a lightweight and versatile download utility that allows users to download files from various sources using multiple protocols. It supports HTTP, HTTPS, FTP, BitTorrent, and Metalink protocols, making it a powerful tool for downloading files from internet. Aria2 is a command-line tool, which means that it is controlled using text commands in a terminal window. Although this may seem daunting to some users, it is actually very easy to use and can be more efficient than using a graphical interface.Installing Aria2Before you can start using Aria2, you need to install it on your Linux system. Fortunately, Aria2 is available in most Linux distributions' software repositories, so it can be installed using distribution's package manager. For example, to install Aria2 on Ubuntu, you can open a terminal window and type following command −sudo apt-get install aria2This will download and install Aria2 and any necessary dependencies. Once installation is complete, you can start using Aria2 to download files.Downloading Files with Aria2Downloading a file with Aria2 is a simple process. You need to open a terminal window and type aria2c command followed by URL of file you want to download. For example, to download a Linux ISO file, you can type following command −aria2c will start download process, and Aria2 will display information about download, such as download speed, remaining time, and percentage of completion. By default, Aria2 will save downloaded file to current directory.Downloading Multiple Files with Aria2Aria2 also allows you to download multiple files at once. To download multiple files, you need to create a text file containing URLs of files you want to download. Each URL should be on a separate line. You can then pass name of text file to aria2c command using -i option. For example, if you have a text file called files.txt containing URLs of three files, you can download all three files using following command −aria2c -i files.txtAria2 will start downloading all three files concurrently, which can save time compared to downloading each file individually.Using Aria2 with BitTorrentAria2's support for BitTorrent protocol makes it an excellent tool for downloading large files, such as Linux distributions or software packages. To download a file using BitTorrent, you need to pass .torrent file to aria2c command using -d option. For example, to download a Linux

The video while downloading, and reducing the chance of file corruption if download is interrupted. This is enabled by default for live streams --no-hls-use-mpegts Do not use the mpegts container for HLS videos. This is default when not downloading live streams --downloader [PROTO:]NAME Name or path of the external downloader to use (optionally) prefixed by the protocols (http, ftp, m3u8, dash, rstp, rtmp, mms) to use it for. Currently supports native, aria2c, avconv, axel, curl, ffmpeg, httpie, wget (Recommended: aria2c). You can use this option multiple times to set different downloaders for different protocols. For example, --downloader aria2c --downloader "dash,m3u8:native" will use aria2c for http/ftp downloads, and the native downloader for dash/m3u8 downloads (Alias: --external-downloader) --downloader-args NAME:ARGS Give these arguments to the external downloader. Specify the downloader name and the arguments separated by a colon ":". For ffmpeg, arguments can be passed to different positions using the same syntax as --postprocessor-args. You can use this option multiple times to give different arguments to different downloaders (Alias: --external-downloader-args) Filesystem Options: -a, --batch-file FILE File containing URLs to download ("-" for stdin), one URL per line. Lines starting with "#", ";" or "]" are considered as comments and ignored --no-batch-file Do not read URLs from batch file (default) -P, --paths [TYPES:]PATH The paths where the files should be downloaded. Specify the type of file and the path separated by a colon ":". All the same TYPES as --output are supported. Additionally, you can also provide "home" (default) and "temp" paths. All intermediary

🤗Huggingface Model DownloaderNote(2025-01-08) Add feature for 🏷️Tag(Revision) Selection, contributed by @Bamboo-D.(2024-12-17) Add feature for ⚡Quick Startup and ⏭️Fast Resume, enabling skipping of downloaded files, while removing the git clone dependency to accelerate file list retrieval.Considering the lack of multi-threaded download support in the official huggingface-cli, and the inadequate error handling in hf_transfer, This command-line tool leverages curl and aria2c for fast and robust downloading of models and datasets.Features⏯️ Resume from breakpoint: You can re-run it or Ctrl+C anytime.🚀 Multi-threaded Download: Utilize multiple threads to speed up the download process.🚫 File Exclusion: Use --exclude or --include to skip or specify files, save time for models with duplicate formats (e.g., *.bin or *.safetensors).🔐 Auth Support: For gated models that require Huggingface login, use --hf_username and --hf_token to authenticate.🪞 Mirror Site Support: Set up with HF_ENDPOINT environment variable.🌍 Proxy Support: Set up with https_proxy environment variable.📦 Simple: Minimal dependencies, requires only curl and wget, while aria2 and jq are optional for better performance.🏷️ Tag Selection: Support downloading specific model/dataset revision using --revision.UsageFirst, Download hfd.sh or clone this repo, and then grant execution permission to the script.you can create an alias for convenienceUsage Instructions$ ./hfd.sh --helpUsage: hfd [--include include_pattern1 include_pattern2 ...] [--exclude exclude_pattern1 exclude_pattern2 ...] [--hf_username username] [--hf_token token] [--tool aria2c|wget] [-x threads] [-j jobs] [--dataset] [--local-dir path] [--revision rev]Description: Downloads a model or dataset from Hugging Face using the provided repo ID.Arguments: REPO_ID The Hugging Face repo ID (Required) Format: 'org_name/repo_name' or legacy format (e.g., gpt2)Options: include/exclude_pattern The patterns to match against file path, supports wildcard characters. e.g., '--exclude *.safetensor *.md', '--include vae/*'. --include (Optional) Patterns to include files for downloading (supports multiple patterns). --exclude (Optional) Patterns to exclude files from downloading (supports multiple patterns). --hf_username (Optional) Hugging Face username for authentication (not email). --hf_token (Optional) Hugging Face token for authentication. --tool (Optional) Download tool to use: aria2c (default) or wget. -x (Optional) Number of download threads for aria2c (default: 4). -j (Optional) Number of concurrent downloads for aria2c (default: 5). --dataset (Optional) Flag to indicate downloading a dataset. --local-dir (Optional) Directory path to store the downloaded data. Defaults to the current directory with a subdirectory named 'repo_name' if REPO_ID is is composed of 'org_name/repo_name'. --revision (Optional) Model/Dataset revision to download (default: main).Example: hfd gpt2 hfd bigscience/bloom-560m --exclude *.bin *.msgpack onnx/* hfd meta-llama/Llama-2-7b --hf_username myuser --hf_token mytoken -x 4 hfd lavita/medical-qa-shared-task-v1-toy --dataset hfd bartowski/Phi-3.5-mini-instruct-exl2 --revision 5_0Download a modelhfd bigscience/bloom-560mDownload a model need loginGet huggingface token from thenhfd meta-llama/Llama-2-7b --hf_username YOUR_HF_USERNAME_NOT_EMAIL --hf_token YOUR_HF_TOKENDownload a model and exclude certain files (e.g., .safetensors)hfd bigscience/bloom-560m --exclude *.bin *.msgpack onnx/*You can also exclude multiple pattern like thathfd bigscience/bloom-560m --exclude *.bin --exclude *.msgpack --exclude onnx/*Download specific files using include patternshfd Qwen/Qwen2.5-Coder-32B-Instruct-GGUF --include *q2_k*.ggufDownload a datasethfd lavita/medical-qa-shared-task-v1-toy --datasetDownload a specific

Distribution using BitTorrent, you can download .torrent file and then type following command −aria2c -d ~/Downloads ubuntu-20.04.3-desktop-amd64.iso.torrentThis will start download process, and Aria2 will display information about download, such as number of seeds and peers and download speed.Using Aria2 with MetalinkAria2 also supports Metalink protocol, which allows you to download files from multiple sources simultaneously. Metalink files contain information about different sources of a file, such as HTTP and FTP servers.To download a file using Metalink, you need to pass URL of Metalink file to aria2c command. For example, to download Ubuntu 20.04.3 desktop ISO using Metalink, you can type following command −aria2c will start download process, and Aria2 will download file from different sources listed in Metalink file, which can increase download speed.Controlling Aria2 DownloadsAria2 allows you to control download process in various ways using command-line options. For example, you can limit download speed using --max-download-limit option, which can be useful if you want to limit impact of downloads on your network. You can also set maximum number of connections using --max-connection-per-server option, which can help you optimize download speed based on your network capacity.You can also pause and resume downloads using Aria2. To pause a download, you can press Ctrl + C keys in terminal window. This will pause download, and you can resume it later using same aria2c command. Aria2 also supports resuming interrupted downloads automatically, so you don't have to worry about losing progress if download is interrupted.In addition to its features, Aria2 is an open-source software with an active community that provides support and regular updates. This means that bugs are fixed quickly, and new features are added regularly. Furthermore, Aria2 is highly configurable, and users can customize it to their liking using command-line options or configuration files.Another useful feature of Aria2 is ability to download files in segments. This means that Aria2 can split a file into multiple parts and download them simultaneously. This can increase download speed significantly, especially when downloading large files from servers that limit download speed for each connection.Aria2 also has a built-in feature that allows users to verify integrity of downloaded files using checksums. This ensures that downloaded file is not corrupted, which can save time and prevent issues caused by corrupted files.One of most significant advantages of using Aria2 is its low resource usage. Aria2 is a lightweight tool that consumes minimal system resources, which means that it can run efficiently even on low-end hardware. This makes it an excellent tool for users who have limited system resources or want to download files without slowing down their system.Furthermore, Aria2 is a cross-platform tool that is available for Linux, Windows, and macOS, which means that it can be used

Aria2 static builds for GNU/Linux & Windows.all the patches are from: 1.37.0 builds (with OpenSSL):Download in Github ReleaseNote: The error with bcrypt.dll library on Windows XP has been fixed.Info: In this version, the Windows builds include manifest for Windows 10 (Contributed by eladkarako)How to install:GNU/Linux instructions:Open the terminal and type the following commands:$ tar jxvf aria2-1.37.0-linux-gnu-[arch]-build1.tar.bz2$ cd aria2-1.37.0-linux-gnu-[arch]-build1$ sudo make installWindows instructions:Unzip the package with 7-zip or Winrar.Copy the files to a folder. For example: C:\Program Files\aria2Add the folder to PATH variable.CA Certificates on Windows (HTTPS):To perform downloads using HTTPS, is necessary the ca-certificates. For default, Linux binaries read the certificates from /etc/ssl/certs/ca-certificates.crt, but with Windows binaries are necessary define the path of certificates.You can define the path of the certificates permanently using aria2.conf located on C:\Users\\.aria2\. If you use Windows XP, the file is located on C:\Documents and settings\\.aria2\. For example, you just have to add the following line:ca-certificate=C:\Program Files\aria2\certs\ca-certificates.crtIf you prefer load the certificates manually, you can use the following command: aria2c --ca-certificate= ">C:\PATH\> aria2c --ca-certificate=PATH-TO-CERTIFICATE> URL>Also, is possible disable the use of certificates with the following command: aria2c --check-certificate=false ">C:\PATH\> aria2c --check-certificate=false URL>Used external libraries: links:aria2 homepagearia2 documentationaria2 source code (Github)

Revision of a modelhfd bartowski/Phi-3.5-mini-instruct-exl2 --revision 5_0Multi-threading and Parallel DownloadsThe script supports two types of parallelism when using aria2c:Threads per File (-x): Controls connections per file, usage: hfd gpt2 -x 8, recommended: 4-8, default: 4 threads.Concurrent Files (-j): Controls simultaneous file downloads, usage: hfd gpt2 -j 3, recommended: 3-8, default: 5 files.Combined usage:hfd gpt2 -x 8 -j 3 # 8 threads per file, 3 files at once This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters #!/usr/bin/env bash # Color definitions RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m' # No Color trap 'printf "${YELLOW}\nDownload interrupted. You can resume by re-running the command.\n${NC}"; exit 1' INT display_help() { cat EOF Usage: hfd [--include include_pattern1 include_pattern2 ...] [--exclude exclude_pattern1 exclude_pattern2 ...] [--hf_username username] [--hf_token token] [--tool aria2c|wget] [-x threads] [-j jobs] [--dataset] [--local-dir path] [--revision rev] Description: Downloads a model or dataset from Hugging Face using the provided repo ID. Arguments: REPO_ID The Hugging Face repo ID (Required) Format: 'org_name/repo_name' or legacy format (e.g., gpt2) Options: include/exclude_pattern The patterns to match against file path, supports wildcard characters. e.g., '--exclude *.safetensor *.md', '--include vae/*'. --include (Optional) Patterns to include files for downloading (supports multiple patterns). --exclude (Optional) Patterns to exclude files from downloading (supports multiple patterns). --hf_username (Optional) Hugging Face username for authentication (not email). --hf_token (Optional) Hugging Face token for authentication. --tool (Optional) Download tool to use: aria2c (default) or wget. -x (Optional) Number of download threads for aria2c (default: 4). -j (Optional) Number of concurrent downloads for aria2c (default: 5). --dataset (Optional) Flag to indicate downloading a dataset. --local-dir (Optional) Directory path to store the downloaded data. Defaults to the current directory with a subdirectory named 'repo_name' if REPO_ID is is composed of 'org_name/repo_name'. --revision (Optional) Model/Dataset revision to download (default: main). Example: hfd gpt2 hfd bigscience/bloom-560m --exclude *.safetensors hfd meta-llama/Llama-2-7b --hf_username myuser --hf_token mytoken -x 4 hfd lavita/medical-qa-shared-task-v1-toy --dataset hfd bartowski/Phi-3.5-mini-instruct-exl2 --revision 5_0 EOF exit 1 } [[ -z "$1" || "$1" =~ ^-h || "$1" =~ ^--help ]] && display_help REPO_ID=$1 shift # Default values TOOL="aria2c" THREADS=4 CONCURRENT=5 HF_ENDPOINT=${HF_ENDPOINT:-" INCLUDE_PATTERNS=() EXCLUDE_PATTERNS=() REVISION="main" validate_number() { [[ "$2" =~ ^[1-9][0-9]*$ && "$2" -le "$3" ]] || { printf "${RED}[Error] $1 must be 1-$3${NC}\n"; exit 1; } } # Argument parsing while [[ $# -gt 0 ]]; do case $1 in --include) shift; while [[ $# -gt 0 && ! ($1 =~ ^--) && ! ($1 =~ ^-[^-]) ]]; do INCLUDE_PATTERNS+=("$1"); shift; done ;; --exclude) shift; while [[ $# -gt 0 && ! ($1 =~ ^--) && ! ($1 =~ ^-[^-]) ]]; do EXCLUDE_PATTERNS+=("$1"); shift;