Skip to content Skip to sidebar Skip to footer
Home / Best Way to Download Open Directories

Best Way to Download Open Directories

Post a Comment

Open Directory Downloader

Indexes open directories listings in 100+ supported formats, including FTP(S), Google Bulldoze, Bhadoo, GoIndex, Go2Index (alternatives).

Written in C# with .NET (Core), which means it is cantankerous platform!

Downloading is not (yet) implemented, but is already possible when yous use the resulting file into another tool (for most of the formats).

Downloading with wget:
wget -ten -i theurlsfile.txt

Downloading with aria2c (Does not back up directory structure..):
aria2c -i theurlsfile.txt

If yous have improvements, supply me with a pull request! If you have a format not yet supported, delight let me know.

Releases / Binaries

For builds (64-bit) for Windows, Linux and Mac, or ARM/ARM64 builds for Pi:

https://github.com/KoalaBear84/OpenDirectoryDownloader/releases

When using the self-contained releases you don't demand to install the .Cyberspace (Core) Runtime.

Prerequisites

When you are NOT using the self-contained releases, you demand to install the latest/current Runtime version of .NET 6:

https://dotnet.microsoft.com/download/dotnet/six.0/runtime

Usage

Command line parameters:

Short Long Clarification
-u --url Url to scan
-t --threads Number of threads (default 5)
-o --timeout Number of seconds for timeout
-q --quit Quit after scanning (No "Press a key")
-c --clipboard Automatically re-create the Reddits stats once the browse is washed
-j --json Save JSON file
-f --no-urls Do not salvage URLs file
-r --no-reddit Do not prove Reddit stats markdown
-l --upload-urls Uploads urls file
-e --verbal-file-sizes Exact file sizes (WARNING: Uses HEAD requests which takes more time and is heavier for server)
--fast-scan Only employ sizes from HTML, non Head requests (Opposite of --verbal-file-sizes)
-s --speedtest Does a speed test after indexing
-a --user-agent Employ custom default User Agent
--username Username
--password Password
-H --header Supply a custom header to use for each HTTP asking. Can be used multiple times for multiple headers. See beneath for more info.
--output-file Output file to utilise for urls file
--proxy-address Proxy address, like "socks5://127.0.0.one:9050" (needed for .onion)
--proxy-username Proxy username
--proxy-countersign Proxy password
--no-browser Disallow starting Chromium browser (for Cloudflare)

Example

Windows

OpenDirectoryDownloader.exe --url "https://myopendirectory.com"

Linux

./OpenDirectoryDownloader --url "https://myopendirectory.com"

If you lot desire to learn more than or contribute, encounter the post-obit paragraphs!

Custom Headers

Headers need to be provided in the post-obit format:

              <Header Name>: <Header Value>

At that place should be but a single : and no ; in the header string. This syntax is compatible with e.yard. cURL, so that you can re-create the headers from a scroll command and re-use them with OpenDirectoryDownloader.

This means y'all can easily "false" a browser request:

  1. On the page/site y'all want to index, open up your browsers dev tools (F12 or CTRL + SHIFT + i)
  2. Go to the Network tab
  3. Reload the page
  4. Right-click on the showtime request/particular in the network tab and select Copy > Re-create as cURL (bash) (might be chosen differently, depending on your browser)
  5. The copied control ends with lots of headers (-H '<something>' -H '<something else>'). Copy just this part of the control and append it to your OpenDirectoryDownloader command, like so: OpenDirectoryDownloader --url "https://myopendirectory.com" -H 'header-proper name-i: header-value-ane' -H 'header-name-2: header-value-2' ...
    You tin of course too use other options with this or omit the --url option to use the prompt instead.

Setting some options similar --username or --user-agent might override some headers, as explicit options accept precedence. Option guild does not matter (this applies to OpenDirectoryDownloader in general).

Copying on Linux

When you want to copy (C key or -c flag) the stats at the terminate on Linux you need to have xclip installed.

Cloudflare

EXPERIMANTAL!! READ THIS FIRST!

IT WILL Not ALWAYS Piece of work!

There is experimental support for Cloudflare. When it detects a Cloudflare effect it will download a Chromium browser, start it, in which the Cloudflare protection tin can be solved. Sometimes this is a captcha which the user (you) needs to solve. For each browser session you take 60 seconds to complete. After that the browser will be killed and yous can retry on next request.

Cloudflare does somehow detect that information technology is not the normal Chromium/Chrome browser and therefore it sadly volition non always piece of work. A good tip is motion your mouse as soon as possible in the browser.

Sometimes it fails and pops up a browser for every request, and also kills it almost immediately when Cloudflare sees that there is no problem with the session. If this happens, kill the indexer!

If everyone have more than info how to get Cloudflare to piece of work better, let me know!

Docker

Every release volition automatically push an prototype to the Docker Hub:

https://hub.docker.com/repository/docker/koalabear84/opendirectorydownloader

Run it like:

docker run --rm -v c:/Scans:/app/Scans -it koalabear84/opendirectorydownloader --quit --speedtest

It will salvage the URLs files onto C:\Scans (windows), or supersede with a custom folder on other Bone-ses.

* Y'all tin can also run it without -5 c:/scans:/app/Scans if you don't want to salvage the results on your host.

Onion / Tor support

  1. Make sure the Tor is running on your machine
  2. Use the correct proxy accost annotation, default for Tor is: "socks5://127.0.0.1:9050"
  3. Start it with --proxy-address parameter

OpenDirectoryDownloader.exe --url "http://*.onion/" --proxy-address "socks5://127.0.0.1:9050"

Getting the code

For Visual Studio (Windows)

  1. Install Visual Studio: https://visualstudio.microsoft.com/vs/customs/
  • With workload: ".NET Core cantankerous-platform evolution"
  • With individual components: Code tools > Git for Windows and Code tools > GitHub extension for Visual Studio
  1. Be sure to install Git: https://git-scm.com/downloads
  2. Clone the repository by clicking "Clone or download" and click "Open in Visual Studio"

For Visual Studio Code

  1. Download Visual Studio Code: https://code.visualstudio.com/download
  2. Be certain to install Git: https://git-scm.com/downloads
  3. Clone the repository: https://code.visualstudio.com/docs/editor/versioncontrol#_cloning-a-repository
  4. More help: https://docs.microsoft.com/en-us/dotnet/cadre/tutorials/with-visual-studio-code

Building

  1. Install the newest .NET 6 SDK.
  2. git clone https://github.com/KoalaBear84/OpenDirectoryDownloader
  3. cd OpenDirectoryDownloader/OpenDirectoryDownloader
  4. dotnet build .
  5. cd bin/Debug/net6.0
  6. ./OpenDirectoryDownloader --url "https://myopendirectory.com"

For Linux (Might not be needed since .NET 6):
Then, if yous need to package information technology into a binary, you can use warp-packer

When yous take cloned the code, y'all can as well run information technology without the SDK. For that, download the "Runtime" and practice "dotnet run ." instead of build.

Google Drive

For Google Bulldoze scanning y'all demand to go a Google Drive API credentials file, it'south gratis!

You lot can use a many steps transmission option, or the 6 steps 'Quickstart' workaround.

Manual/customized:

  1. Become to https://console.deject.google.com/projectcreate
  2. Fill in Project Proper name, similar "opendirectorydownloader" or then, exit Location unchanged
  3. Change Project ID (optional)
  4. Click "CREATE"
  5. Wait a couple of seconds until the project is created and open up it (click "VIEW")
  6. On the APIs pane, click "Go to APIs overview"
  7. Click "ENABLE APIS AND SERVICES"
  8. Enter "Bulldoze", select "Google Drive API"
  9. Click "ENABLE"
  10. Become to "Credentials" menu in the left menu bar
  11. Click "CONFIGURE CONSENT SCREEN"
  12. Choose "External", click "CREATE"
  13. Fill up in something like "opendirectorydownloader" in the "Application proper name" box
  14. At the bottom click "Save"
  15. Become to "Credentials" menu in the left menu bar (again)
  16. Click "CREATE CREDENTIALS"
  17. Select "OAuth client ID"
  18. Select "Desktop app" as "Application blazon"
  19. Change the name (optional)
  20. Click "Create"
  21. Click "OK" in the "OAuth client created" dialog
  22. In the "OAuth two.0 Client IDs" section click on the only create Desktop app line
  23. In the peak bar, click "DOWNLOAD JSON"
  24. You volition get a file like "client_secret_xxxxxx.apps.googleusercontent.com.json", rename information technology to "OpenDirectoryDownloader.GoogleDrive.json" and supervene upon the i in the release

Wow, they really made a mess of this..

Alternative method (easier):

This volition 'abuse' a 'Quickstart' project.

  1. Get to https://developers.google.com/drive/api/v3/quickstart/python
  2. Click the "Enabled the Drive API"
  3. "Desktop app" will already be selected on the "Configure your OAuth client" dialog
  4. Click "Create"
  5. Click "DOWNLOAD CLIENT CONFIGURATION"
  6. Yous volition become a file similar "credentials.json", rename it to "OpenDirectoryDownloader.GoogleDrive.json" and replace the one in the release

On the first utilise, you will get a browser screen that you lot demand to grant access for information technology, and because we haven't granted out OAuth consent screen (This app isn't verified), nosotros go an extra warning. You can use the "Avant-garde" link, and use the "Go to yourappname (unsafe)" link.

Back up

If you like OpenDirectoryDownloader, please consider supporting me!

❤️ Sponsor

Contact me

Reddit https://www.reddit.com/user/KoalaBear84

DOWNLOAD HERE

Posted by: baileytakintious.blogspot.com

Post a Comment for "Best Way to Download Open Directories"