- JavaScript 64.1%
- CSS 28.8%
- HTML 7.1%
MagikAppl
6332654b14
- Adds support for Chromium - Migrates to `declarativeNetRequest` API for Firefox ESR 140 incognito cookie workaround, with: - reduced permissions, and - slightly better efficiency Closes #1 See merge request archive-the-fallen/chunky-dl-webext!1 |
||
|---|---|---|
| icons | ||
| form-download.js | ||
| main.html | ||
| main.js | ||
| manifest.json | ||
| pkgreq | ||
| README.md | ||
| style.css | ||
Chunky-dl
A browser extension intended to aid in downloading and archiving a set of known attachments described across multiple "chunk" files.
Overview
Chunky-DL was created as an alternative to squishypillow's scraper, particularly for those who might be less comfortable with (or otherwise not set up to use) a command-line based solution like theirs.
Get Started
Take a look at the Setup section below if you are in need of guidance on how to install the extension from this repository. Then take a look at the Usage section below if it is unclear how to proceed and download chunks for archival.
If you are looking for help in finding the chunk metadata itself or which chunks to download, please take a look at Squishypillow's repository for archival management.
Usage
Click on the Chunky-DL extension icon on the browser's nav bar (usually to the right of the address bar, near the options and overflow menu). A new window should appear with an area to paste chunk information.
Find the chunk file on your filesystem, open it in a text editor, and then
copy the full contents of the file. Paste that into the large text area
in the Chunk-DL extension window. If everything looks right, click on the
Download chunk attachments button above the text area.
Chunky-DL will begin downloading each of the described attachments one at a time and save them in the default downloads directory specified in your browser's preferences in separate directories per chunk id. Chunk-DL uses the name for the attachment as provided in the chunk and does not save the original file's name.
Setup
To get the code from this repository working in a browser:
-
Pack the source files into a
.ziparchive (optional), then -
Install / Load the extension into your browser
Post-installation, you will also need to ensure that you can access any files to be downloaded by logging-in to the website(s) where the files are located, solving any CAPTCHAs, etc' before running the extension. If you can access it, the extension will also be able to! (More info on cookie usage)
Pack
If the repository has a Releases or Build Artifacts section, then this step can be skipped by downloading the extension archive from that location. Otherwise, the following instructions guide the creation of that same extension archive. Alternatively, extensions can usually be loaded in development modes without being packed. However, this may prevent the extension from storing data across browser sessions or add other restrictions.
Ensure that you have all of the files from this Git repository. To do this,
either update using git fetch and git merge or download the source code
by clicking the Code drop-down above the top right of the file list and
following the instructions there.
Using your preferred Zip archiving software (eg' 7-zip, Peazip, & most file
managers also have a Compress to Zip option which can be used), compress all
of the files from the top directory (aka' where manifest.json is found) into
a Zip-formatted archive. If space is a concern, the compressed files need only
include those listed in the pkgreq file (which can be read in a text editor).
This archive need not be named in any particular way, but may be difficult to
load if it does not end with the expected extension (.zip or .xpi for
Firefox or .crx for Chromium).
Install
Some browsers do not let users install arbitrary extensions, but even these browsers generally allow users to "test" extensions via some means.
Firefox
Standard versions of Firefox do not generally let users install arbitrary, non-officially-recognized extensions via the usual method. Instead, such extensions may only be "temporarily" installed for development and debugging.
To temporarily install this extension in Firefox, navigate to
about:debugging#/runtime/this-firefox
using the address bar. On the This Firefox, select the button labeled
Load Temporary Add-on.
Using the file open dialog that appears, navigate to and select either:
- the
manifest.jsonfile in the directory with the Chunky-DL source code - the
chunky-dl.zipfile you downloaded or packed yourself (see earlier), or
The extension should now be loaded and ready to use within the browser.
However, if you plan to use the extension in Private Browsing (Incognito)
Mode, make sure that the extension is allowed to "Run in Private Windows"
in the extensions manager at about:addons.
If you run into trouble, retry while following the official guide in the Firefox developer documentation
Official variants of Firefox like Developer, Nightly, and ESR as well as many derivatives such as Tor Browser honor the
xpinstall.signatures.requiredsetting underabout:configwhich can be disabled to allow permanent installation of the extension if desired.
Tested with
Firefox 140.6.0esr
Chromium
NOTE: Chromium browsers SHOULD work, but thorough testing was difficult to perform on common variants, so anticipate that issues may arise when using these browsers. All base functionality has tested to be working, though!
Chromium-based browsers (Chrome, Brave, Opera, Vivaldi, Edge, etc') generally allow users to permanently install any extension, but requires a small toggle to be enabled on the extensions management page.
To install the extension in a supported Chromium browser, navigate to
chrome://extensions/
and ensure that the 'Developer Mode` toggle in the upper right is enabled.
Then do either of the following:
- Click on
Load unpackedthen navigate to and select themanifest.jsonfile from the Chunk-DL source directory, or - Drag and drop the packed
chunky-dl.crxfile you downloaded, if available
The extension may need to be provided permission to run in Incognito Mode. To
do this, click on Details in the card that shows up for Chunky-DL, scroll
down to the section for Allow in Private, and click on the toggle to enable
it if it is not already.
Additional Documentation
Here are some additional topics that might come in handy if troubleshooting or curious about the design of the extension.
DDoS Protections and Cookies
tl;dr: If you have access to the files to download from the browser normally, so will the extension!
In order to limit the interference of anti-DDoS solutions, Chunky-DL makes use of the same cookies that the browser normally stores as a ticket to bypass such protections when downloading attachments. Using these also ensures that any other cookie-based context (such as login sessions) are available to the script anywhere that individual permissions might be needed.
What this means for users in particular is that any logins, CAPTCHAs, etc' will need to be acted on by the user before attempting to download file via this extension, but Chunky-DL will automate the downloading afterwards. As long as you--the user--have access to the files from your browser normally (just by typing the URL into the address bar), the extension will have access too!
Currently, the cookies to use are determined by the domain of the URL of the
first item in the chunk metadata file. If this is incorrect, an alternative
domain to use cookies from may be set within the chunk data by adding a
"domain": "example.com", entry to the start of the chunk file with the other
options. This is non-standard, but could help in some rare circumstances.
If you find that you are having trouble with multiple downloads failing in a row, consider refreshing the extension's cookie cache by clicking the "Refresh Cookies" button at the bottom of its main popup window.