Index ⇒ General Topics ⇒ Operation LW2Usenet - Backing up our uploads on newsgroups

[Molitor]

TL;DR
LW uploads are being uploaded to newsgroups for backup purposes.
Click HERE and HERE for an introduction to Usenet.
Click HERE to find the NZBs.


Introduction & background
In a 2016 post, Shattered outlined the possibility of using newsgroups to back up our uploads. His test upload from September 2010 can be found here for reference.
The outstanding benefit of newsgroups is their retention, meaning how long the files remain available on the servers. Unlike cyberlockers, files posted on newsgroups do not "expire" depending on their popularity; and nowadays, many providers reach 3000+ days of retention, meaning you can access files posted 8 years ago --provided they haven't been DMCA'd meanwhile-- and that files uploaded now should remain available for the next 8 years --provided they aren't DMCA'd in the meantime--. Binary retention is still growing for most of the providers, on a daily basis.
What I'm looking forward to is to provide a mirror for all LW uploads - from the oldass early 2000s Rips to the latest ISOs.

The main downsides
Because it can't be perfect.

• First major disappointment: Subscribing for a good Usenet access is not free. However it is usually very cheap. If you are a humble downloader, looking for a few MBs there and there, a free trial at a major provider will leave you satisfied. If you need to leech some terabytes off newsgroups, a paying subscription is probably what you'll need.
• Second drawback: all files meant to be backed up for LW2Usenet are repacked and passworded before they are posted to newsgroups. This implies that double-packed scene releases, for example, become triple-packed, hence you will sometimes get files insides archives inside archives inside archives, à la matrioshka.
  This also means that in order to get a specific file inside these encrypted archives, you need to download the whole set of archives. E.g, you want rld-btaa.r60 from the scene release of Arkham Asylum? It's only 95MB, but you'll need to download the whole 8GB set and decompress it to do so. Which brings us to the third part:

What's the point then?
Providing a long-term, last-chance backup for uploaders, and a viable option for everyday downloaders.
In other words, if you are an uploader, this mirror will likely be useful for you in case you are unable to locate the files from your original upload.
If you are a downloader, chances are that you chose to download off Usenet because the original upload was broken in some way and you are looking for an alternative before the upload is officially fixed.

Modus operandi
Files downloaded off LW will be packed into 200MB archives and uploaded to alt.binaries.archive.encrypted - with 10% recovery volume.
If the original download is broken or misses files, I upload a replacement instead when possible, e.g when it's a scene release or something easy to find - otherwise all I can do is skip it, report it and process it later if a reupload happens.
There are two movements to this leeching process. The first part consists in mirroring the uploads #1 to #9999. In the meantime, I'm also processing uploads starting from #10000 in the hopes of catching up the latest uploads (#14100 and so on as I'm writing this).

Stats

[Molitor]

Where to find them
For the uploads that have been already processed, you may find them on your favourite indexer:

Code: Select all

https://nzbindex.com/search/?poster=LW2Usenet+%3CLW2U%40Legendsworld.net%3E&more=1
https://binsearch.info/?server=2&max=250&g=alt.binaries.archive.encrypted&a=LW2Usenet+%3CLW2U%40Legendsworld.net%3E
http://www.nzbking.com/poster/LW2Usenet%20%3CLW2U%40Legendsworld.net%3E

• NZBIndex and BinSearch have a retention of 3 years, so the earliest backups from 2017-2018 are not available anymore on these indexers. However they cover the later uploads just fine.
• NZBKing has a killer retention, you will find all the uploads in there. However, its search function does not support the underscore ( _ ) character; if you are looking for upload n°1750, type in "1750" instead of "LW_1750", otherwise you will get no results.

Alternatively, you can download the NZB dumps here to keep an offline copy, and not have to rely on indexers:

Code: Select all

Uploads 1-9999: https://alterupload.com/?y48ylvr2mf73ri1tpzl2 (321MB)
Uploads 10000-17000: https://alterupload.com/?efkrr1n1v9ykct94n7nn (295MB)

The file sets are named after their LW upload number, and presented as such: LW_[mirror n°].
Ex: You want to download "Batman: Vengeance - AlcoholClone ISO". Notice in the URL how the mirror number is specified: "download.php?mirror=10100". So this upload is mirror #10100. Thus it can be found as "LW_10100" on your indexer.

All archives are protected with the password LW
Keep in mind that even after unpacking these files with the LW password, the files inside may require a second password, provided in the Uploader's notes.

[Scaryfun]

That's a massive undertaking. for all your efforts. I'm doing an update tonight, but I'll check tomorrow. It's sad when we lose games that have been upped before. So, with this, it will prevent further losses.

[Shattered]

Great job!

[Shattered]

Free Usenet providers

XS Usenet

- Download speed: 2 Mbit/s
- Connections: 5
- 25 GB data limit
- Unlimited days
- No SSL
- 1100 days binary retention

Note:

A free account needs to be renewed every month: both the username and password will change with every renewal.

Code: Select all

https://www.xsusenet.com/

Free-Usenet

- Download speed: 1 Mbit/s
- Connections: 1
- Unlimited traffic
- Unlimited days
- Free SSL included
- 3100+ days binary retention

Note:

A free account needs to be renewed every 3 hours: the password will change with every renewal.

Code: Select all

https://free-usenet.com/

[Molitor]

Update on the progress after approx. 1 month 1/2 of running the service:
-Uploads #1-1500 are now covered, as well as #10000-10600.
-Many have been skipped, though: most of them are rips uploaded by the long-away-now MasteromaN. All of these rips have an ISO equivalent anyway, so they are not exactly high-priority stuff. There's a bit of backlog due to incomplete downloads awaiting to be fixed (I have a good hundred of links to report, although I won't be reporting them all at once, that would be so mean of me)
-The recent bandwidth limitations set up by MEGA are somewhat hindering the process, especially with the most recent uploads which are mostly hosted there. But time pays off: JD2 be blessed for queueing the files and processing everything for me overnight. 4S is still very much fine in terms of bandwidth, as long as there are enough accounts to use.
-Alternate download location added, with search feature, so you can still access the NZBs when your favourite indexer is offline. (NZBIndex and Binsearch having a 1000ish days retention, having a copy of the NZBs will come in handy. NZBKing is good but a pain to browse). On this page, you can get direct links to a NZB - which can be fed to your binaries downloader.

[Scaryfun]

Just a note that some uploads in database might no longer be on site as they were reported down in the past but weren't deleted. So, those hundred might not actually be that much.

[Shattered]

Thanks again for your efforts.

I've updated my previous post with more information about the free providers. I've also changed the font colour and size to match your post.

Regarding the double or triple packing of the uploads: I think the longevity of the files is the most important.

[Molitor]

Stats are available now, mainly for self-gratification purposes but also to satisfy potential curiosity about the ongoing progress. A few notes about them:
-The success rate indicates how many uploads have been successfully backed up compared to the overall number of uploads in the range. For instance the 2000-2999 range theoretically contains 1000 uploads, but it's not the case: as mentioned above by Scaryfun, some old uploads are still present in the DB while they are not linked anymore to any game entry. There is also a number of "blank" entries which do not contain any link.
-Range 1-999 has a lower success rate because most of its uploads are old were broken long before I started backing them up. But most of them are rips which already have an ISO equivalent available.
-For the aforementioned reasons, a lower success rate doesn't mean that many important uploads have been skipped, and a 100% success rate is unachievable.
-At some point I will try to correct the stats by removing the blank and unused mirrors from the overall number of uploads. I could do so with a script that check every mirror page and looks up for the words "No links found". But even with an adjusted number, the success rate wouldn't change that much. Out of 5200 mirrors (#1 to #3400 + #10000 to #11800) I have checked so far, there are ~370 which have not been processed but are fixable and maybe ~200 which are unused and thus will never be processed.
-I also note down the size if each backup range, so we can have a slight idea of the volume of data involved.

[Scaryfun]

That's amazing work, Molitor!
It's also valuable to know which ones you find that are not working currently , that way two birds are killed with one stone.
We should then put links directly to usenet from each game entry, so people know to use it.

[Shattered]

I have searched high and low for the Ghost Recon Advanced Warfare 2 files without success but was able to get them back from Usenet. Thank you Molitor!

I've also updated the "free Usenet providers" post.

[TISKER]

Please tell us what usenet provider that you are using when uploading this gems!

to calculate how many days the gems will be available on the usenet ?

[Shattered]

The availability depends on the Usenet provider which provides the longest retention, regardless of the one which has been used to upload the games. Usenet provider Eweka has 3727 days retention at the moment, so theoretically they should be available for at least 10 years.

[TISKER]

Awesome

[Scaryfun]

Indeed, I bow to Molitor for his massive efforts.