Cloning defunct websites off archive.org ?

[exxos]

Can anyone help with such things ?

I'm still working on our wiki but a lot of links / sites are gone now. While linking to archive.org is possible, I don't know how complete those sites are etc. I would like to host a copy of such things so more than 1 copy exists. Hopefully less broken copies as well.

[prog99]

Do you have an example? In my experience it's (archive.org) fine for static content but starts getting a bit iffy with dynamically generated stuff.
(You can use wget or curl to clone a site locally)

[exxos]

Do you have an example? In my experience it's (archive.org) fine for static content but starts getting a bit iffy with dynamically generated stuff.
(You can use wget or curl to clone a site locally)

Sure

https://web.archive.org/web/20140125080 ... leaks.org/

The website for the past few years has just been some casino advert.

I would think the majority of websites with them being old will probably not be so much dynamically generated as websites are today.

[stween]

something as straightforward as

Code: Select all

wget -r --no-parent  https://web.archive.org/web/20140125080442/http://atarileaks.org/

ought to grab anything not dynamic; you might need to fiddle with the number of levels (-l) to descend on more structured sites. The archive is one of the more stable things out there, though, and generally one of the safest places for things to be.

[exxos]

Yeah I don't expect the archive to disappear anytime soon. I just think it would be more elegant and faster just a link to copies of old sites. The problem is with the archive is I don't think you can get rid of all the wayback headers etc easily ?

I copied a couple of sites as other sites were hosting two dead links and such .But cannot remember how I did that now. I know it was time-consuming. But not only that file attachments another odd random things can be missing as well.

Of course it really needs to be a community effort because all this is in mind of building the ultimate Wiki. none of us are getting any younger and with stuff always disappearing and becoming harder to find, and search engines not really ranking this stuff any more. I think we need to correlate all this information in a proper archive.

Problem is I don't want it to just end up being me doing all this work. I stopped work on the Atari music network site because nobody seemed interested in preserving the stuff. Indeed there are still probably 30-50 articles which still need porting over to the forum yet. I was asking for a long time for help but unfortunately nobody stepped forward.

The ultimate goal would be to correlate every possible scrap of information out there and dump it all onto a central wiki. The thought being ultimately that the wiki can be zipped up and shared on torrents or whatever to preserve it all.

There are younger people who are getting these machines to experiment with who don't even know that the long button floppy drives of belts which fail etc. I don't think I ever saw any solid documentation of such things. That is why I think is important for everyone to start writing the snippets of information down. As soon as all us old farts retire there is only going to be archives for such information for any future generations / people to refer back to.

There are not many remaining Atari sites up any more. If anyone in the future even going to know this stuff is on archive.org ? or know how to find it ? Any pages existing will likely drop off the face of the earth because Google is not listing them any more. I know there are archives of all sorts out there but even I cannot remember where they are any more or even if any of the old FTP servers are up.

Regulars here know I've been wanting to start a Wiki for a while now. I did actually write a wiki engine as some people are aware of but I am not sure about using it because I have other projects in mind as well now. Articles on the forum do not easily show up on google, and I don't think my website ever gets listed anywhere. So information is already disappearing. I want to transfer a lot of the content over to a more Wiki format. Again because my goal is to have something which is ultimately "zippable" all shareable in a easy format. So should I disappear and this forum dies in the future then hopefully there will be some torrents of this information out there which will live on.

[b_squared]

I think I used https://github.com/hartator/wayback-machine-downloader before, precisely to download a disappeared MSX website. Unfortunately not all files were downloaded, the original page looked more like an FTP site.

[exxos]

.All files downloaded are the original ones and not Wayback Machine rewritten versions. This way, URLs and links structure are the same as before.

Not sure I follow.. So it's going to link to original files... That would defeat the object of the way back machine preserving lost websites :roll:

[stween]

This might be worth a look: https://archivarix.com/en/ (edit: hm, maybe not as useful as it first looks. It takes the page, rebundles it into some weird PHP thing with a loader in front. So all the content turns up in the .zip file, but not structured how I'd expect for a simple page!)

I could probably fool around with the easy cases without issue, but web stuff gets complex and I can see that timesink in front of me. I'm trying to not take on extra projects right now, but I see the value in accumulating things for the wiki...

[exxos]

Interesting thanks.