The resurrection of atari-wiki.com

[chronicthehedgehog]

Something about tubes

phish flavoured spam

[elterwater]

I've done some quick analysis of the output attachments so far and have compiled a list of keywords which you can use to filter and exclude lines such as "webcam", "husband" etc. This doesn't filter everything just yet but should reduce the manual verification eyeballing to more manageable levels.

I did it by using tr to split each line into discrete word per new line then sorting and counting the words to look for obvious most frequent offenders (like husband, hotel, toyota etc) and then adding those to the excluded-keywords file to repeat again.

Output remaining entries to manually filter

Code: Select all

cat output_*.txt | grep -vif excluded-keywords.txt

Word count to pick candidates to add to the excluded-keywords.txt file

Code: Select all

cat output_*.txt | grep -vif excluded-keywords.txt | tr '_' '\n' | sort | uniq -c | sort

It's dropped the total lines from 56079 to 1659 for me to eyeball manually, but it may be more helpful for future output files too.

[exxos]

I've done some quick analysis of the output attachments so far and have compiled a list of keywords which you can use to filter and exclude lines such as "webcam", "husband" etc. This doesn't filter everything just yet but should reduce the manual verification eyeballing to more manageable levels.

Possibly could help. Thanks.

I have seen common faults in spam pages like "<BR>>". So I am trying to remove spam pages based on "faulty content" rather than keywords currently.

The next 3 letters have finished.

output_FGK.zip

(311.03 KiB) Downloaded 52 times

[exxos]

I've been checking some actual Atari pages and some don't load. Not sure why. But it causes the same errors as the spam pages . So I'm not sure my method is going to work as it was based on the assumption the proper content loaded properly. Some do, some don't

@Icky is working on e "backend" clean up on the database while I am trying to attack it from the "front end". Though as something seems broken in the wiki engine somewhere, my method probably isn't going to work very well

There is a list of " dead-end pages" which are pages which are not linked in the index anywhere. so I'll see if I can dump a list of that as I doubt spammers would take the trouble to link the spam pages to the main index anywhere.

[exxos]

I've got the AI to write some scripts to speed things up like 100x. So those scripts are currently running now.

Capture.PNG (37.79 KiB) Viewed 2193 times

[exxos]

There are actually pages with the word Atari in them which are actually spam as well

Capture.PNG (408.88 KiB) Viewed 2188 times

Also it seems none of these pages are linked to any index page. so unless you know the direct link you will never be able to find them. of course last one is likely spam

Atari Falcon motherboard revisions
Atari STE and Mega STE sound fixes
Atari ST Composite Out
Atari ST Emulation FAQ
Atari ST SCART Cable
Atari ST to Amstrad CM14 monitor
Atari Workshop ST HD Mod
Atari s Betting Its Future On Gays And Gamblers

Though some of the pages seem completely pointless anyway..

Capture.PNG (8.72 KiB) Viewed 2188 times

I think the wiki is more broken than I had ever thought possible

[exxos]

New script gets the top 100 common words, sounded good until I saw the list

Code: Select all

To: 431330
Of: 315583
The: 307385
From: 205587
How: 186270
You: 172100
For: 151330
A: 149557
Your: 147578
No: 128916
Number: 123306
Num: 122539
Wedding: 117386
Dresses: 115639
And: 113138
In: 111364
-: 106113
Information: 100582
Online: 97885
Advice: 96207
Info: 94541
Tip: 92577
Best: 66774
About: 66364
Tips: 63457
With: 60266
Is: 58440
On: 54821
What: 54136
Slot: 42896
It: 38091
Can: 36404
Business: 36143
That: 34995
Marketing: 34786
Be: 34242
Get: 33027
Out: 32661
These: 32153
Better: 31354
Ways: 29907
Do: 29615
Make: 28830
Are: 27778
s: 27747
This: 27593
Have: 26742
Will: 25689
Why: 22643
Drive: 22239
An: 21175
Know: 20248
Way: 20240
Car: 19673
Things: 19039
Unlimited: 18848
Should: 18716
Public: 18414
Weight: 18220
At: 18025
Like: 17171
Seo: 17156
More: 16967
Dogs: 16871
Most: 16756
When: 16678
Shopping: 16553
Money: 16500
Top: 16312
Simple: 16242
Video: 16041
Ideas: 16034
Or: 15997
Help: 15938
By: 15915
Web: 15810
Selling: 15530
Learn: 15027
Need: 14982
As: 14811
Methods: 14549
World: 14521
Choosing: 14511
Not: 14246
Gacor: 14243
Auto: 14107
Home: 14046
Health: 13836
Find: 13758
Movies: 13735
Speaking: 13711
Merchandising: 13598
Guest: 13228
One: 13067
All: 12999
Suggestions: 12502
t: 12466
Good: 12283
Free: 12131
Automobile: 12102

I guess I will do one for common phrases this time...

[Icky]

Well I think I have identified the spam vs the real atari wiki pages

SPAM pages count 2675334
Real Atari Wiki pages count 2942

Now to get rid of the spam

[elterwater]

You'll find that if you knock out some of the urls with words such as wedding, dresses, webcam, cams, merchandising, diploma etc you'll take out a lot of these other entries with them so the mountain becomes much smaller. Hence the keywords list I added above (and there's a few more I've added locally to my own file for now) which brings 50,000+ page titles down to about 3000 instead. It's really not that bad but it sounds like Icky has found a more robust way of filtering from the DB side now.

New script gets the top 100 common words, sounded good until I saw the list

Code: Select all

To: 431330
Of: 315583
The: 307385
From: 205587
How: 186270
You: 172100
For: 151330
A: 149557
Your: 147578
No: 128916
Number: 123306
Num: 122539
Wedding: 117386
Dresses: 115639
And: 113138
In: 111364
-: 106113
Information: 100582
Online: 97885
Advice: 96207
Info: 94541
Tip: 92577
Best: 66774
About: 66364
Tips: 63457
With: 60266
Is: 58440
On: 54821
What: 54136
Slot: 42896
It: 38091
Can: 36404
Business: 36143
That: 34995
Marketing: 34786
Be: 34242
Get: 33027
Out: 32661
These: 32153
Better: 31354
Ways: 29907
Do: 29615
Make: 28830
Are: 27778
s: 27747
This: 27593
Have: 26742
Will: 25689
Why: 22643
Drive: 22239
An: 21175
Know: 20248
Way: 20240
Car: 19673
Things: 19039
Unlimited: 18848
Should: 18716
Public: 18414
Weight: 18220
At: 18025
Like: 17171
Seo: 17156
More: 16967
Dogs: 16871
Most: 16756
When: 16678
Shopping: 16553
Money: 16500
Top: 16312
Simple: 16242
Video: 16041
Ideas: 16034
Or: 15997
Help: 15938
By: 15915
Web: 15810
Selling: 15530
Learn: 15027
Need: 14982
As: 14811
Methods: 14549
World: 14521
Choosing: 14511
Not: 14246
Gacor: 14243
Auto: 14107
Home: 14046
Health: 13836
Find: 13758
Movies: 13735
Speaking: 13711
Merchandising: 13598
Guest: 13228
One: 13067
All: 12999
Suggestions: 12502
t: 12466
Good: 12283
Free: 12131
Automobile: 12102

I guess I will do one for common phrases this time...

[Icky]

This clean up just has to be done by SQL as there is so much in all the tables.

So far I have removed:
~1.5 million users
~4.4 million revisions created by those users out of
~1.9 million pages

And that was my first pass after figuring out spam vs real users