exxos blog - random goings on

[Steve]

Did you write this tool Chris? Very cool stuff. Also nice work finding that short, that's a very useful tool to have!

[alexh]

Can the DRAM controller in the ST536 utilise 68030 cache burst mode? Is that what "L1 Cache Read" is?

[exxos]

Can the DRAM controller in the ST536 utilise 68030 cache burst mode? Is that what "L1 Cache Read" is?

Burst mode is exactly why the scores are higher. Sequential access is full burst. It was a pita to figure it all out. The original TF code had burst but there was a bit of room for improvement. L1 is where burst fits inside the cache.

[exxos]

Did you write this tool Chris? Very cool stuff. Also nice work finding that short, that's a very useful tool to have!

AI wrote it :) I'll plonk it in the download section later. It matches the speeds of other benchmarks. So seems accurate. I think having a more indepth test is super cool :)

Yeah low resistance meter.
https://www.exxosforum.co.uk/atari/last ... /index.htm

Edit / clarification:
There's been a comment elsewhere about this project along the lines of:

"Nice tool and web frontend, but ... the parser, probe-set logic, arithmetic coder, and 68k decompression routine are all original work ... IMHO would be fair to mention that the tool has been created completely by AI"

I want to address that directly, because attributing the project "completely to AI" is unfair to the work that actually went into it.

Yes, AI wrote much of the code under my direction — I've never hidden that, and I said so myself when I first posted about the project. But there's a meaningful difference between "AI wrote the code" and "AI created the tool." Those aren't the same thing.

The design of the format, the choice of algorithms, the parser strategy, and the 68k decompression stub conventions were all my decisions, tested and refined over many iterations. I ran the experimental programme — testing every common compression algorithm, sweeping context-width and probe-set parameters, exploring preprocessing transforms, evaluating MRU schemes — and made the engineering calls about what worked, what didn't, and why. I found and fixed real bugs along the way. I designed and built the web frontend through countless revisions to get the live progress feedback, file handling, and forum integration the way I wanted. The conversation log alone runs to over 200 pages.

AI handled a lot of the typing; the engineering, design, experimentation, debugging, and integration were mine. Saying I used AI doesn't mean I didn't do the work — it just means I used a tool to help write the code, the same way developers have always used the best tools available to them.
I just wanted to put together something cool and different for the community, the way I've been doing for the past 30+ years.

The atariscne.org article has been updated to mention AI involvement explicitly so there's no ambiguity there either.

[exxos]

Been experimenting with packing algorithms for the past couple of days. Just a side interest more than anything. I'm not really sure Atari Packers are much of a thing these days with mass storage devices being so cheap..

It was more out of interest than anything but I may actually put up a webpage to allow simple compressions of PRG files at some point.

Basically have my own packer routines. There's like 15 variations which it lives through to find the best compression. I also wanted to add raster effects because I think it is part of the ST heritage !

I started out using ICE packer as a reference. That seems to be doing compression tricks on actual image data which is pretty cool but hard to replicate.

I started out with basically a 15% saving on GB6 compression over ICE. But on a 700K(ish) PRG I ended up with 20% larger size than ICE But that could be down to image data I suppose..

After much puzzling in the experiments I unforgot about UPX. Last time I tried it running on a Atari just bomb so I just never bothered with it again. But it seems to be basically the gold standard these days and it pretty much blew everything else out of the water !

My current results are basically looking like this..

Code: Select all

  EXXOS:    GB6B32_EXXOS.PRG             44,384B  depack ~ 4.1s + rasters
  UPX_FAST: GB6B32_UPX_FAST.PRG          39,681B  depack ~ 1.3s (UPX BST)
  UPX_MAX:  GB6B32_UPX_MAX.PRG           37,757B  depack ~27.7s (UPX LZM)

I mean UPX wins on compression and the unpack time easily. I still have a few avenues left to explore, but I am obviously not going to get near UPX compression & performance. I did manage to beat compression by a whopping 71 bytes at one point though :lol: :roll:

I think part of the problem is that the more you compress the longer it takes to depack and actually the longer it takes to compress the thing in the first place. I mean if the compressor ran for all day long we could potentially get really good results but nobody wants to wait all day for GB6 to load :lol: so I'm just searching for the best compromises at the moment. One thing I was trying to do is not make depack take longer than actually loading the data from the floppy in the first place..

I just find it a little bit of a interesting distraction more than anything really. If anything, I think it is pretty cool that I can have my own packer after 30+ years :D

[exxos]

Code: Select all

  Option                           Size  Saving  Rasters  Pick
  ────────────────────────── ────────── ───────  ──────── ──────────
  EXXOS BPE+lazy               166,744B   47.9%  YES
  EXXOS Adaptive               161,132B   49.6%  YES
  EXXOS Huffman+DP *           152,466B   52.3%  YES      EXXOS
  EXXOS NRV2B (rasters)        153,924B   51.9%  YES      NRV
  UPX BST NRV default          130,512B   59.2%  NO       UPX_FAST
  UPX MAX -9                   130,552B   59.2%  NO
  UPX N2B --nrv2b              131,624B   58.9%  NO
  UPX N2D --nrv2d              131,172B   59.0%  NO
  UPX N2E --nrv2e              130,512B   59.2%  NO
  UPX LZM --lzma               111,644B   65.1%  NO       UPX_MAX
  UPX BRU --brute              111,644B   65.1%  NO
  UPX UBR --ultra-brute        111,644B   65.1%  NO

Those are yesterday's results..

But I'm going to drop the first Packers (EXXOS) because they are just not going to get near the NRV2 algorithms. Apparently those UPX guys have had 25 years to perfect the algorithms. So I don't think me and the AI are going to manage much better over the next few hours :lol: .. Or will we .. ;)

I'm also going to drop the last ones in the compressions as well because they literally take like 20-30 seconds to depack SYSINFO. It's just not worth it.

There is basically always going to be a compression versus speed aspect to this. While maximum compression can be achieved, like I said earlier, in terms of mass storage I don't think we really need it. And the long depack wait is just going to be torturous each time.

UPX compressions are done with the external library currently. But I am porting a "new" NRV2 compression routine over to a new script entirely. This way I don't have to depend on external libraries and this also means I can tweak the things however I like. Like adding the Raster effects ;)

Yesterday's scripts were all in Python and it was just incredibly slow. So things are being converted over to C scripts now. like 30x faster at compression. Which means we can afford to spend a bit more time looking for the best compression strategies.

I think overall there will just likely be two programs output. Basically just going to be the vanilla UPX compression. Then my patched version. I may not be able to compete with UPX directly of course, but I don't think ultimately there's going to be much difference in compression sizes at the end of the day anyway.

[exxos]

Capture.PNG

Interesting results now !

My current packer looses about 2K but is twice as fast as UPX. Problem with all this is, higher packing = more depacking time. That also depends on exactly what data is being packed.

In some tests (and other packers I tried, mostly used on Amiga) things like GB6 compress like 20% better, but then other programs compress 20% worse. So finding a middle ground is very tricky.

Packers are evil as well ! :lol: :roll: Compress more = slower depack = more code. Compress more = more code = higher file sizes ! :lol: :roll: I must have gone though like 30 packing ideas and mostly ended up involving more data to store, which packed better, sort of, but the extra overhead make the whole thing overall larger.

AI said in the end..

7.3 working-with-human patterns

Don't immediately implement user suggestions. When Chris suggests X, simulate/test FIRST before committing. Several "good ideas" were measured-bad ideas.

I'd all like to think of it as being methodical, rather than bad ideas :hide: :lol:

I have pretty much pushed everything to the limits now. Mostly I'm just trying to optimise the assembly code de-packer. Not so simple when I don't really understand assembly and neither does AI :lol:

I also need to figure out why the packer is using such incredible amounts of RAM. I mean SYSINFO pushes past 4GB, and I only have 4GB RAM on the server. :lol: Server says no and starts killing things. So if the forum goes offline, that's probably why :P

[exxos]

Interesting that the top two programs are highly compressible.. To the point where the extra decompression code overhead, which my later protocols have, actually dominate the total file size :roll:

Ironically my later compression algorithms actually compress BETTER, but the size gets bumped by the extra decompressor code overhead :roll:

Capture.PNG

I suppose that is just more of a corner case as it is unusual that a file would compress that highly in general.. I suppose anyway...But this is why I have multiple compressor options in my setup, to catch such corner cases.

Though it is not just down to the compression either. It is the amount of time spent de-packing. More compression equals more code equals higher decompression time. Other than running through the combinations and letting the user decide, there isn't really any other way of crunching the numbers other than "suck it and see" type approach.

I am trying to build in a small "training model" which takes compression times from real-world programs to build up a better estimate. Unfortunately this is purely down to the type of data in the file. So I'm not sure the estimate is useful, because it is basically churning out times like between 2mins to 30mins. Its not that it is inaccurate, the time estimate cannot know how long something will take until it actually does it because the data is variable drastically between programs. So all I can do is train my model to list the theoretical fastest and slowest times to compress.

I'm running a batch script at the moment which tests all compression algorithms and there is a patch to the decompress code that will save the decompression time to a file and have the program quit back to desktop before actually running the program. This way I can chuck all the programs in the AUTO folder and just let them all run to see how long decompression actually takes. Again, this will give a bit of a realistic approximation of decompression time but again, the user would have to manually verify which one they wanted to use anyway. Also again, more compression equals longer depack at times, which only the user can decide which trade-offs he/she/it wants.

[alexh]

Are you looking to challenged Leonard of Oxygene?

He's "Mr.Packer" at the moment having just released Strinkler which is a variant of the Amiga packer Shrinkler and L-Packer (Automatically try 3 compression algorithms on your exe : deflate, zx0, LZ4)

[exxos]

He's "Mr.Packer" at the moment having just released Strinkler which is a variant of the Amiga packer Shrinkler and L-Packer (Automatically try 3 compression algorithms on your exe : deflate, zx0, LZ4)

I looked into Shrinkler..ZX0 etc already. Those were the problems I mentioned earlier, some file types it compresses better and others worse. It seems to be geared for code compression. I think it beats my compressor by about 3 % in that respect. But on larger data blocks, it starts to loose. So basically I abandoned those compressors a few days ago from my comparisons. I don't think I checked depack times on those either. I may add it to my tests if helps on more code based file where my compressor falls down a bit on.

There is another issue that data which is already compressed needs even more care. Simple compressors fall apart very quickly the more complex things get. That's why I am trying to find a middle ground to it all. There is simply never going to be a "one size fits all" algorithm unfortunately. It really depends on what is being packed as to what method ideally suited. Which is one reason why I am doing multiple compression methods in my scripts.

Currently I am waiting for my batch script to finish so I can see the strengths and weaknesses of the various compressors.. it's going to be running for literally hours..

EDIT:

Actually it even says the same in the git page..

Note for large exes
STrinkler is your best option for 4KiB demo, no contest. But for larger exe (like 64KiB or more), the depacking time could very big. In this case, if you can afford slightly less compression ratio, you should try L-Packer (It depacks fast and always beats UPX in size)

My decompressor takes about 3.2K alone.. so no way I can compete, but I am not targeting small files like that. Small files need smaller code = less compression. But theres the thing again, the compression is probably worse than mine, but because the decompressor is smaller, the overall packed size is less.. Thats the problem, complex data needs complex code = bigger file sizes..