Lha bails out with CRC error on unpacking
  • Order of the Butterfly
    Order of the Butterfly
    asrael22
    Posts: 273 from 2014/6/11
    From: Germany
    OK, Andreas mentioning the 6th byte led me to have a closer look.

    So here are the last console lines of the uncompress attempt:
    Code:

    Extracting: ( 19510/ 19510) B/BadCat/data/bike.sam
    Extracting: ( 3632/ 3632) B/BadCat/data/boing.sam
    Extracting: ( 0/ 0) B/BadCat/data/c/
    Extracting: ( 2324/ 2324) B/BadCat/data/c/Run
    Extracting: ( 0/ 0) B/BadCat/data/devs/
    Extracting: ( 232/ 232) B/BadCat/data/devs/system-configuration
    Extracting: ( 2992/ 2992) B/BadCat/data/down.sam
    Extracting: ( 2188/ 2188) B/BadCat/data/exe
    Extracting: ( 200/ 200) B/BadCat/data/hall of fame
    Extracting: ( 7444/ 7444) B/BadCat/data/hey.sam
    *** Error on file 'i' : Failed CRC Check
    WARNING: Skipping corrupt/extraneous data
    264 files extracted, 1 file failed.

    Operation not entirely successful.


    The next file of name 'i' is weird but it is correct. The next file in the archive is a file named 'i'.

    Then I looked with a hex editor:
    The 6th byte of the archive file is "d" => "-lhd-". Not sure what that means.

    Finding the file "hey.sam" in the hext code reveils that it is compressed with -lh5-.
    Code:
    -lh5-Ô����ÑIΩ# ˜U��g 
    �hey.sam


    Then, searching for the next occurence of "-lh" should find the 'i' file.
    And indeed.
    But it seems to be compressed with -lh5- as well.
    Code:
    -lh5-y3��Ëg��»IΩ# ÔjU��}�i�BˇB



    That leads me to think there might be a bug in MorphOS lha?



    Manfred
  • »27.04.17 - 07:04
    Profile
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Andreas_Wolf
    Posts: 9331 from 2003/5/22
    From: Germany
    > The 6th byte of the archive file is "d" => "-lhd-". Not sure what that means.

    'lhd' means the respective object (first object in the archive in this case) is a directory. As you figured out already despite my confusing comment, each object in the archive is preceded by such an indicator as different objects may have been processed using different methods.
  • »27.04.17 - 08:13
    Profile
  • MorphOS Developer
    Piru
    Posts: 374 from 2003/2/24
    From: finland, the l...
    @asrael22

    Okay that should not happen.

    I managed to find the "i" file myself, and problem is reproduced. I will need to look into this issue.

    [ Edited by Piru 27.04.2017 - 17:24 ]
  • »27.04.17 - 13:33
    Profile
  • MorphOS Developer
    Piru
    Posts: 374 from 2003/2/24
    From: finland, the l...
    Some extensive background - some of it technical - follows. So bear with me, this is rather interesting story. I had merrily forgotten all about it, but it has already been 14 years since I last laid eyes on that particular code fragment...

    Issue: Amiga and UNIX LhA archives are incompatible. That is you can have a old Amiga LhA archive that cannot be decompressed with UNIX LhA. Similarly, you can have a UNIX LhA archive that cannot be decompressed with Amiga LhA. The actual difference is regarding what values the algorithm can expect to find from the dictionary: ' ' (space) or 0 (zero). Amiga LhA uses zero, UNIX LhA uses space.

    I believe the problem was solved at some point by making sure that Amiga LhA compressor creates archives that are compatible with the UNIX LhA in all situations. I believe this might actually result in slightly larger archives vs the older incompatible compressor. However, this cannot possibly fix the already existing incompatible archives.

    Sometime around the same time a decision was made to modify the Amiga LhA decompressor to work similar to the UNIX LhA: That is, it would fail with those older archives.

    Now, here comes the dilemma: The new LhA would create archives that are always ok, regardless of which LhA you used to decompress them. However: It would now itself fail to decompress the old incompatible Amiga LhA archives that had been created earlier! I didn't like that much, since for example it lead to the situation where many existing Amiga LhA archives would just fail to extract. So I made it so that MorphOS LhA worked with the Amiga LhA archives in this corner case, instead of the UNIX ones.

    Now, this meant that occasionally you would run into UNIX LhA archives that MorphOS LhA command would fail to extract. And this is exactly what is happening here.

    To remedy the situation I decided to create somewhat of a hack:
    1. By default I now expect to get UNIX like archive at decompress.
    2. If an CRC error occurs I will try to decompress it again with the old AMIGA LhA specific setup. If the checksum is correct now the extract succeeds.

    This LhA version (2.7.14) will be available in the next MorphOS release.
  • »27.04.17 - 19:25
    Profile
  • Order of the Butterfly
    Order of the Butterfly
    asrael22
    Posts: 273 from 2014/6/11
    From: Germany
    OK, thanks for this information.

    It's unfortunate that the source code for Amiga LhA is not free.
    The m68k version can unpack it. Possibly they have applied a similar 'hack'.
    Hacks are not nice but sometimes necessary.
    Regarding point 1: the majority of archives would be 'Amiga' ones.
    When you expect a UNIX archive, wouldn't you have a higher number of CRCs (and double decompress operations) than when expecting an Amiga archive?


    Manfred
  • »27.04.17 - 20:12
    Profile
  • MorphOS Developer
    Piru
    Posts: 374 from 2003/2/24
    From: finland, the l...
    Quote:

    asrael22 wrote:
    OK, thanks for this information.

    It's unfortunate that the source code for Amiga LhA is not free.
    The m68k version can unpack it.

    Yep. The Amiga LhA decided to be compatible with UNIX LhA instead of old Amiga Archives.

    Quote:

    Regarding point 1: the majority of archives would be 'Amiga' ones.
    When you expect a UNIX archive, wouldn't you have a higher number of CRCs (and double decompress operations) than when expecting an Amiga archive?

    Nope. 99% of Amiga LhA archives have been created with LhA that never generates archives which could show the issue. The Amiga LhA compression algorithm has been tuned to avoid that.

    Amiga LhA archives that show the issue are actually quite rare, so the performance impact should be limited.
  • »27.04.17 - 20:20
    Profile
  • Order of the Butterfly
    Order of the Butterfly
    polluks
    Posts: 301 from 2007/10/23
    From: Gelsenkirchen,...
    For lh7 try lhasa
    Pegasos II G4: MorphOS 3.9, Zalman M220W
    Power Mac G3: OSX 10.3 · PowerBook 5,8: OSX 10.5, MorphOS 3.9
  • »27.04.17 - 21:58
    Profile
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Andreas_Wolf
    Posts: 9331 from 2003/5/22
    From: Germany
    > For lh7 try lhasa

    ...or XAD that comes with MorphOS :-)
  • »27.04.17 - 22:17
    Profile
  • jPV
  • Paladin of the Pegasos
    Paladin of the Pegasos
    jPV
    Posts: 1307 from 2003/2/24
    From: po-RNO
    XAD is also used when you browse into an archive from Ambient, you can double click the archive and copy files from it that way too.

    I think that XAD is somehow underrated and ignored many times, but I find it pretty great invention.

    You can unarchive about any archive with the same command ("XADUnFile archive dest:" in the simpliest form) and no need to re-learn syntax of every format specific command every time you happen to need it :) XAD can unpack zip, gzip, tar, lzx, lha, bzip2, rar, xz, and about 100 other formats just with a single command... and XAD has also made possible to browsing into all kinds of archives for example with Ambient and DOpus... we would miss a lot without it.
  • »28.04.17 - 08:34
    Profile Visit Website
  • Order of the Butterfly
    Order of the Butterfly
    asrael22
    Posts: 273 from 2014/6/11
    From: Germany
    Quote:

    jPV wrote:
    XAD is also used when you browse into an archive from Ambient, you can double click the archive and copy files from it that way too.

    I think that XAD is somehow underrated and ignored many times, but I find it pretty great invention.

    You can unarchive about any archive with the same command ("XADUnFile archive dest:" in the simpliest form) and no need to re-learn syntax of every format specific command every time you happen to need it :) XAD can unpack zip, gzip, tar, lzx, lha, bzip2, rar, xz, and about 100 other formats just with a single command... and XAD has also made possible to browsing into all kinds of archives for example with Ambient and DOpus... we would miss a lot without it.



    If I'm not completely mistaken, then 'The Unarchiver' on macOS (which I'm using regularly) uses XAD as well.


    Manfred
  • »28.04.17 - 12:01
    Profile
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Andreas_Wolf
    Posts: 9331 from 2003/5/22
    From: Germany
    > If I'm not completely mistaken, then 'The Unarchiver' on macOS [...] uses XAD as well.

    Yes, it uses libxad.
  • »28.04.17 - 13:34
    Profile
  • Order of the Butterfly
    Order of the Butterfly
    Korni
    Posts: 325 from 2006/2/23
    From: the Planet of ...
    XAD is indeed nice, but ugly slow for some reason. Results from G5 2GHz.

    http://aminet.net/game/demo/ToT_Amiga_V0.57.lha

    Time LhA x ToT_Amiga_V0.57.lha RAM:
    Time Elapsed: 6.6433s

    Time XADUnFile ToT_Amiga_V0.57.lha RAM:
    Time Elapsed: 28.0831s

    This happens also for other archives, not just lha. Also with the quiet option which can help in certain cases (when unpacking many small files).
    http://korni.ppa.pl/modkowypaczek/ | My Rifle, My Bunny, and Me
  • »28.04.17 - 15:22
    Profile Visit Website
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Andreas_Wolf
    Posts: 9331 from 2003/5/22
    From: Germany
    > XAD is [...] ugly slow for some reason. [...] This happens also for other archives, not just lha.

    I just tried with bzip2 archives and found that other than with LhA archives, XAD is only marginally slower than Bz2 command.
  • »28.04.17 - 21:18
    Profile
  • Order of the Butterfly
    Order of the Butterfly
    Korni
    Posts: 325 from 2006/2/23
    From: the Planet of ...
    Quote:

    Andreas_Wolf wrote:
    > XAD is [...] ugly slow for some reason. [...] This happens also for other archives, not just lha.

    I just tried with bzip2 archives and found that other than with LhA archives, XAD is only marginally slower than Bz2 command.


    I got the same times for bzip2. xz and gz was a bit slower when using xad. Looks like xad struggles when untaring. Also I remember xad was a lot slower with some random zip from the internet, packing tot with zip2.3 makes no diff though.

    Ram Disk:>Time XADUnTar Q talesoftamar_amiga.tar.bz2 RAM:
    Processed 2286 files and 66 directories.
    There are 2560 additional tar-bytes at fileend.

    Time Elapsed: 60.5093s

    Ram Disk:> Time tar xf talesoftamar_amiga.tar.bz2

    Time Elapsed: 60.9077s

    Ram Disk:> Time tar xf talesoftamar_amiga.tar.xz

    Time Elapsed: 24.9833s

    Ram Disk:> Time XADUnTar Q talesoftamar_amiga.tar.xz ram:
    Processed 2286 files and 66 directories.
    There are 2560 additional tar-bytes at fileend.

    Time Elapsed: 30.0210s

    Ram Disk:> Time tar xf talesoftamar_amiga.tar.gz

    Time Elapsed: 22.8524s

    Ram Disk:> Time XADUnTar Q talesoftamar_amiga.tar.gz ram:
    Processed 2286 files and 66 directories.
    There are 2560 additional tar-bytes at fileend.

    Time Elapsed: 19.6157s

    Ram Disk:> Time XADUnTar Q talesoftamar_amiga.tar ram:
    Processed 2286 files and 66 directories.
    There are 2560 additional tar-bytes at fileend.

    Time Elapsed: 13.9966s

    Ram Disk:> Time tar xf talesoftamar_amiga.tar

    Time Elapsed: 4.9367s
    http://korni.ppa.pl/modkowypaczek/ | My Rifle, My Bunny, and Me
  • »29.04.17 - 01:00
    Profile Visit Website
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Andreas_Wolf
    Posts: 9331 from 2003/5/22
    From: Germany
    > I got the same times for bzip2.

    So it's not XAD that is inherently slow but just XAD's built-in LhA client.

    > gz was a bit slower when using xad. Looks like xad struggles when untaring.
    > Also I remember xad was a lot slower with some random zip from the internet

    Does MorphOS come with commands for unpacking gz, tar and zip (except from XAD, that is)? I could find them in MOSSYS:C only for bzip2, lzma, rar and xz.
  • »30.04.17 - 12:34
    Profile
  • Order of the Butterfly
    Order of the Butterfly
    polluks
    Posts: 301 from 2007/10/23
    From: Gelsenkirchen,...
    Quote:

    Andreas_Wolf schrieb:
    Does MorphOS come with commands for unpacking gz, tar and zip (except from XAD, that is)? I could find them in MOSSYS:C only for bzip2, lzma, rar and xz.

    tar and gzip are part of the SDK

    @asrael22:
    You didn't need any hex editor, option "vv" is your friend :-)


    [ Editiert durch polluks 03.05.2017 - 20:44 ]
    Pegasos II G4: MorphOS 3.9, Zalman M220W
    Power Mac G3: OSX 10.3 · PowerBook 5,8: OSX 10.5, MorphOS 3.9
  • »30.04.17 - 12:56
    Profile
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Andreas_Wolf
    Posts: 9331 from 2003/5/22
    From: Germany
    > tar and gzip are part of the SDK

    Thanks, found them. Now when I attempt to unpack a tar.bz2 archive with tar, it says it needs lbzip2 (tar doesn't need it for gz and xz archives, though). Where can I find lbzip2?

    Edit: The error message is somewhat misleading. What tar needs is the bzip2 executable in the GG:bin/ path (which is not created when the SDK is not installed). After creating the respective assignment, it works well.

    [ Edited by Andreas_Wolf 02.05.2017 - 20:36 ]
  • »30.04.17 - 23:59
    Profile