Problem copying many files and memory down
  • Paladin of the Pegasos
    Paladin of the Pegasos
    Papiosaur
    Posts: 1134 from 2003/4/10
    From: France
    Hello all,

    i have a big problem with Ambient, when i copy many files (6 Gb), system memory (1,5 Gb) down and not free after the end of the copy. i copy many files from NTFS to SFS.

    Now i have just 320 Mb of free memory only...

    Any ideas ?

    Thanks for your help :-)

    [ Edité par Papiosaur 25.08.2018 - 13:19 ]
  • »25.08.18 - 12:14
    Profile Visit Website
  • MorphOS Developer
    Piru
    Posts: 384 from 2003/2/24
    From: finland, the l...
    If you use command line "copy" command to copy the files, does the memory leak occur?

    If you manage to exit the NTFS filesystem with MOSSYS:C/FSDie, does the memory come back?
  • »25.08.18 - 18:03
    Profile
  • Order of the Butterfly
    Order of the Butterfly
    KennyR
    Posts: 349 from 2003/3/4
    From: #AmigaZeux, Gu...
    Quote:

    Papiosaur wrote:
    Hello Piru, i'm testing...

    another problem, NTFS volume are full and write protected, normal ?



    How are you mounting these volumes?
  • »26.08.18 - 16:03
    Profile
  • Paladin of the Pegasos
    Paladin of the Pegasos
    Papiosaur
    Posts: 1134 from 2003/4/10
    From: France
    @Piru :

    all is OK with command copy

    with FSDie, the partition disappear but memory is always low.
    What is the usage for FSDie please ? What is his utility ?
    Question what volume i must die, source or destination?
    Partition come back after reboot (ouf)

    Maybe with problem of memory is for other filesystem too (from Ambient copy), i will test from SFS to SFS too.


    @KennyR:

    I've do anything, NTFS filesystem preinstalled on MorphOS 3.11 i suppose :-)

    [ Edité par Papiosaur 26.08.2018 - 17:39 ]
  • »26.08.18 - 16:10
    Profile Visit Website
  • MorphOS Developer
    Piru
    Posts: 384 from 2003/2/24
    From: finland, the l...
    Quote:

    another problem, NTFS volume are full and write protected, normal ?

    that is normal.
  • »26.08.18 - 16:55
    Profile
  • MorphOS Developer
    Piru
    Posts: 384 from 2003/2/24
    From: finland, the l...
    Quote:

    Papiosaur wrote:
    @Piru :

    all is OK with command copy


    Ok so the problem is likely related to ambient itself then.

    Quote:


    with FSDie, the partition disappear but memory is always low.
    What is the usage for FSDie please ? What is his utility ?
    Question what volume i must die, source or destination?
    Partition come back after reboot (ouf)


    As I said earlier: NTFS. I just wanted to know if the memory was held by NTFS filesystem, but apparently this is not the case.
  • »26.08.18 - 16:57
    Profile
  • Order of the Butterfly
    Order of the Butterfly
    WeiXing3D
    Posts: 326 from 2012/6/13
    From: Pacifica, CA
    Maybe this is associated to this topic, if not my apologies to Papiosaur for hijacking hos thread.

    I'm trying to create a back up pf my Work: partition which includes some large files (hdf images, and other large apps). When manually copying (drag and drop) files to a new location in a USB stick, some of this files just stop and crash the system before finishing copying.

    Then, what is the best or recommended way to create a back up of Work in an external USB drive?

    I'm preparing to install and migrate the apps installed in my existing setup (Work) in a new installation of MOS 3.11.
    MacMini G4 1.5GHz with MorphOS 3.9 FPGA MiST w/AmigaOS 3.9 (PFS3), FPGA Replay w/AmigaOS3.9, Amiga 1200 SCSI CD-RW, X-SURF 100 w/Rapidroad USB, External FDD with Chinon and Gotek units and Acer Aspire One ICAROS
  • »26.08.18 - 19:31
    Profile
  • Order of the Butterfly
    Order of the Butterfly
    Tcheko
    Posts: 435 from 2003/2/25
    From: France
    Quote:

    WeiXing3D wrote:

    Then, what is the best or recommended way to create a back up of Work in an external USB drive?




    Format the USB drive with an Amiga file system (mandatory for preserving file attributes!).

    Code:

    copy source:#? destination: all clone quiet


    Should do the job. Just be patient. :)
    Quelque soit le chemin que tu prendras dans la vie, sache que tu auras des ampoules aux pieds.
    -------
    I need to practice my Kung Fu.
  • »26.08.18 - 19:41
    Profile Visit Website
  • Order of the Butterfly
    Order of the Butterfly
    KennyR
    Posts: 349 from 2003/3/4
    From: #AmigaZeux, Gu...
    Quote:

    WeiXing3D wrote:
    Then, what is the best or recommended way to create a back up of Work in an external USB drive?


    There's no generic best way.

    If you format in a native MOS filesystem like FFS, SFS or PFS, you won't be able to access them on anything but MorphOS or an Amiga. Those kernel modules on Linux are now old and deprecated. If the worst happens and you no longer have access to MorphOS or Amiga, it'll be a battle to get them back. But as pointed out, it's the only way of preserving Amiga file attributes and file comments.

    If you format in NTFS or exFAT, performance on a non-Windows machines will be poor and/or limited. And there will be no way for you to fix corrupted volumes except on a Windows machine. I've always thought NTFS was awful for removable drives and seems to suffer from all sorts of corruption and file permissions issues when used for that. exFAT was supposed to fix that but instead became the black sheep, poorly supported by everyone.

    If you format in EXT3/EXT4, performance on non-Linux/BSD systems will be poor and/or limited. And you will need Linux to repair corrupted volumes.

    FAT is probably the best all-rounder and compatible everywhere but has the filename length, partition size and file restrictions many find unacceptable. And you'll STILL need a Windows machine to fix problems.

    [ Edited by KennyR 26.08.2018 - 22:08 ]
  • »26.08.18 - 22:06
    Profile
  • MorphOS Developer
    Piru
    Posts: 384 from 2003/2/24
    From: finland, the l...
    Recent tar (should be included with the SDK) includes support for amiga file attributes and comments. You need to use --xattrs option to enable the support.

    Archive directory t:foo and save it to backup:foo.tar.gz:

    Code:
    tar --xattrs -z -c -v -f backup:foo.tar.gz -C t: foo


    Extract the archice to t: restoring the amiga attributes:

    Code:
    tar --xattrs -z -x -v -f backup:foo.tar.gz -C t:


    Note that unix systems don't know how to restore the amiga file attributes or comments. Neither some old builds of tar.

    Technical notes: This support was realized by adding a fake "xattrs" layer to ixemul.library. It reports amiga attributes and comments as special xattrs. When tar is compiled against a recent MorphOS SDK, it will automatically pick up xattrs support (just ./configure and make).

    File amiga protection modes are stored in "user.amiga.mode" xattr. The data is a 32bit big endian byte order integer. The amiga file comment is stored in "user.amiga.comment" xattr. The data is ISO 8859-1 (latin1) bytes forming the file comment.

    So inclined may check the ixemul source code. The implementation is at:
    Code:
    library/listxattr.c
    library/setxattr.c
    library/removexattr.c
    library/getxattr.c
    include/sys/xattr.h


    PS. MorphOS xadmaster.library Tar slave supports these xattrs as well. The implementation is at:
    Code:
    clients/Tar.c



    PPS. (L)GPL isn't hard.

    [ Edited by Piru 27.08.2018 - 00:52 ]
  • »26.08.18 - 22:41
    Profile
  • Order of the Butterfly
    Order of the Butterfly
    KennyR
    Posts: 349 from 2003/3/4
    From: #AmigaZeux, Gu...
    Well there you go, if you still need to use NTFS or EXT then I recommend you use Piru's new tar build to backup files. They will keep the Amiga attributes but all files still be accessible from all other systems in the event of disaster. It's also a better use of space as the cluster size on most NTFS filesystems wastes loads of physical space with lots of relatively small files such as from a MorphOS system. Not to mention the mess it makes of the MFT.

    I think it may be "safer" to store one giant file on NTFS as well rather than lots of little ones, but don't quote me on that.

    Tar isn't very flexible (being mostly just a sequential blob of concatenated files), so it's not the kind of backup where you can easily or quickly restore a single file. But it's extremely robust and reliable and there's really not much at all can go wrong with it: there's no index to get corrupted and will still unpack even if the file itself is truncated.

    ...but all this of course doesn't fix Ambient's memory leak.
  • »27.08.18 - 01:15
    Profile