Will donate to bounty to get rid obnoxious Webkit RAM bug?
  • Moderator
    Kronos
    Posts: 2237 from 2003/2/24
    @red

    7 hours uptime, and sofar no need to restart OWB after visiting plenty sites.

    60% (about 1GB) of RAM used, so it probraly won't last for days, but 1 day is no problem.

    I can't even imagine what crappy sites I would need on constant reload to run into trouble several times an hour.
  • »10.05.15 - 14:39
    Profile
  • Paladin of the Pegasos
    Paladin of the Pegasos
    Jupp3
    Posts: 1193 from 2003/2/24
    From: Helsinki, Finland
    This definitely affects every webkit-based browser out there. Sure, on many platforms it only means that leaked memory will slowly pile up in swap (which doesn't really slow down system that much, as once there, it never needs to be accessed). Sure, it goes "via ram" (swapped out when some other program needs it), but that's about it.

    Basically means that if you are low on memory + swap or do some REALLY heavy browsing, you have to restart browser once or twice a day.

    This far, I intentionally left out one very major hardware group: mobile phones & tablets. There, these memory issues are much worse, due to less ram & swap.

    The point is, if it was easy to fix, some of the "big players" (Apple, Google, KDE, Opera...) would have already done that. If they can't, what hope would a lone MorphOS coder have?

    Of course Firefox engine is "at least as bad", and considering even Apple picked 3rd party solution, rather than creating an own closed one should prove how big task creating a proper web engine from scratch would be.

    As for (not) checking memory allocations, why isn't it done then?

    Simple answer: (on some platforms, namely those it was developed on), the only difference memory allocations would do is increase binary size (and memory requirements) slightly.

    Consider following (slightly technical):
    Quote:


    // Assuming size is >0 and either there is or isn't enough memory available:
    if((allocation=malloc(size)))
    {
    // MorphOS never ends up here, unless there was enough memory for allocation.
    allocation[0]=value;

    // On linux, if there was not enough memory available, kernel notices it during the previous call, and kills a seemingly random task (which might or might not be the one that caused running out of memory) to free memory.

    // If allocation was successful, MorphOS always reaches this point without problems.
    }
    else
    {
    // Memory allocation failed.
    // MorphOS ends up here, if there wasn't enough memory available.
    // Linux practically never ends up here, especially on 64bit systems.
    // Linux CAN end up here, if user tries to make an allocation, that exceeds the virtual memory the task has left. This has nothing to do with available memory, and is basically "how many different addresses can be represented by 32/64/whatever bits. On 32bits that's 4 gigabytes (- some used by kernel), on 64 bit... Well, each additional bit doubles the amount.
    }



    That assuming "default" linux settings. As you can see, the allocation check doesn't really make much difference there.

    Fab:
    Would it be possible to make download manager a somewhat separate program, that would crash independently from OWB? (Meaning, if I have to quit OWB, downloads would still keep running)
  • »10.05.15 - 14:40
    Profile Visit Website
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Andreas_Wolf
    Posts: 12075 from 2003/5/22
    From: Germany
    > if it was easy to fix, some of the "big players" (Apple, Google, KDE, Opera...)
    > would have already done that.

    These "big players" don't use WebKit1 anymore anyway but WebKit2 or Blink*, so their browsers are not affected. There is no incentive for them to fix this WebKit1 issue at hand, whether they are able to or not.

    * Blink itself doesn't have multi-process architecture, but existing browsers using Blink implement this feature at browser level instead of engine level.
  • »10.05.15 - 19:32
    Profile
  • Order of the Butterfly
    Order of the Butterfly
    Raf_MegaByte
    Posts: 430 from 2004/10/10
    From: Nella grande r...
    Quote:

    Yasu wrote:
    @Fab

    Is porting to v2 at all realistic? If yes, will you do it for a bounty? And how much?


    You stole the words from my mouth.

    So then let's create a bounty for Webkit_2 then. ^_^
    :)
    Bill Gates "Think!", Steve Jobs: "Think different!" So... Let these guy continue blabbering thinking and enjoy computing! We are on Amiga!
  • »11.05.15 - 00:29
    Profile
  • MorphOS Developer
    jacadcaps
    Posts: 2971 from 2003/3/5
    From: Canada
    Quote:

    Raf_MegaByte wrote:
    So then let's create a bounty for Webkit_2 then. ^_^
    :)


    You do realize that MorphOS or any other amigaoid OS won't benefit that much from WebKit2? Even if each tab is a (set of) separate processes, there still won't be a way to clean it up nicely if it crashes (you'd be surprised how often Chrome crashes and reloads a page before you even notice), and there still won't be a way to regain the lost memory. Also mind that if you run out of memory when just using a single website like Facebook with current Odyssey, you would still run out of memory with a WebKit2 based one.
  • »11.05.15 - 06:32
    Profile Visit Website
  • Order of the Butterfly
    Order of the Butterfly
    weiseb
    Posts: 210 from 2003/3/29
    My questions would be:

    What are the benefits of webkit2?
    Does it have a higher memory footprint?
    Are bugs still fixed for webkit1?
    More generally asked: Is the step avoidable or only deferrable

    @Fab
    Are you willing to do it in the next couple of month?
    Is money (an amount a bounty can realistically generate) a motivation for you?
    Are there other morphos stuff you would prefer to do, that need funding?
  • »11.05.15 - 07:27
    Profile
  • Fab
  • MorphOS Developer
    Fab
    Posts: 1331 from 2003/6/16
    Quote:

    weiseb wrote:
    My questions would be:

    What are the benefits of webkit2?
    Does it have a higher memory footprint?
    Are bugs still fixed for webkit1?
    More generally asked: Is the step avoidable or only deferrable

    @Fab
    Are you willing to do it in the next couple of month?
    Is money (an amount a bounty can realistically generate) a motivation for you?
    Are there other morphos stuff you would prefer to do, that need funding?


    WebKit2's main advantage is that it has one process by browser object, but as jaca said, it's not as much interesting for us as it could be on platforms with virtual spaces and memory protection.

    It probably has a higher memory footprint, since it creates a JS VM instance (and possibly also a WebCore instance) per browser (which IIRC is at least 4MB at initialization), plus all the process intercommunication overhead, of course.

    Bugs are still fixed for webkit1, and in fact, it doesn't matter. Actually, WTF, JavascriptCore and WebCore are common to WebKit1 and WebKit2 (same source code, only initialization differs). For the record, WebKit1/WebKit2 are *just* the highest level layers in the browser, the ones responsible for the user interface and the object initialization. In WebKit2, everything is done in a multiprocess manner (JavascriptCore and WebCore are instanciated for each browser "view"), Network, download manager, events, rendering modules are designed around a central IPC controller, and so on...

    Do I have the time to do it? No, and a bounty won't change that, and as said above, it's not that interesting, anyway. ATM, i'm working on sync'ing WebKit to a current version, which is more important IMO.

    [ Edited by Fab 11.05.2015 - 07:21 ]
  • »11.05.15 - 08:19
    Profile Visit Website
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Andreas_Wolf
    Posts: 12075 from 2003/5/22
    From: Germany
    > What are the benefits of webkit2? Does it have a higher memory footprint?
    > Are bugs still fixed for webkit1?

    Fab already answered, but let me mention that these questions are also answered therein (plus other questions you might deem interesting regarding WebKit1 vs. WebKit2 vs. Blink):

    http://events.linuxfoundation.org/sites/events/files/slides/slides_3.pdf
    (it even mentions Sputnik and OWB)
  • »11.05.15 - 09:06
    Profile
  • Paladin of the Pegasos
    Paladin of the Pegasos
    Jupp3
    Posts: 1193 from 2003/2/24
    From: Helsinki, Finland
    Quote:

    Andreas_Wolf wrote:
    > if it was easy to fix, some of the "big players" (Apple, Google, KDE, Opera...)
    > would have already done that.

    These "big players" don't use WebKit1 anymore anyway but WebKit2 or Blink*, so their browsers are not affected.

    Wow, they managed to fix all the memory leaks & improve memory handling in the 2.x version? Wouldn't have believed that by using any of the "non-affected" browsers. </sarcasm>

    Of course I was talking about WebKit "in general", new or old.

    And as others already pointed out:
    -Lots of WebKit1 issues still exist on WebKit2 (especially for us)
    -Lots of code is still shared between the versions

    But yes, there might be some memory-related fixes that haven't yet made it to WebKit1 (or are hard to backport due to architectural changes)
  • »11.05.15 - 22:08
    Profile Visit Website
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Andreas_Wolf
    Posts: 12075 from 2003/5/22
    From: Germany
    >>>> JavascriptCore allocate (lots of) memory blocks in virtual memory and these blocks
    >>>> can only be released when all runtime references created inside these blocks
    >>>> (javascript variables, objects, whatever) are released [...]. By design, in WebKit1,
    >>>> the VM is shared by all instances, and it relies on that. In WebKit2, that's different,
    >>>> each browser is isolated in a thread, with its own js VM instance, so it wouldn't
    >>>> be a problem there. At task destruction, all the memory could easily be released.

    >>> if it was easy to fix, some of the "big players" (Apple, Google, KDE, Opera...)
    >>> would have already done that.

    >> These "big players" don't use WebKit1 anymore anyway but WebKit2 or Blink*,
    >> so their browsers are not affected.

    > Wow, they managed to fix all the memory leaks & improve memory handling in the 2.x version?

    I don't know about *all* the leaks, but they managed to circumvent the type of memory leak this thread is about and thus improved memory handling in WebKit2 compared to WebKit1 by going WebKit2's multi-process architecture route, so that closing a browser tab means all associated runtime references are actually released.

    > Wouldn't have believed that by using any of the "non-affected" browsers.

    It is you who claims that the "big players" not fixing this issue in WebKit1 means it is hard to fix. It may indeed be hard (or even impossible) to fix, but how can you conclude this from the "big players" not fixing it when they "simply" worked around the issue by going multi-process and gaining even more advantages this way in today's multi-core world as a sideline? Point is, they don't even need to fix this WebKit1 issue.

    > Of course I was talking about WebKit "in general", new or old.

    There is no need to fix the issue in WebKit2 because its multi-process architecture prevents it from having impact. And there is no need for the "big players" to fix the issue in WebKit1 because they use WebKit2 (see above) or Blink (with browser-level implementation of multi-process architecture).

    > Lots of WebKit1 issues still exist on WebKit2

    Not the one this thread was started about.

    > especially for us

    The "big players" you talked about (and to which I replied) don't care for us.

    > Lots of code is still shared between the versions

    The "fix" of the issue at hand lies in the part of the code that is not shared between the versions.

    > But yes, there might be some memory-related fixes that haven't yet made it
    > to WebKit1 (or are hard to backport due to architectural changes)

    You almost got it. Go read Fab's postings and you'll know what these "fixes" to the issue at hand are.
  • »11.05.15 - 23:31
    Profile
  • Priest of the Order of the Butterfly
    Priest of the Order of the Butterfly
    connor
    Posts: 570 from 2007/7/29
    Quote:

    Fab wrote:
    WebKit2's main advantage is that it has one process by browser object, but as jaca said, it's not as much interesting for us as it could be on platforms with virtual spaces and memory protection.



    With http://aminet.net/search?query=vmm would that give us some benefit? The source code is there. Sadly I could not get VMM to run on MOS so far because it always tells me for all applications that they are not running although I started them. I do not know why it says so. But maybe someone else can get it to run. Yes, this VMM program is a hack but maybe a good one for now because it will take some more years until we get real virtual memory in MOS.
  • »12.05.15 - 19:20
    Profile
  • Priest of the Order of the Butterfly
    Priest of the Order of the Butterfly
    connor
    Posts: 570 from 2007/7/29
    slow speed and the memory consumption is the biggest problem for me in OWB. Like when I want to watch a short movie on Youtube which is less than a minute then the progress bar says it needs to download maybe 2MB. The download window is only downloading an access site. But I can stay and watch for minutes that the CPU load is full, the fan is blowing after some seconds, the video is never started to play but more than 300MB of memory is eaten up or even more until the requester pops up: Retry - Quit - Crash.
    When I do not wait for it but I press F12 before this requester then I get back about 1MB (values before and after in screenbar) but not the other 300 MB that the Youtube page needed. And the next problem is that when I close some tabs, then quit and restart OWB but with less open tabs than before, the memory was freed in the meantime but then OWB crashes again because it gets into the same trouble again. Why? Because RAM does not seem to be defragmented after freeing.
  • »12.05.15 - 19:33
    Profile
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Zylesea
    Posts: 2053 from 2003/6/4
    Quote:

    connor schrieb:
    Quote:

    Fab wrote:
    WebKit2's main advantage is that it has one process by browser object, but as jaca said, it's not as much interesting for us as it could be on platforms with virtual spaces and memory protection.



    With http://aminet.net/search?query=vmm would that give us some benefit? The source code is there. Sadly I could not get VMM to run on MOS so far because it always tells me for all applications that they are not running although I started them. I do not know why it says so. But maybe someone else can get it to run. Yes, this VMM program is a hack but maybe a good one for now because it will take some more years until we get real virtual memory in MOS.


    It is said that VM was done for MorphOS ages ago already, but MorphOS team decided against adding it to MorphOS. AFAU mainly because it added no real beenfit as it has also the 1.5 GB limitation snd was rather slow.

    But as there are machines that cannot upgraded to physical 1.5GB (mac mini, Efika,..) and SSDs changed the behavior (access speed) of hdds I would suggest a reconsideration about adding the option for VM eventually. Actually I experience quite a difference with my mini with 1 GB and my Powerbook with 1.5 GB. The latter virtually never runs out of mem, while with the mini this is an issue (let's not even start to talk about Efika...)
    --
    http://via.bckrs.de

    Whenever you're sad just remember the world is 4.543 billion years old and you somehow managed to exist at the same time as David Bowie.
    ...and Matthias , my friend - RIP
  • »13.05.15 - 00:02
    Profile Visit Website
  • Order of the Butterfly
    Order of the Butterfly
    weiseb
    Posts: 210 from 2003/3/29
    @Fab
    Thank you, for taking the time to answer my questions

    @Andreas_Wolf
    Thanks for the link, interesting addition
  • »13.05.15 - 07:15
    Profile
  • Paladin of the Pegasos
    Paladin of the Pegasos
    Jupp3
    Posts: 1193 from 2003/2/24
    From: Helsinki, Finland
    Quote:

    Zylesea wrote:It is said that VM was done for MorphOS ages ago already, but MorphOS team decided against adding it to MorphOS. AFAU mainly because it added no real beenfit as it has also the 1.5 GB limitation snd was rather slow.



    Yeah, it definitely would have all the same limitations, that already make it completely pointless on AmigaOS4.

    With virtual memory (which doesn't have anything to do with swapping) it would be more interesting, but that would be a bigger change that would break lots of backwards compatibility, and not just with "old 68k software".
  • »13.05.15 - 12:44
    Profile Visit Website
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Andreas_Wolf
    Posts: 12075 from 2003/5/22
    From: Germany
    > WebKit2 [...] is not as much interesting for us as it could be on platforms with
    > virtual spaces and memory protection. [...] Do I have the time to do it? No, and
    > a bounty won't change that, and as said above, it's not that interesting, anyway.

    You're doing it anyway?

    http://translate.google.com/translate?sl=de&tl=en&u=http://www.a1k.org/forum/showpost.php?p=836842
  • »20.05.15 - 22:44
    Profile
  • Just looking around
    Frek
    Posts: 15 from 2007/9/4
    I don't believe this for a second. I worked at multiple companies that manufactures setup boxes for DVB/IPTV; and basically everyone uses WebKit (regardless of the browser used, most commonly Opera, Netfront or Ekioh) essentially no setup box has more than 512MB RAM (where of about half is reserved for system use- especially MPEG acceleration), there is no swap at all.

    Several runs their UI in a web browser, they have a second browser running HbbTV and often the same is used for arbitrary user surfing, there is no known problem with leaks that I ever heard of (atleast not to the extent it causes issues).
    The same applies to essentially all "smart tv" etc.
  • »21.05.15 - 07:23
    Profile
  • MorphOS Developer
    jacadcaps
    Posts: 2971 from 2003/3/5
    From: Canada
    Quote:

    Frek wrote:
    The same applies to essentially all "smart tv" etc.


    Nope. Plenty of smart TVs have memory issues and do have broken memory management related to JavaScript. I know at least one widely used js piece of code that'd cause TVs (Samsung, Philips) to run out of memory.
  • »21.05.15 - 08:15
    Profile Visit Website
  • Yokemate of Keyboards
    Yokemate of Keyboards
    Andreas_Wolf
    Posts: 12075 from 2003/5/22
    From: Germany
    > setup box

    ;-)
  • »21.05.15 - 09:44
    Profile
  • Fab
  • MorphOS Developer
    Fab
    Posts: 1331 from 2003/6/16
    Quote:

    Frek wrote:
    I don't believe this for a second. I worked at multiple companies that manufactures setup boxes for DVB/IPTV; and basically everyone uses WebKit (regardless of the browser used, most commonly Opera, Netfront or Ekioh) essentially no setup box has more than 512MB RAM (where of about half is reserved for system use- especially MPEG acceleration), there is no swap at all.


    Well, the leaks are still there nonetheless. But in set top boxes case, the processes using WebKit are most likely disposed as soon as possible to regain memory, and also, the UI is hopefully not as resource hungry as something like google services or facebook, for instance. :) Keep in mind the js leaks are closely related to the way the page is coded. Some pages using Javascript might not cause WebKit to leak.
  • »21.05.15 - 13:37
    Profile Visit Website
  • Just looking around
    Frek
    Posts: 15 from 2007/9/4
    Quote:

    jacadcaps wrote:
    Quote:

    Frek wrote:
    The same applies to essentially all "smart tv" etc.


    Nope. Plenty of smart TVs have memory issues and do have broken memory management related to JavaScript. I know at least one widely used js piece of code that'd cause TVs (Samsung, Philips) to run out of memory.


    I'm not saying it's impossible to get a smart TV or a setup box to run out of memory, we have that issue all the time; but except for a handful of bugs reported through the years there been not residents from previous visited pages.
  • »21.05.15 - 20:09
    Profile
  • Just looking around
    Frek
    Posts: 15 from 2007/9/4
    Quote:

    Fab wrote:
    Quote:

    Frek wrote:
    I don't believe this for a second. I worked at multiple companies that manufactures setup boxes for DVB/IPTV; and basically everyone uses WebKit (regardless of the browser used, most commonly Opera, Netfront or Ekioh) essentially no setup box has more than 512MB RAM (where of about half is reserved for system use- especially MPEG acceleration), there is no swap at all.


    Well, the leaks are still there nonetheless. But in set top boxes case, the processes using WebKit are most likely disposed as soon as possible to regain memory, and also, the UI is hopefully not as resource hungry as something like google services or facebook, for instance. :) Keep in mind the js leaks are closely related to the way the page is coded. Some pages using Javascript might not cause WebKit to leak.




    That's what I thought I said, most setup boxes uses the browser to render the UI permanently, some do however reboot when you enter standby but most don't (it's not until recently it's been a requirmenent to enter real standby at all, previously most just disabled video output and pretended to be in standby.)

    Webkit don't do JS at all, that's JSCore or V8 depending on the branch you use, if you are aware of a situation where webkit leaks using Javascript, don't hesitate to mail me cause I'll certainly forward it do the development departments of each of the mentioned browser vendors if I can reproduce it. The problem is that most require a standalone page with example code, and that can sometimes be hard to deliver if you don't know exactly what cause it; but so far for each issue I encountered where I could produce an sample it's been resolved (however I have to admit I haven't reported more than approximately 2 memory related issues in 6 years, so to me they are rare).
  • »21.05.15 - 20:15
    Profile