App catalogs eating away RAM

Description

This is basically a thread on the community forum becoming a ticket since the behavior is still there in 22.02.1.

https://www.truenas.com/community/threads/middlewared-having-a-memory-leak-in-22-02-0.99836/

The app catalog(s) eat away RAM until you get an out of memory situation. And the more items the catalogs contain the faster you can notice this if you have not this much RAM.

As long as you only got the OFFICIAL catalog installed it's barely noticeable but the behavior is also there.

But as soon as you also add TRUECHARTS it becomes pretty obvious.

Problem/Justification

None

Impact

None

Activity

Show:
Waqar
May 27, 2022 at 12:48 PM

please follow the related issue where we will investigate/fix the issue. Thanks!

Whiskydrinker
May 19, 2022 at 6:31 PM

It was pretty much a permanent growth. Take a look at the screenshot.

  • On February 2nd I added Truecharts for the first time.

  • On February 19th Truenas became very slow when accessing file services. So I logged in, saw the memory consumption and rebooted.

  • On March 1st the memory consumption started to grow again. (Interestingly enough everything was clam the days before.)

  • Rebooted again on March 13th.

  • Reboot on March 17th was made because memory was growing again. (This was the point when I really started to think that this is a problem you can actually reproduce and not something that just goes wrong sometimes.)

  • Reboot on March 22nd was because of the upgrade to the 22.02.0.1 security update.

And then was April 9th. Free memory was once again becoming scarce. This got me thinking to what has actually changed since the memory consumption started to grow for the very first time.

  • The use of file services has not changed.

  • I also haven't change my pattern of logging in the web frontend around every 7 days to see if everything is alright. (So I don't think that the websocket theory is true in my case.)

  • Also the virtual machine that's permanently running has not changed.

  • But I have added Truecharts.

So I removed Truecharts on April 9th and rebooted. Now with only the Official app catalog installed the memory still was growing very slowly. But nowhere as fast as with Truecharts.

And this was the point when I reached the conclusion for me that I must be somehow related to the number of catalog entries in the apps section of Truenas.

Muhammad
May 19, 2022 at 12:38 PM

can you please confirm the following facts ?

  1. Does memory continue growing or does it become constant after say 15 minutes of adding the catalog ?

  2. Do you experience memory issues after you remove the truecharts catalog ?

It is expected for the resource usage to go up for a few minutes after the catalog is added as system goes through it's contents but for the memory to continue growing ( even slowly ) over hours is not okay..

Please do confirm the above,thanks

Whiskydrinker
May 18, 2022 at 6:21 PM

I can't say that it's the catalog for sure. It's just a guess from when I saw the memory consumption growing quickly first.

I have one virtual machine running a Unifi controller. This way Scale was running with pretty much the same shares of memory regarding services and ZFS cache for a very long time and without getting closer to an out of memory situation.

To give the Kubernetes version a try I added the Truecharts catalog. And after that the memory started to grow quickly for the first time. Removed the Truecharts catalog after getting close to out of RAM, rebooted and the behavior of the memory consumption was like it was before adding Truecharts.

Right not I have only the Official catalog installed.

Moonshine
May 17, 2022 at 2:15 PM

Hello, I'm also experiencing increasing middlewared memory use (and occasionally stuck middleware processes burning cpu).  The forum issue referenced in this issue has been updated with a new and potentially interesting find here:

https://www.truenas.com/community/threads/middlewared-having-a-memory-leak-in-22-02-0.99836/page-2#post-696910

Basically sitting on the UI dashboard page (and potentially other pages with increased WebSocket data) seems to increase middlewared memory use that doesn't seem to get released eventually.  Personally I've reached 20GB+ several times now before restarting the server or the process.  It seem worth a look.

Duplicate

Details

Assignee

Reporter

Labels

Impact

Time remaining

0m

Components

Fix versions

Affects versions

Priority

Katalon Platform

Created May 9, 2022 at 7:13 AM
Updated July 6, 2022 at 8:56 PM
Resolved May 27, 2022 at 12:48 PM