App catalogs eating away RAM

please follow the related issue where we will investigate/fix the issue. Thanks!

It was pretty much a permanent growth. Take a look at the screenshot.

• On February 2nd I added Truecharts for the first time.

• On February 19th Truenas became very slow when accessing file services. So I logged in, saw the memory consumption and rebooted.

• On March 1st the memory consumption started to grow again. (Interestingly enough everything was clam the days before.)

• Rebooted again on March 13th.

• Reboot on March 17th was made because memory was growing again. (This was the point when I really started to think that this is a problem you can actually reproduce and not something that just goes wrong sometimes.)

• Reboot on March 22nd was because of the upgrade to the 22.02.0.1 security update.

And then was April 9th. Free memory was once again becoming scarce. This got me thinking to what has actually changed since the memory consumption started to grow for the very first time.

• The use of file services has not changed.

• I also haven't change my pattern of logging in the web frontend around every 7 days to see if everything is alright. (So I don't think that the websocket theory is true in my case.)

• Also the virtual machine that's permanently running has not changed.

• But I have added Truecharts.

So I removed Truecharts on April 9th and rebooted. Now with only the Official app catalog installed the memory still was growing very slowly. But nowhere as fast as with Truecharts.

And this was the point when I reached the conclusion for me that I must be somehow related to the number of catalog entries in the apps section of Truenas.

can you please confirm the following facts ?

1. Does memory continue growing or does it become constant after say 15 minutes of adding the catalog ?

2. Do you experience memory issues after you remove the truecharts catalog ?

It is expected for the resource usage to go up for a few minutes after the catalog is added as system goes through it's contents but for the memory to continue growing ( even slowly ) over hours is not okay..

Please do confirm the above,thanks

I can't say that it's the catalog for sure. It's just a guess from when I saw the memory consumption growing quickly first.

I have one virtual machine running a Unifi controller. This way Scale was running with pretty much the same shares of memory regarding services and ZFS cache for a very long time and without getting closer to an out of memory situation.

To give the Kubernetes version a try I added the Truecharts catalog. And after that the memory started to grow quickly for the first time. Removed the Truecharts catalog after getting close to out of RAM, rebooted and the behavior of the memory consumption was like it was before adding Truecharts.

Right not I have only the Official catalog installed.

Hello, I'm also experiencing increasing middlewared memory use (and occasionally stuck middleware processes burning cpu).  The forum issue referenced in this issue has been updated with a new and potentially interesting find here:

https://www.truenas.com/community/threads/middlewared-having-a-memory-leak-in-22-02-0.99836/page-2#post-696910

Basically sitting on the UI dashboard page (and potentially other pages with increased WebSocket data) seems to increase middlewared memory use that doesn't seem to get released eventually.  Personally I've reached 20GB+ several times now before restarting the server or the process.  It seem worth a look.