Memory Full with very high speed download

If you found a bug, please report it here.
Forum rules
When reporting a bug, please be as specific as possible.

To access the logfile in Finder:

If the programs crashed, please attach or email us the crash report AND logfile.

In Finder press SHIFT-CMD-G (or in menu Go->Go to Folder)
Then in the textfield enter "~/Library/Application Support/Usenetic" (including the tilde ~)
There you'll find the logfile called: Usenetic.log

Other issues:

You can always attach an image/screenshot with the problem for better understanding.
ciccio
Posts: 14
Joined: Sun Dec 03, 2017 5:03 pm

Memory Full with very high speed download

Post by ciccio » Sun Dec 03, 2017 5:55 pm

Hi,

I am starting to use Usenetic: I found it very great usenet client (for my tests I used Usenetic's last version)

When I use it with a very high speed download (100 MBytes/s with 40 connections to usenet server), I get a "memory full" problem on my IMac (16GB RAM, High Sierra 10.13.1, Cpu I7 3,4Ghz 4 core, NZB download directory on SSD) after 5 minutes: Usenetic process consumes 13GB RAM and my IMac "stops" (memory pressure is red on activity monitor).

Similar behaviour if I use speed limit to 50MBytes/s (memory pressure on red, kernel_task 9GB, Usenetic 4GB).

If these conditions I pause NZB download, Usenetic memory usage grows (slowly) until 5GB.

If I quit Usenetic in these conditions, I get a pop-up with:

Updating servers table: Done
Closing Connections: Done
Flusing disk to files: 12 files
Exporting SaveState: Waiting

Files number slowly decreases and memory pressure returns to green (kernel_task 2GB, Usenetic between 3 and 4GB): after 30 minutes flushing files number returns to 0 and then Usenetic quits.

On usenetic log file, I get thousand of these messages:

[INFO] 18:50:01.572 | [Usenetic 407057]Waiting for remaining files to be written (6)

Can you give me some suggestions?

Thanks

User avatar
Support
Posts: 143
Joined: Tue Jun 06, 2017 12:38 pm

Re: Memory Full with very high speed download

Post by Support » Sun Dec 03, 2017 7:31 pm

First of all make sure there are no daemons running that are monitoring the system usage, like TechTools.
See: this thread

If such daemon is not the case than it seems that the incoming data is faster than your disk can handle.
This can be solved by enabling the write buffer.

The write buffer can be enabled in the preferences.

Screenshot to enable the writebuffer:
Preferences_writebuffer.png
Preferences_writebuffer.png (72.39 KiB) Viewed 116328 times
If you like Usenetic, please give it 5 stars at MacUpdate: https://www.macupdate.com/app/mac/59624/usenetic

ciccio
Posts: 14
Joined: Sun Dec 03, 2017 5:03 pm

Re: Memory Full with very high speed download

Post by ciccio » Sun Dec 03, 2017 11:22 pm

Thanks for your reply but also using write buffer I have yet memory full (I choose 10 as pause threshold and 0 as resume threshold).

In my test case, NZB file has 36 files of 1GB each: minimum pause threshold files number is 10 and this is too much in this case.

Do you think will be possible to modify your application to choose also a lower number of files (for example 1, 3 and 5) as pause threshold?

Thanks in advance

User avatar
Support
Posts: 143
Joined: Tue Jun 06, 2017 12:38 pm

Re: Memory Full with very high speed download

Post by Support » Mon Dec 04, 2017 8:14 am

1GB per file? Unfortunately some people really don't know how to post to usenet.

The next release will deal with this kind of large individual files.

What you can do In the meantime as a workaround is download each file separately.

If you are downloading from MegaSearch, don't select the fileset checkbox, but expand the fileset instead and select the first file. Then select the next file after the first is downloaded, etc.

If you are downloading from a NZB file than this is not possible, but in that case try searching the fileset in MegaSearch and follow the above steps.
If you like Usenetic, please give it 5 stars at MacUpdate: https://www.macupdate.com/app/mac/59624/usenetic

ciccio
Posts: 14
Joined: Sun Dec 03, 2017 5:03 pm

Re: Memory Full with very high speed download

Post by ciccio » Mon Dec 04, 2017 4:59 pm

I know, 1GB per file is a monstrosity.

Thanks for your support.

User avatar
Support
Posts: 143
Joined: Tue Jun 06, 2017 12:38 pm

Re: Memory Full with very high speed download

Post by Support » Mon Dec 04, 2017 5:13 pm

The update that solves this issue is almost finished and will most likely being released tomorrow.

We've replaced the write buffer with a new mechanism that limits the maximum program memory usage to a configurable value.
This way Usenetic shall only use the maximum allowed memory. When required it will pause the download so other heavy tasks can use more resources to finish their job.
If you like Usenetic, please give it 5 stars at MacUpdate: https://www.macupdate.com/app/mac/59624/usenetic

User avatar
Support
Posts: 143
Joined: Tue Jun 06, 2017 12:38 pm

Re: Memory Full with very high speed download

Post by Support » Mon Dec 04, 2017 11:08 pm

Version 0.94 is released. If you haven't done already, please upgrade.
This version has fixed the memory issues when downloading huge files.

Please let me know how this works for you.

Thanks.
If you like Usenetic, please give it 5 stars at MacUpdate: https://www.macupdate.com/app/mac/59624/usenetic

ciccio
Posts: 14
Joined: Sun Dec 03, 2017 5:03 pm

Re: Memory Full with very high speed download

Post by ciccio » Tue Dec 05, 2017 5:52 pm

Hi,



I retried with same previous NZB file (36 files of 1GB each) with 0.94 version but the results are similar: I setup 5GB was memory limit.

When I reach near 50% NZB download, my system becomes (near) unresponsive: in these conditions, I paused NZB download and I noted (from activity monitor) that Usenetic memory usage fluctuate between 500MB and 3GB but "kernel_task" process memory usage was 9GB.

On download directory, I only see few files (from 01 to 10): after 20 minutes, my system returns to normal (kernel_task memory back to 1GB, usenetic only 83MB) and on download directory I see all downloaded files (from 01 to 20, of 36).

Download dir is on a internal SSD disk 256GB (Apple, APFS) with a throughput of (write) 140 MB/s.

So, I think the problem is on kernel_task memory, maybe caused from not great speed of SSD disc?

What do you think?

TIA

User avatar
Support
Posts: 143
Joined: Tue Jun 06, 2017 12:38 pm

Re: Memory Full with very high speed download

Post by Support » Tue Dec 05, 2017 6:05 pm

Yes, maybe the kernel can't handle the load when the data is coming in with such speed. Please try a lower memory limit like 2 GB and see how that works out.

I've done some testings downloading a fileset with 1.3 GB files on a 2015 Mac with an external HDD on USB3 and didn't had any issues at. Unfortunately I don't have such a fast internet connection to test on, but the 213 mb/s (= about 25 MB/s) don't give any issues at all using the above configuration.

Are you sure you don't have a program running that is monitoring the system? Especially the IO?
Someone else had the same issue and after he deinstalled TechTools the problem was gone.
If you like Usenetic, please give it 5 stars at MacUpdate: https://www.macupdate.com/app/mac/59624/usenetic

ciccio
Posts: 14
Joined: Sun Dec 03, 2017 5:03 pm

Re: Memory Full with very high speed download

Post by ciccio » Tue Dec 05, 2017 7:26 pm

I will retry with 2GB limit.

As monitoring tools, I use TGPro (to monitor Fan and temperature) and Little Snitch (to monitor network connection): I don't think these tools monitor IO.

Anyway, I will check also if there is some other tools in memory, before repeating the tests.

Post Reply