Usenetic

Hi,

I am starting to use Usenetic: I found it very great usenet client (for my tests I used Usenetic's last version)

When I use it with a very high speed download (100 MBytes/s with 40 connections to usenet server), I get a "memory full" problem on my IMac (16GB RAM, High Sierra 10.13.1, Cpu I7 3,4Ghz 4 core, NZB download directory on SSD) after 5 minutes: Usenetic process consumes 13GB RAM and my IMac "stops" (memory pressure is red on activity monitor).

Similar behaviour if I use speed limit to 50MBytes/s (memory pressure on red, kernel_task 9GB, Usenetic 4GB).

If these conditions I pause NZB download, Usenetic memory usage grows (slowly) until 5GB.

If I quit Usenetic in these conditions, I get a pop-up with:

Updating servers table: Done
Closing Connections: Done
Flusing disk to files: 12 files
Exporting SaveState: Waiting

Files number slowly decreases and memory pressure returns to green (kernel_task 2GB, Usenetic between 3 and 4GB): after 30 minutes flushing files number returns to 0 and then Usenetic quits.

On usenetic log file, I get thousand of these messages:

[INFO] 18:50:01.572 | [Usenetic 407057]Waiting for remaining files to be written (6)

Can you give me some suggestions?

Thanks

First of all make sure there are no daemons running that are monitoring the system usage, like TechTools.
See: this thread

If such daemon is not the case than it seems that the incoming data is faster than your disk can handle.
This can be solved by enabling the write buffer.

The write buffer can be enabled in the preferences.

Screenshot to enable the writebuffer:

Thanks for your reply but also using write buffer I have yet memory full (I choose 10 as pause threshold and 0 as resume threshold).

In my test case, NZB file has 36 files of 1GB each: minimum pause threshold files number is 10 and this is too much in this case.

Do you think will be possible to modify your application to choose also a lower number of files (for example 1, 3 and 5) as pause threshold?

Thanks in advance

1GB per file? Unfortunately some people really don't know how to post to usenet.

The next release will deal with this kind of large individual files.

What you can do In the meantime as a workaround is download each file separately.

If you are downloading from MegaSearch, don't select the fileset checkbox, but expand the fileset instead and select the first file. Then select the next file after the first is downloaded, etc.

If you are downloading from a NZB file than this is not possible, but in that case try searching the fileset in MegaSearch and follow the above steps.

I know, 1GB per file is a monstrosity.

Thanks for your support.

The update that solves this issue is almost finished and will most likely being released tomorrow.

We've replaced the write buffer with a new mechanism that limits the maximum program memory usage to a configurable value.
This way Usenetic shall only use the maximum allowed memory. When required it will pause the download so other heavy tasks can use more resources to finish their job.

Version 0.94 is released. If you haven't done already, please upgrade.
This version has fixed the memory issues when downloading huge files.

Please let me know how this works for you.

Thanks.

Hi,



I retried with same previous NZB file (36 files of 1GB each) with 0.94 version but the results are similar: I setup 5GB was memory limit.

When I reach near 50% NZB download, my system becomes (near) unresponsive: in these conditions, I paused NZB download and I noted (from activity monitor) that Usenetic memory usage fluctuate between 500MB and 3GB but "kernel_task" process memory usage was 9GB.

On download directory, I only see few files (from 01 to 10): after 20 minutes, my system returns to normal (kernel_task memory back to 1GB, usenetic only 83MB) and on download directory I see all downloaded files (from 01 to 20, of 36).

Download dir is on a internal SSD disk 256GB (Apple, APFS) with a throughput of (write) 140 MB/s.

So, I think the problem is on kernel_task memory, maybe caused from not great speed of SSD disc?

What do you think?

TIA

Yes, maybe the kernel can't handle the load when the data is coming in with such speed. Please try a lower memory limit like 2 GB and see how that works out.

I've done some testings downloading a fileset with 1.3 GB files on a 2015 Mac with an external HDD on USB3 and didn't had any issues at. Unfortunately I don't have such a fast internet connection to test on, but the 213 mb/s (= about 25 MB/s) don't give any issues at all using the above configuration.

Are you sure you don't have a program running that is monitoring the system? Especially the IO?
Someone else had the same issue and after he deinstalled TechTools the problem was gone.

I will retry with 2GB limit.

As monitoring tools, I use TGPro (to monitor Fan and temperature) and Little Snitch (to monitor network connection): I don't think these tools monitor IO.

Anyway, I will check also if there is some other tools in memory, before repeating the tests.