Memory Cache can cause corrupt downloads

[spamwerbung]

Hi,

first of all I don't read here often so I don't know if this has been reported already. I noticed that Alt.Binz may cause file corruption if memory cache is enabled. I don't know yet if it is the size of the cache or if it caused by the file size and the amount of parts of the files.

I had set the memory cache to the max. value of 2000 because I have plenty RAM (16GB) available. The file size of each file of the download was 1,000,000 bytes with ~2600 parts and 64 files in total. Some of the files were corrupt, I downloaded them again with a different provider and the files were sometimes ok, but sometimes corrupt again. Then I disabled the memory cache completely (set to 0) and tried again. Now could ALL files be downloaded correctly from all providers I have tried before and where the files was corrupt.

It seems that Alt.Binz has somewhere issues to assemble the downloaded parts correctly into the finally written downloaded file if the file size and amount of parts is very large. I have reduced the memory cache now to 1000 and will monitor Alt.Binz now if the issue happen again with such large files.

I had never any such issues before, but also never had a download with such a large file size, most of my downloads have max. 100,000 - 500,000 byte large files.

[Rdl]

Set mem cache size to something resonable like 500mb-1gb. After that if you still got corrupt downloads then you should check your memory with memtest86+ or goldmem

[spamwerbung]

After downloading the problematic release I did not came across this issue since now. However, now have I again a post with large files (~700MB) with over 1800 parts per file. I had running this download over night, the first half of the files are fine, the other half all corrupt with one missing block.

I made then some tests with a 4 GB RAM disk and as soon as the memory cache was enabled the downloaded files were corrupt. Using the RAM disk as temp folder, all downloads are fine. Restart Alt.Binz also cleaned the memory cache and downloaded files are fine again, even they were corrupt before.

So I doubt my RAM is causing the corruption. I'm afraid that Alt.Binz as 32bit application cannot handle the memory cache over longer time for larger files than 100-200MB without corrupting files, which is useless for me as I have 16 GB RAM available. Would it be possible to have a 64bit executable of Alt.Binz for testing?

If needed I can provide you with the posts I have trouble with so you can make your own tests.

[Rdl]

What's you cache set in altbinz? I wouldn't set it in this case over 1500MB.

[spamwerbung]

As advised I use max. 1000 MB, as this ensure the current downloaded file is completely cached.

[Rdl]

But is the file decoded before 300MB free cache memory is filled?
You say half of the files are ok and half not, can you check log and verify that every 2nd file is bad (looking by the download order)

[spamwerbung]

Yes, the files were decoded within seconds. I also made tests and just downloaded one single file (and also just with one single connection too) so the memory cache would not have parts of another file and is not flooded with parts but with the same result of file corruption.

I also thought it might be because of too many connections so Alt.Binz would flooded with data, but even with only one connection and downloading article by article without any error messages the corrupt files ended up corrupt again.

What I meant with "other half were corrupt":
The RAR archive has ~320 parts like file_name.partxxx.rar, where .part001.rar till .part140.rar were ok, but all files from .part141.rar to .part320.rar had one missing block except nine or ten files.

The same files downloaded without the memory cache were fine. I tested it with different usenet providers. Did not matter, as long as memory cache was in use (or Alt.Binz was not restarted), the downloaded files that were corrupt before ended up corrupt again.

It looks like that Alt.Binz struggle over some bit musters files contain (or some sort of "memory lock") and keep specific parts of a file "locked" in the memory what then would cause that the downloaded files end up corrupt all over again, because it use the corrupt parts from the "locked" memory cache for that file to puzzle the parts to a complete file. With a restart of Alt.Binz the memory cache is cleared and re-initiated so when downloading the corrupt files again, Alt.Binz won't have the corrupt memory locks/blocks and re-downloaded files (that were corrupt before) end up fine.

Could also a problem caused by the compiler you are using and not Alt.Binz itself. I don't know. I just try to help to identify the cause of the issue so it can be fixed.

Is there something hidden in Alt.Binz that I can enable so Alt.Binz would log something you can have a look at to find out what Alt.Binz was exactly is doing when a downloaded file ended up as corrupt? Because the verbose log just say "article downloaded fine" even the file end up as corrupt. But re-download the same articles without memory cache end up with an intact file.

[Rdl]

PM me the link to those large 700MB files.

[spamwerbung]

I have PM'd you the links, hope this help you.

[spamwerbung]

Any news here? Have you found something useful?

[Rdl]

Nope, I've looked through the code after your pm but couldn't find anything obvious.