Author Topic: alt.binz takes forever to close since new queue management  (Read 3561 times)

Offline therealjoeblow

  • Contributor
  • ***
  • Posts: 84
alt.binz takes forever to close since new queue management
« on: December 05, 2010, 06:22:15 PM »
I have a large queue, the queue file takes ~800mb on harddrive, and alt.binz uses about 1.7gb or ram when fully loaded (please don't start with the debate on whether large queues are good or bad, it's been done before and the net result is a lot of wasted effort - it's purely a user preference, some use large queues, some don't).

The problem is that since alt.binz was changed in v33.3 to use a different queue structure, it takes *forever* to close down now.

In previous versions, if I closed it and watched task manager, it would use 50% CPU for about 30-60 seconds and the memory useage would stay constant.  Then it would start giving back memory in 200-400mb chunks and be done closing in about 1 minute.  That worked *just fine* for me, I had no issue with it at all.

Now it's taking over 30 minutes to close!  The memory is returned to the system in kb chunks instead of hundreds of megabytes.

If this new queue management code is optimized for small queues, could I respectfully ask that you please put the old code back in, and give users an option in the setup screen to select which method they want to use?

It's nearly unusable now, and has been working fine in this respect in the past.

Thanks
The REAL Joe

Offline Rdl

  • Administrator
  • *****
  • Posts: 3918
Re: alt.binz takes forever to close since new queue management
« Reply #1 on: December 05, 2010, 06:27:54 PM »
I have a large queue, the queue file takes ~800mb on harddrive, and alt.binz uses about 1.7gb or ram when fully loaded (please don't start with the debate on whether large queues are good or bad, it's been done before and the net result is a lot of wasted effort - it's purely a user preference, some use large queues, some don't).

The problem is that since alt.binz was changed in v33.3 to use a different queue structure, it takes *forever* to close down now.

In previous versions, if I closed it and watched task manager, it would use 50% CPU for about 30-60 seconds and the memory useage would stay constant.  Then it would start giving back memory in 200-400mb chunks and be done closing in about 1 minute.  That worked *just fine* for me, I had no issue with it at all.

Now it's taking over 30 minutes to close!  The memory is returned to the system in kb chunks instead of hundreds of megabytes.

If this new queue management code is optimized for small queues, could I respectfully ask that you please put the old code back in, and give users an option in the setup screen to select which method they want to use?

It's nearly unusable now, and has been working fine in this respect in the past.

Thanks
The REAL Joe
Huh. That's news to me. New queue structure was implemented exactly for large queues.

Offline Rdl

  • Administrator
  • *****
  • Posts: 3918
Re: alt.binz takes forever to close since new queue management
« Reply #2 on: December 05, 2010, 06:54:41 PM »
Do you have auto save queue on xx minutes?
How long does it take for altbinz to load that queue?
How many collections are in queue?
How many files are there in queue dir?

Offline therealjoeblow

  • Contributor
  • ***
  • Posts: 84
Re: alt.binz takes forever to close since new queue management
« Reply #3 on: December 05, 2010, 07:35:45 PM »
Do you have auto save queue on xx minutes?
Yes, every 60 seconds
How long does it take for altbinz to load that queue?
~25 seconds
How many collections are in queue?
857
How many files are there in queue dir?
859

Cheers
The REAL Joe

Offline Rdl

  • Administrator
  • *****
  • Posts: 3918
Re: alt.binz takes forever to close since new queue management
« Reply #4 on: December 05, 2010, 07:39:12 PM »
And EVERY time you try to exit it takes 30min?
How long does it take to save queue every 60sec?

Offline therealjoeblow

  • Contributor
  • ***
  • Posts: 84
Re: alt.binz takes forever to close since new queue management
« Reply #5 on: December 09, 2010, 02:17:35 AM »
And EVERY time you try to exit it takes 30min?
How long does it take to save queue every 60sec?

Sorry, must have been something pooched earlier - I tested it over the past 2 days, and while running it looks like it's only taking ~400mb now instead of the 1.7gb before, and now it's closing in 30~60 sec.

Thanks for replying anyway
The REAL Joe

Offline therealjoeblow

  • Contributor
  • ***
  • Posts: 84
Re: alt.binz takes forever to close since new queue management
« Reply #6 on: December 15, 2010, 04:45:22 AM »
OK, so after considerably more testing, I can verify it takes a *long* time to close with the new queue.  It's not >30 mins like I reported earlier, there was something wrong with my system then; but it is between 10 and 15 minutes regularly.  Ive verified a dozen times over the last several days with reboots in between to be sure.

With the old queue. memory would be released in 200-400mb chunks when I watched it in taskmanager, and it would close in 30 or so seconds.  Now memory releases in 2-4mb chunks and it takes forever.

Can you make queue management the old way an option RDL?  Or is that no longer possible?

Cheers,
The REAL Joe

Offline Hecks

  • Contributor
  • ***
  • Posts: 2011
  • naughty cop
Re: alt.binz takes forever to close since new queue management
« Reply #7 on: December 15, 2010, 08:13:36 AM »
Unfortunately, nobody else seems able to reproduce your problem. We've tested, and so far the results are all in seconds rather than minutes for very large queues of the type you've described. There's also no logical reason why the new approach should take so long, since considerably less data is saved to disk.

Have you tried on another machine?

Offline Rdl

  • Administrator
  • *****
  • Posts: 3918
Re: alt.binz takes forever to close since new queue management
« Reply #8 on: December 18, 2010, 12:39:55 PM »
OK, so after considerably more testing, I can verify it takes a *long* time to close with the new queue.  It's not >30 mins like I reported earlier, there was something wrong with my system then; but it is between 10 and 15 minutes regularly.  Ive verified a dozen times over the last several days with reboots in between to be sure.

With the old queue. memory would be released in 200-400mb chunks when I watched it in taskmanager, and it would close in 30 or so seconds.  Now memory releases in 2-4mb chunks and it takes forever.

Can you make queue management the old way an option RDL?  Or is that no longer possible?

Cheers,
The REAL Joe
I'm 99% sure something is wrong with your computer. Do you have write back enabled on HD which stores queue?
I could leave both queue save/load routins, but then I would have to maintain both with future changes and I'm not really willing to do that.
As Hecks said, once you reported problem, we've tested with queues larger then yours and in worst case scenario (when all data needs to be saved) it took around ~40 seconds to write it (~30sec to load it) on a regular 3 years old HD and around 10 sec on SSD.
That's the worst case scenario that should never happen. Ussualy with moderate queue changes it should take sub 1sec.