batch job performance

[bestman 8]

currently you can only have parralel processing on or off when it's on for every thread you have it does that thing and when it's off only 1 task runs at a time.

i think it would be better if you can set the number of tasks it runs similtaniously up to the number of  threads you have (or if it automatically detects how many to run at a time for close to ideal speed but that sounds like a nightmare to make).

sometimes it's faster to run 3 batch jobs simultaniously with parralel processing off then 1 with it on at least on my laptop with a ryzen 5 4600H (6 cores 12  threads) it seems to be that i then don't fully run out of memory which slows down my device and sometimes even crashes photo 

[carl123]

On Windows you can use Task Manager to specify which cores (how many) an app can use 

Can't remember where it is but a Google search should find it for you

[bestman 8]

that doesn't solve the issue because the issue is more memory related and this way it does limit the cpu but it still has to allocate everything in memory so no performance improvement (actually even worse other than that the pc is more usable )

[carl123]

13 minutes ago, bestman 8 said:

that doesn't solve the issue because the issue is more memory related

You can specify the maximumj memory the app can use inside the app in the Performance settings

[Pšenda]

5 hours ago, bestman 8 said:

the issue is more memory related

In your case, the disk is also fully utilized, so swapping is probably being used, which is a very big loss in terms of performance.

[bestman 8]

5 hours ago, carl123 said:

You can specify the maximumj memory the app can use inside the app in the Performance settings

you can indeed but it just means that it'll use the disk instead 
for the "affinity" settting (yes the thing to change the number of cpu cores is called that in windows at least) it's really not something that  should be used.

 

trying to run 22 .arw files from my a7III in the batch processor at the same time just saving to jpeg with multiprocessing enabled it takes 5:31 minutes (so 5 min and 31 sec)
doing the same thing without parrallel it takes 1:56 min 
doing the same thing without parralel but running it in 2 tasks simultainiously(there's a lot of overhead from manually running it so we'll run the first one with 13 files and the second one with 9 to make up some of the time loss) we end at 1:43 min
so limiting the jobs to 2 or 3 could potentially be more efficient than 12 or 1 in my case and setting affinity in task manager only makes it worse which is expected 

[carl123]

When doing batch processing with multiple core CPUs your PC has to have the right balance of memory and cpu cores (enabled) and disk resources to get optimal performance. Otherwise enabling parallel processing can make the batch job run a lot slower

It's always been a bit of an issue so personally I always run batch jobs without Parallel Processing