slow load/unload time for files into batch dialogue queue

[riveryeti]

I'm experimenting with file export right now, and I was unimpressed with how long it took to load a thousand .ARW files into the batch dialogue queue, and the fact that the CPU was at ~1% and disk (NVMe) was at 0.4% the whole time, which makes me think there's an inefficiency in code somewhere?

So I figured I'd be a good beta tester and I did a few tests on the batch dialogue input file queue and found a couple interesting things. I tested with 1000 images in .ARW and .DNG (converted from the same .ARW)

To clarify, by "batch dialogue queue" I mean opening the "New Batch Job" dialogue from within Affinity Photo via File/New Batch Job/ and selecting the [Add] button to add files to the processing queue.

1. It took 2.25 minutes to load 1000 .ARW into the batch dialogue queue.

2 It took 15 SECONDS to load DNGs into the dialogue queue (9x faster than ARW)

3 It took 1.2 minutes to REMOVE the .ARW OR the .DNG from the batch dialogue queue.

4. Most of the times (I think all but the first) The dialogue queue disappeared because focus shifted to the main affinity photo window and the batch dialogue became hidden behind it.

Platform was Windows 10/Threadripper 3960x with 256GB RAM and NVMe drive.

Andy

[carl123]

Out of interest what is the total file size of the 1000 .ARW files and the 1000 .DNG files?

 

[Chris B]

Hey riveryeti,

I will reply to your  post whilst pretending not to be jealous of your machine  

I think you are definitely onto something. Ideally, we should be using whatever resources are available to us. I am somewhat surprised that the CPU and disk usage was so low. I will ask a developer to comment on this.

I would be interested to see what the average machine would do in this case. I'm sure I've seen my machine take longer to do a tenth of the files you queued up. 

[Chris B]

So I've just tried it with 100 raw files and it took 15 seconds on a much slower machine. So if I was doing 1000 I imagine it would take roughly the same amount of time, maybe a bit longer on my slower machine.

I did notice a spike in CPU though. I'm not sure if your machine should be doing it much quicker but this is what I'll try and find out.

Oh, I checked on my 2017 iMac and 100 raw files loaded instantly. As soon as I clicked ok in Finder, they were in the Sources queue.

[Chris B]

@riveryeti - I've spoken with dev and we think we may have to get our hands on one of these threadrippers. Of course I've requested it for myself  🙏. However due to lockdown and everyone working from home, I don't know how soon this would happen. It might take some time if I'm being honest with you.

I really appreciate you flagging this up though. 

[riveryeti]

10 hours ago, carl123 said:

Out of interest what is the total file size of the 1000 .ARW files and the 1000 .DNG files?

 

The 1000 .ARW files total 34.7GB and the 1000 DNGs total 36.2GB. I generated the DNGs with Adobe DNG converter and the "no thumbnails" option.