Jump to content
You must now use your email address to sign in [click for more info] ×

Batch processing of NEF-Rawfiles failed if "simultaneous processing" is active


Recommended Posts

Hello support,

I am encountering a problem in Affinity Photo 1.9.2.1035 in Windows 10 20H2 x64.
I want to process a batch of 245 RAW-files made with my Nikon Z6 (.NEF with a file size of 28-32 MB per file) with the following options:

- Save file to a different folder
- convert to JPEG using standard settings (quality 100)

As long as I do not activate the "parallel processing" option, everything runs fine, but somewhat slow since the pictures are processed one by one. As soon as I activate the option, 32 files will be processed simultaneously (32 CPU Threads), but no picture will be saved. Every file will have the "I am working" animation (see marker in example image attached), but it will run forever. The RAM usage is approx. 43GB then.

My machine contains the following parts:

- AMD Ryzen 9 5950X (16 Cores / 32 Threads)
- 64GB DDR4-3600 CL16
- Gigabyte X570 Mainboard
- AMD Radeon RX6900XT 16GB
- source disc: Samsung 1TB 980 Pro SSD (NVME)
- target disc: Corsair MP510 1TB SSD (NVME)
- newest drivers and Windows updates

Is the "parallel processing" not working a normal behaviour? Is this option designed for machines with far more CPU cores (e.g. workstations with 64C/128T) or is this a probably a buggy behaviour?

Thank you very much for your answer and your investigation.

Regards from Germany!

AF.PNG

Link to comment
Share on other sites

  • Staff

Hey Lareneg,

I've just done my own test of 272 .afphoto files and they all worked with Paralell Processing enabled. Ideally, I'd like to try with your files so we can attempt to debug it. Can you upload them to this private Dropbox folder and also attach a screenshot of the settings you're using in the Batch dialog?

https://www.dropbox.com/request/09D1Ktsjn1Irb2eKXKrV

Link to comment
Share on other sites

Hello @Chris B


thank you for your answer and your time. Since the photos were made at a private appointment and nearly all of them contain people and therefore privacy has to be taken into account, I am sorry but I cannot upload them to even a private dropbox. 

I can deliver the batch settings of course (attached).

AF_01.png

AF_02.PNG

Link to comment
Share on other sites

Just a stab in the dark but what if you try and save to a location other than the D:\tmp\ file path? I try and stay out of my tmp directories.

Mac Pro (Late 2013) Mac OS 12.7.4 
Affinity Designer 2.4.0 | Affinity Photo 2.4.0 | Affinity Publisher 2.4.0 | Beta versions as they appear.

I have never mastered color management, period, so I cannot help with that.

Link to comment
Share on other sites

  • Staff

I'm not sure if that tmp folder is just one Lareneg has made.

23 hours ago, Lareneg said:

I am sorry but I cannot upload them to even a private dropbox. 

I understand - absolutely.

I've tried with those settings and it was fine with my test images. Your machine should be more than capable to be honest. I will try and get hold of some Nikon Z6 examples.

Link to comment
Share on other sites

@Old Bruce This folder was created by me and is a normal folder (not e.g. a temp file created by the application). This should have been explained by me, thanks for asking.

@Chris B Thank you very much for your effort. If the problem cannot be reproduced or you cannot get your hands onto some Z6-RAWs, I could shoot a series of RAW-files without content (e.g. set to unsharp using manual focus) to get us some testing material?

Link to comment
Share on other sites

  • Staff

I usually use https://www.rawsamples.ch/index.php/en/nikon for my raw samples but we often keep raw files sent to us by customers in a library. I've only found a few from customers so if you could possibly shoot a few that would be grand. I could easily duplicate them so we have a couple hundred and we might be able to see where the app is falling over... so I think shooting a few with maybe a few different settings would be fine.

I would not expect you to shoot a few hundred just for this test and then upload them all.

If you want to, you can upload them into this Dropbox folder:

https://www.dropbox.com/request/LT3SqKWBBWbDXbnjPmAI

The camera is on our supported list so I can't imagine it's the actual raw file causing any bother. Saying that, there is an open issue with the Z6 images opening black or partially black. I'll know more once I get a few samples :)

Link to comment
Share on other sites

  • 3 weeks later...

@Chris B I am just uploading 261 Z6 RAWs (.NEF) to the dropbox you linked in you post above. Those are "real" pictures taken but sorted out during processing after the event. Uploading will be done in about an hour.


Even tough those pictures were taken at on open event, please treat them with the respective privacy. 
I hope those help and thank you for your support!

 

 

Link to comment
Share on other sites

  • Staff

Hey @Lareneg - I just downloaded them and ran 2x batches of each setting (with and without Parallel Processing) and both completed.

Your machine is more capable than this one I'm using for work so I'm a tiny bit perplexed as to what the issue is. 

This machine only has 16GB RAM but I did not see usage beyond 6GB. Can you go to Preferences > Performance and see what the RAM cap is on your setup? Try reducing it if you can.

Link to comment
Share on other sites

@Chris B Thank you for your answer. I tried reducing the useable amount of RAM to 16GB without success. As you can see in the Screenshot attached, Affinity Photo (updated to the last recent version 1.10.x today) will occupy the memory, but no picture will be saved (output folder is visible in the explorer).

The status of the single files is "saving JPEG" or sometimes "loading" (see arrows), but no output is written. The programm can only be closed by using the Task Manager, because it will always say "processing is still active" when trying to close it.

AF2.PNG

Link to comment
Share on other sites

@Chris B To add one more information: If i take only e.g. 11 RAWs and put them trough a parallel processing batch, they will finish, but this already takes ages: All files switch to "saving JPEG" very fast (approx. 5 seconds), but it takes more than one minute until the files are really saved. The RAM usage is around 29 GB (without cap) then.

Link to comment
Share on other sites

  • Staff

Thanks - I will try and find some more info from the developers. I don't know if it's your CPU doing something. I don't think any of us have a processor with so many cores/threads.

I've tried it on my personal machine with an i9 10850k which still doesn't have as many cores or threads but is about on par performance wise and that was fine too.

That machine has 32GB RAM and was using nowhere near as much RAM either. 

 

Link to comment
Share on other sites

  • Staff
On 7/12/2021 at 7:31 PM, Lareneg said:

but it will run forever.

Can I just double check to see what you do when this happens - does the app seemingly hang and if so, is there a hang dump from Task Manager for the app? The developers have asked if we can get hold of one as it might help identify why it's stopping.

Link to comment
Share on other sites

5 hours ago, Chris B said:

Can I just double check to see what you do when this happens - does the app seemingly hang and if so, is there a hang dump from Task Manager for the app? The developers have asked if we can get hold of one as it might help identify why it's stopping.


Thank you for your answer and your further investigation. The app does not hang, it is still responsive in terms of the "working"-animation and other buttons such as minimize / maximize. It just cannot be closed because it says "processing is still running" and therefore the only way of closing it is by using the Task Manager and killing the process.

Do you need a memory dump nevertheless?

If i can contribute in finding the cause of this (e.g. by doing some tests / giving you remote Teamviewer-Access to the machine to test stuff / ...), just let me know.

Link to comment
Share on other sites

Another data point.  I downloaded a sample .nef file and created 50 copies to test Parallel Processing.  I setup a simple Batch job as above, convert each file to jpg and save to a different location. I started the job, the first 16 files started processing: Loading, Saving JPEG, a few actually finished before the program crashed.  The CPU utilization hit 100% several times during the "Saving JPEG" phase.

Further experimentation showed that (on this system) using Parallel Processing for this job with 10 or more files crashes Photo with somewhere between zero and 3-4 files saved.  I'm attaching the .nef file and a few .dmp's.  The process seems ok (if slow) with PP turned off.
Intel i7-11700
NVIDIA GTX 1660 Ti, 8/10/21 driver
16 GB RAM

sample1.nef 8c0436d9-352e-4db3-b95f-adff8a8b1698.dmp 10735941-834b-47d9-8a96-f5a426127999.dmp 6874b18d-81b8-43e1-864c-5a05dcf0c1a2.dmp

Link to comment
Share on other sites

  • 5 weeks later...
  • Staff

Hey Lareneg,

I'm afraid not. The developers have a threadripper in the office and they are returning to the office soon so I will ask them to investigate once they have settled back in.

I've tried again today on my new work laptop and it's fine so there's surely something specific to the Ryzen.

Link to comment
Share on other sites

  • 2 months later...
  • Staff

Hi Lareneg,

I've asked the developers to try this however the spec isn't exactly the same:

Lareneg's machine = AMD Ryzen 9 5950X (16 Cores / 32 Threads)
Dev machine = AMD Ryzen 9 3950X (16 Cores / 32 Threads)

So once we've done a test we can report back. 

Link to comment
Share on other sites

  • 4 months later...

Hello Affinity Friends!

After some google searching I ended up on this forum thread. I'm having this very same problem.

I'm trying to batch process 300 ARW files (ARW is the RAW file extension from my Sony Alpha Camera). The batch process consists in only converting to JPEG.

My computer has 16gb ram, and an Intel i7 CPU (6 cores / 12 threads).

It works fine with the "parallel processing" option off. But when I turn it on, it will eventually crash after some minutes. I have noticed in the task manager that the ram use goes to its limits.

I don't mind if there is no a fix for this bug yet, but I do have a question:

Is there a way to keep this "parallel processing" option OFF as the default option? Each time I have to batch process, the option is ON by default, and if I forget to disable it (which happens to me all the time) the program will crash, and also the computer becomes very unstable because it seems that it will use all the computer resources.

Thanks for reading this!.

 

Link to comment
Share on other sites

6 minutes ago, fluons said:

Is there a way to keep this "parallel processing" option OFF as the default option? Each time I have to batch process, the option is ON by default, and if I forget to disable it (which happens to me all the time) the program will crash, and also the computer becomes very unstable because it seems that it will use all the computer resources.

Welcome to the Serif Affinity forums.

No, it seems to always default to having Parallel Processing turned on. There are no options for that that I've seen.

-- Walt
Designer, Photo, and Publisher V1 and V2 at latest retail and beta releases
PC:
    Desktop:  Windows 11 Pro, version 23H2, 64GB memory, AMD Ryzen 9 5900 12-Core @ 3.00 GHz, NVIDIA GeForce RTX 3090 

    Laptop:  Windows 11 Pro, version 23H2, 32GB memory, Intel Core i7-10750H @ 2.60GHz, Intel UHD Graphics Comet Lake GT2 and NVIDIA GeForce RTX 3070 Laptop GPU.
iPad:  iPad Pro M1, 12.9": iPadOS 17.4.1, Apple Pencil 2, Magic Keyboard 
Mac:  2023 M2 MacBook Air 15", 16GB memory, macOS Sonoma 14.4.1

Link to comment
Share on other sites

You're welcome.

-- Walt
Designer, Photo, and Publisher V1 and V2 at latest retail and beta releases
PC:
    Desktop:  Windows 11 Pro, version 23H2, 64GB memory, AMD Ryzen 9 5900 12-Core @ 3.00 GHz, NVIDIA GeForce RTX 3090 

    Laptop:  Windows 11 Pro, version 23H2, 32GB memory, Intel Core i7-10750H @ 2.60GHz, Intel UHD Graphics Comet Lake GT2 and NVIDIA GeForce RTX 3070 Laptop GPU.
iPad:  iPad Pro M1, 12.9": iPadOS 17.4.1, Apple Pencil 2, Magic Keyboard 
Mac:  2023 M2 MacBook Air 15", 16GB memory, macOS Sonoma 14.4.1

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

Terms of Use | Privacy Policy | Guidelines | We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.