Jump to content
You must now use your email address to sign in [click for more info] ×

Failures on batch processing


Recommended Posts

Canon 5D Mark III CR2 files -> JPEG with Convert to sRGB. Parallel processing on.

First, the process crashed twice trying to convert a folder of 506 raw files. 127 "succeeded" the first pass, 220 the second pass, the rest on the third pass. 

Second: approx. 29 files had their JPG results corrupted - these were not files that were "obviously" in process during the crashes but with parallel processing on, it's difficult to guarantee. One example is attached.

 

Aph 1.8.3, macOS 10.15.5 (system specs and "performance prefs" attached as screenshots), 

_D7A0177.jpg

Screen Shot 2020-07-04 at 3.25.17 PM.png

Screen Shot 2020-07-04 at 3.26.14 PM.png

_D7A0177.CR2

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

  • Staff

Hey Brad Brighton,

How much space do you have on disk?

Can you try pulling down the RAM usage to say 2400 and try the process again? 

I saw a similar post a week or two again where that solved it. 

Also, can you colour pick those artifacts and see what colour value they have? Do they report back as NaN?

Link to comment
Share on other sites

@Chris B Thanks for the response.

Around 600G available on the target/system disk.

On RAM usage, do you mean 24GB?

I'll see what I can come up with for recreating.

For color picking, you mean in the resultant image? I'm happy to gather the data but I'm to clear what you're asking me to do. Not all the blocks are "blank" - some have color, even in the example I posted.

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

  • Staff
22 hours ago, Brad Brighton said:

On RAM usage, do you mean 24GB?

 

Yes, but the slider will ask you in MB so just pop 2400 in there.

So if you use the colour picker and mouse over those artifacts, do they give a colour value or report NaN? Do they move around at zoom?

Link to comment
Share on other sites

52 minutes ago, Chris B said:

Yes, but the slider will ask you in MB so just pop 2400 in there.

So if you use the colour picker and mouse over those artifacts, do they give a colour value or report NaN? Do they move around at zoom?

@Chris B

I think you're missing an extra zero there unless you really mean 2.4GB and not 24GB (hence the question on it). With 24GB (roughly), crashed sooner this time 80 files processed instead of 127 (so any 'magic number' idea ruled out). Crash file attached.

WRT color picker, I'm guessing you mean inside APh? Whether the eye dropper in APh or Digital Color Meter, the colors are mapped to RGB. The blocks do not move around at zoom nor do they appear to change from the Finder preview/thumbnail to the full version opened in APh.

Of the 80 files converted prior to crashing in the test run, 7 (quick visual scan) appear to be corrupted in similar ways.

So,... both the crash and the corruption seem to reproducible in some fashion. What would you like next?

Screen Shot 2020-07-07 at 7.36.12 AM.png

_D7A0216.jpg

Screen Shot 2020-07-07 at 7.16.08 AM.png

Affinity Photo_2020-07-07-072530_SpamBook2018.crash

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

  • Staff
25 minutes ago, Brad Brighton said:

So,... both the crash and the corruption seem to reproducible in some fashion. What would you like next?

 

The .afphoto file please.

I have a feeling I've seen this elsewhere and I can't for the life of me find the potentially related post... 

Link to comment
Share on other sites

2 minutes ago, Chris B said:

The .afphoto file please.

I have a feeling I've seen this elsewhere and I can't for the life of me find the potentially related post... 

@Chris B I haven't created one for any of these files. This was a CR2->JPG batch process. What am I missing about your request?

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

Just now, Chris B said:

Ah sorry I was maybe assuming the .afphoto file was left checked on the batch. I can't reproduce it with the CR2 and I've tried a few times. Does the JPEG show the corruptiuon on export? If so, I'll have that please

The two example JPGs attached in this thread are the native outputs from the convert process. I'll attach their original CR2s. Since parallel processing is on, I suppose it's possible that these are reflecting the interrupted conversion, *if* APh writes JPG data incrementally but that seems more like a red herring than an actual deduction.

 

BTW, on the second set of images+screenshots I posted, I also posted the batch process settings dialog in case you want to confirm those settings.

Also: Whatever is causing the corruption does not appear to be inherent in the files, at least not in an obvious way. The corrupted output is different (both in content and in which files) in each of the two batch passes.

_D7A0177.CR2 _D7A0216.CR2

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

2 minutes ago, Brad Brighton said:

Does the JPEG show the corruptiuon on export? If so, I'll have that please

@Chris B And here's an exported version of a corrupted JPG opened in APh. Quick visual says the corruption hasn't changed.

_D7A0177_exported.jpg

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

@Chris B And finally, I can do a private upload of the folder of images (~13GB of CR2 RAW) if you want to attempt a direct repro.

More info on "why/what I was trying to accomplish": I was exploring the batch feature (only used it a few times previously), so I was attempting to make the functional equivalent of a JPG contact sheet (as opposed to the CR2 QuickLook I could already do).

I don't shoot RAW+JPG because I don't often want/care to process the entire shoot; I pick and choose. This time, I did the direct color batch, then used a batch on those results to get B&W results so I could eyeball what ones I wanted to "production process" and in what fashion.

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

  • Staff

Thanks for all the info! I would really like to have a stab at this so I've made you a Dropbox folder:
https://www.dropbox.com/request/RVsCN94m1W3gHxyBOW72

Once the upload has finished, can you give me a nudge as I won't get a notification otherwise.

Thanks! 

Link to comment
Share on other sites

On 7/9/2020 at 7:46 AM, Chris B said:

Thanks for all the info! I would really like to have a stab at this so I've made you a Dropbox folder:
https://www.dropbox.com/request/RVsCN94m1W3gHxyBOW72

Once the upload has finished, can you give me a nudge as I won't get a notification otherwise.

Thanks! 

Consider yourself nudged, please! The zip didn't save any space but did make it a convenient one-file upload.

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

4 hours ago, Chris B said:

Sorry Brad the file is enormous so I'm having difficulty downloading it in one go. I've connected an ethernet cable and it's looking good at the moment. 

Oops. I thought I was doing a favor by bundling them. If needed, send me another upload link and I'll do it as individual files.

Thanks for the followup status regardless! 🙂

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

  • Staff

It's my end that's the issue. The wiring in my 160 year old cottage is abysmal so my internet is not great! It would have taken seconds at Seif HQ!

Anyway, I just ran the batch, converting to sRGB with Paralell Processing enabled and to JPEG and 487/487 files were processed without any issue. 

I did however uncheck the .afphoto option and I did this on the current macOS Photo beta. I randomly picked about 40 images and none of them had any rendering artifacts either.

Would you mind trying the beta? If that works, great—we know it will work when the beta gets pushed to live but if that fails, we need to try and do some digging. 

Screenshot 2020-07-20 14.47.31.png

 

Link to comment
Share on other sites

@Chris B Oh well, crashy crashy in 1.8.4.184. Crash report attached.

Seemingly irrelevant (but who knows at this point) details that haven't been noted previously:

  • Source media: SanDisk Extreme SSD, USB 3/C connected to OWC Thunderbolt 3 Dock (front port)
  • Destination media: MBP2018 Internal storage
  • I'm scrolling to the end of the batch item list and leaving it there, ostensibly to be able to see the progress on the last few chunks of processing -- I noticed I did it originally so have continued to do so.

In this test with the beta, 121 of 487 files "processed", quick visual shows at least one file with corruption.

Affinity Photo Beta_2020-07-21-073120_SpamBook2018.crash

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

Just now, MEB said:

Hi Brad,
Although unlikely, have you tried to copy the files from the SSD to the system drive and perform the batch job just to check if you also get errors?

 

Hi @MEB, I haven't. There are many, many permutations I have not tried. ;-) 

I'll give it a shot though.

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

@MEB @Chris B And while that additional test is running, I checked the console logs. In this pass, nothing identifiable is coming up (no read errors, no device timeouts, etc) with respect to the source device.

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

@MEB @Chris B 172 of 487 before the crash, source and destination both internal. Definite image corruption involved. ~600GB free. Report attached.

Affinity Photo Beta_2020-07-21-083715_SpamBook2018.crash

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

  • Staff

Hi Brad

All the crashes are happening during noise reduction, so could you try turning off noise reduction in Develop Assistant (reached via the icon that looks like a butler's tuxedo towards the right of the toolbar), run the batch again and see if it still crashes.

Thanks

Justin

Link to comment
Share on other sites

13 minutes ago, Justin said:

Hi Brad

All the crashes are happening during noise reduction, so could you try turning off noise reduction in Develop Assistant (reached via the icon that looks like a butler's tuxedo towards the right of the toolbar), run the batch again and see if it still crashes.

Thanks

Justin

Hi @Justin, production, beta, or does it matter? Also, it's currently set to "color reduction" -- you want it set to "Take no action"?

On a side note, has that setting always been there? Thanks for pointing it out -- it looks like I have more configuration to do to turn things off.

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

1 hour ago, Justin said:

Hi Brad

All the crashes are happening during noise reduction, so could you try turning off noise reduction in Develop Assistant (reached via the icon that looks like a butler's tuxedo towards the right of the toolbar), run the batch again and see if it still crashes.

Thanks

Justin

@MEB @Chris B 

Results:

  • External source, internal destination
  • Environment Change as requested: Noise reduction = "Take No Action"
  • Beta 1.8.4.184: Crash = NO, Result image corruption = YES
  • Production 1.8.3: Crash = NO, Result image corruption = YES

I have to say, this surprised me. I had pretty much assumed the corruption was a function of the crash during parallel processing but now it looks like they are separate issues.

Screen Shot 2020-07-21 at 11.24.04 AM.png

https://bmb.photos | Focus: The unexpected, the abstract, the extreme on screen, paper, & other physical outputTools: macOS (Primary: Ventura, MBP2018), Canon (Primary: 5D3), iPhone (Primary: 14PM), Nikon Film Scanners, Epson Printers

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

Terms of Use | Privacy Policy | Guidelines | We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.