Batch Job Skips Files

[LinkLayer]

I have 97 JPEG source files. I ran batch jobs to apply crops to these and save the outputs as JPEG to another folder. For three runs of this, I had the following results:

1. All 97 files processed okay
2. 1 file was skipped
3. 3 files were skipped

By skipped, I mean that the affected file simply did not appear in the output folder. The files skipped appeared random. They were not at the beginning or at the end of the list. There were no error messages. I manually redid the skipped files without problems. The input files all had identical dimensions of 600 x 1800 pixels. The batch macros consisted of one crop only -- no other changes.

MacOS 10.13.6;  iMac (21.5-inch, Mid 2011) 2.7 GHz Intel Core i5;  12 GB RAM; Affinity Photo 1.6.7

I would be willing to provide info or test cases to reproduce this.

[Dan C]

Hi LinkLayer

I'm sorry to hear this, it certainly isn't something I've heard of before.

Could you please upload some sample JPEGs (10-15 would be perfect) and the macro that you're running to the following link? This will allow me to look into this for you!

https://www.dropbox.com/request/2EBRoejpuetmfTgT7rgs

[LinkLayer]

I uploaded some files from that batch. I don't know how to upload a macro. [edit, I think I figured it out and added the macro to the dropbox link.]

Also, I'm sorry if I reported this to the wrong forum.

Thanks!

[Dan C]

Thanks for that! Did you have 'Parallel Processing' ticked in the Batch Processing panel before starting the job? We're aware of a current issue with batch processing on the Mac when Parallel Processing is selected, and this may be related.

I've run the batch job 5 times on my Mac (with batch processing switched off) and I've not yet had any files skipped. Could you please try switching this option off and running the batch into a new folder? Does it miss any this time?

[LinkLayer]

Yes. 'Parallel Processing' was on.  (It was on by default.)

I had some trouble reproducing the problem myself (with 'Parallel Processing' still on), however, I did get it to miss one file (out of 97) on one run.

I think originally, I had created the output folder beforehand in the Finder, and had that finder window open when running the batch job in Affinity Photo. When I finally reproduced the problem, I did it that way again, and that finder window was open.

I then tried to reproduce the problem with 'Parallel Processing' turned off. I ran the batch job five times (97 files each time), and the problem did not recur. That doesn't seem conclusive to me, considering it didn't happen all the time anyway.

I noticed something else weird. I expected to see the open folder window get populated with the output files more or less in real time, and I did see that sometimes. However, sometimes it took quite a long time before any of the output files became visible.  (measured at about one minute in one case)

A user interface nit: It would give me a warmer feeling if the batch job dialog box reported the number of files to be processed as you add them. The file count is displayed when you actually start the batch job, but at that point it is too late if you see it is not the right number.

 

[LinkLayer]

I duplicated my original files until I had 776 input files.

With that, I had no trouble reproducing the issue.

Parallel Processing on:  776 input;  286 output; 490 missing

Parallel Processing off: 776 input; 295 output; 481 missing

 

[R C-R]

5 hours ago, LinkLayer said:

I noticed something else weird. I expected to see the open folder window get populated with the output files more or less in real time, and I did see that sometimes. However, sometimes it took quite a long time before any of the output files became visible.  (measured at about one minute in one case)

Of no help resolving your issue but just FYI, Finder is often 'lazy' about updating folder contents -- technically, the OS assigns a low priority to this, so when there are other tasks running with higher priorities, it can take quite a long time for Finder views to catch up to changes. In fact, it may be necessary to close the window & open a new one to the same folder, or to switch to another folder & back again, to force Finder to update the folder view.

Supposedly, Mojave's Finder is better about this than the last few macOS releases but I have not yet upgraded to Mojave so I can't comment on that.

[Dan C]

Thanks for trying this, it's more than appreciated.
Below are my results from running batch processes using your files across different operating systems, at all times having Finder open where the folder was created:

 Operating System ¦ Input ¦ Output ¦ Missing ¦ Parallel Processing?
------------------------------------------------------------------
10.13 High Sierra ¦  601  ¦   599  ¦    2    ¦ On
10.13 High Sierra ¦  601  ¦   598  ¦    3    ¦ Off
10.14 Mojave      ¦  768  ¦    0   ¦   768   ¦ On (Crashes)
10.14 Mojave      ¦  768  ¦   768  ¦    0    ¦ Off

Unfortunately I can't replicate missing files to the same degree as you're seeing currently. 

I'd recommend resetting your app to it's default settings as this may help. You can reset your app back to its default settings by holding down the CTRL key while launching the app. When the Clear User Data window appears the first three checkboxes should be ticked, then select Clear. The app will now open.

Be aware that resetting your app back to its default settings will lose any custom settings such as keyboard shortcuts and brushes etc. To backup such settings before resetting the app, please watch our Affinity - Exporting User Settings and Keyboard Shortcuts video.

14 hours ago, LinkLayer said:

A user interface nit: It would give me a warmer feeling if the batch job dialog box reported the number of files to be processed as you add them. The file count is displayed when you actually start the batch job, but at that point it is too late if you see it is not the right number.

I agree, I'll log this as an improvement with our devs

[Dan C]

Another thought, are you using the Mac App Store or the Affinity.store build of Affinity Photo please?

[LinkLayer]

I bought Affinity Photo on the Mac App Store.

I'm happy that you were able to reproduce this.  That way I don't look like a crazy guy. When I found this problem, I think it was only about the second time I had ever used the batch file feature. I'm surprised if I am the first to report this problem.

This is not impacting my work or anything. It'll probably be fixed before I need to use it again.

I'm happy with the speed! The batch processing runs plenty fast, even on my older Mac.  What's not so fast is adding files to the list to be processed. That takes much longer than to actually process them.

Thanks!

[v_kyr]

12 hours ago, R C-R said:

... Finder is often 'lazy' about updating folder contents -- technically, the OS assigns a low priority to this, so when there are other tasks running with higher priorities, it can take quite a long time for Finder views to catch up to changes. ...

In such cases just use a shell/terminal window on the output folder and perform repeatedly the "ls" command to list the files. In order to count the files in the output folder use

"ls | wc -l"

etc.

[R C-R]

41 minutes ago, v_kyr said:

In such cases just use a shell/terminal window on the output folder and perform repeatedly the "ls" command to list the files.

If nothing else, repeatedly doing that would use cpu time & memory that, even though it isn't a lot of either one, is something I would prefer to avoid.

[v_kyr]

1 minute ago, R C-R said:

If nothing else, repeatedly doing that would use cpu time & memory that, even though it isn't a lot of either one, is something I would prefer to avoid.

The finder process often uses even more cpu time & memory repeatedly to update, as do most UI and event based processes, so this is then an alternative less resource hungry way in order to still get an overview.

[R C-R]

4 hours ago, v_kyr said:

The finder process often uses even more cpu time & memory repeatedly to update ...

But not when it is being 'lazy' -- which is the point of assigning it a low priority. 

4 hours ago, v_kyr said:

... so this is then an alternative less resource hungry way in order to still get an overview.

Maybe it is less resource hungry for the computer, but not for me, since I have to repeatedly press a minimum of two keys to run the command every time I want to update the count during what could be a very long-running batch job, & then I would still have to refresh Finder or another terminal command to see which files were missed.

[v_kyr]

18 hours ago, R C-R said:

But not when it is being 'lazy' -- which is the point of assigning it a low priority. 

Maybe it is less resource hungry for the computer, but not for me, since I have to repeatedly press a minimum of two keys to run the command every time I want to update the count during what could be a very long-running batch job, & then I would still have to refresh Finder or another terminal command to see which files were missed.

Nope, just use ...

$ watch ls -ltr

...or via some simple shell script...

while [ 1 ]
do
  clear
  ls -lrt
  sleep 60
done

...so you can automate that boring manual keys task and repeatedly display new files that are written to the directory, sort of poor mans auto-refresh.

[R C-R]

4 hours ago, v_kyr said:

Nope, just use ...

$ watch ls -ltr

-bash: watch: command not found

Besides, all I really care about is the end result -- which (if any) files are skipped, with or without parallel processing. The order they are written to the drive is not important.

[v_kyr]

"watch" is generally found on Linux systems, but can also be added and used on MacOS. - It's easy to add via Homebrew:

brew install watch

or via ports:

$ sudo port install watch

etc.

 

20 hours ago, R C-R said:

Besides, all I really care about is the end result -- which (if any) files are skipped, with or without parallel processing.

Inspecting such things might help to find out what is skipped and if there is possibly a reproducible pattern for these omissions.

[R C-R]

3 hours ago, v_kyr said:

Inspecting such things might help to find out what is skipped and if there is possibly a reproducible pattern for these omissions.

Sure, but that can be done with Finder a few minutes after the batch job completes, when there won't be any uncertainty about which files are still being processed. There is no need to add any commands to the CLI or run Terminal to do that.

[v_kyr]

19 minutes ago, R C-R said:

Sure, but that can be done with Finder a few minutes after the batch job completes, when there won't be any uncertainty about which files are still being processed. There is no need to add any commands to the CLI or run Terminal to do that.

If you look back at your own initial comment here ...

Quote

... Finder is often 'lazy' about updating folder contents -- technically, the OS assigns a low priority to this, so when there are other tasks running with higher priorities, it can take quite a long time for Finder views to catch up to changes. ...

... you should usually understand that I just showed another alternative handling way for such cases when the GUI doesn't update immediately or reacts in time. If you aren't a terminal guy then you don't have to try it that way and so just wait for the finder to complete updating, it's that easy. - However we can probably stop any further fruitless debating and wasting time about folder GUI/CLI updating themes, since people as the OP might be more interested at all in a fixed well behaving solution instead.

[R C-R]

59 minutes ago, v_kyr said:

... you should usually understand that I just showed another alternative handling way for such cases when the GUI doesn't update immediately or reacts in time.

Note that I also mentioned two ways to cause Finder to update its views, both of which are very quick & easy to do, & don't require any additions to any part of the OS or using Terminal. I have no problem using Terminal when it does something I can't just as easily do from the GUI; I just don't think it is necessary for this, & is a needless complication many users would prefer to avoid.

As for fixes, that is something Serif/Affinity must do. The issue & how to trigger it has already been identified, so there is not much more users can contribute to that.

[v_kyr]

@R C-R Don't know if there is some reproducible pattern or if it is always random, meaning the file omissions from supplied lists during tryouts here. Or then if supplied files are always processed in an ordered sequence, or not, if it depends on the amount of files and sizes given, or then only if macros are applied etc.