Processing Large Stacks is Veeeeeery Slow ...

[Didge]

HI,

I regularly capture sets of many hundreds or even thousands of images during an astrophotography session.

I'm trying to process a stack of 698 images, which I converted from FITs to (uncompressed) TIFF (because AP only imports FITS as grayscale for a'normal' stack).

I started the import of these TIFFs just over 90 minutes ago - and AP is still going.

There is no progress indicator, and nothing is displayed on the canvas. AP responds to clicks on the menu bar etc - so it has not frozen - but otherwise, nothing appears to be happening.

According to my CPU/RAM monitor, AP is very busy processing something - but there is no indication of what it's doing, or how long I should expect this process to take.

How 'linear' is the time taken to process a stack ?

In other words, would a stack of 300 images take twice as long as processing 150 images ?

Based on experience, stacks of 100 or so images don't take too long, but I suspect the processing time is *not* linear - but exponential.

BTW - I'm importing these images as 'Scaled' rather than 'Perspective', and without 'Live Alignment', on the assumption that perspective is a more complex process and would probably take longer.

Also, in the Help document, in the section on Image Stacks, it says :-

Check Live Alignment to allow the above perspective adjustment to be made non-destructively (this may affect performance depending on size and number of images to be stacked).

I assume this means :-

1. the 'Live Alignment' function only applies to perspective alignment; and

2. Using 'Live Alignment' takes longer ?

Any advice here would be appreciated

Gary

v1.10.0 250

=====

PS : 100 minutes now - still no progress - CPU usage has fallen to < 4%, but AP seems to think its very busy ... at least according to this info on the main screen ...

No document loaded

Memory Pressure : 1050%

Memory efficiency : 1180%

PPS : OK - After an hour and fifty minutes, AP finally displayed the stack - not terribly well aligned unfortunately.

These are images of the recent lunar eclipse ...

Images where the moon was > 50% (001-359) have aligned very nicely, the remaining images (360-698) not so well.

They havent aligned with the first 359 images, or with each other.

Edited July 30, 2021 by Gary.JONES

[Chris B]

Hey Gary.JONES,

The app is probably loading the images into memory, hitting swap and slowing right down. What's the end goal here? Is there a reason you are not using the Astro Stack feature?

How much RAM do you have? If you have a considerable amount, say 32GB you might get away with disabling hardware acceleration which should prevent the app from using VRAM and it will solely rely on the RAM alone.

I will need to check if the processing is linear or expoential - I have no idea currently. 

 

[Didge]

Hi Chris,

Thanks for your quick reply - yes - that was my guess too. I was monitoring page swaps - it was pretty high.

My Mac has 48GB RAM - I've allocated as much to AP as I can (well, all of it really), and quit other apps while processing images.

My OS and apps live on SSD, and all my storage is high-speed SSD RAID, so my system is pretty fast.

I use the Stack feature (rather than 'Astro' Stack) because I just want to align images without stacking them.

I'll try disabling hardware acceleration (rather counter-intuitive though) and see how it goes.

Thanks again for your quick reply,

Best regards,

 

Gary

=====

[Chris B]

Hi Gary,

It looks like the processing is not linear after all. 

Were you hoping to end up with an aligned, single-layered image after all this? Or were you content with the multiple layers? I'm asking becuse the developers are keen to find out what you're goal is and are protneially interested in adding some functionality. 

[Didge]

Thanks Chris,

Yes - I guessed as much - I tested 200 images and AP took a lot more than twice the time it took to process 100 images.

What I'm trying to do (bearing in mind I only bought AP yesterday) is to :-

1. start with a set of slightly mis-aligned FITs (the recent Lunar Eclipse in this case);

2. import the FITS directly as colour without having to convert them to TIFF first;

3. align and crop the images - so I end up with a set of nicely aligned but separate (unstacked) images;

3. combine the images into a movie to show the stages of the eclipse.

I am getting the result I want - ie a nice movie of the eclipse - but it is a bit clunky and slow - it could be a lot more efficient.

BTW - I/m a computer scientist / astronomer with a specialty in usability / UX - so can suggest a few ideas if they're helpful.

Gary

[Didge]

BTW - If you want to add a great feature, being able to export a set of images as a movie directly from the export persona would be very cool

[Chris B]

Thanks Gary. I have fed this back to the developers. 

[Didge]

Thanks Chris,

Here is some more info for you that might be helpful ...

I set up AP to import a set of 294 TIFFs exported from 16Mpx 48-bit FITs astro images.

It took a very long time - and there was no progress bar during the import - hence no way to know what AP was doing.

IMHO - it should display a progress bar here.

I waited an hour - then went to bed, expecting it to be ready in the morning.

But - now that Ive had my breakfast, AP is still processing - after 8 hours !!!

AP imported the images and they're listed in the Layers view, but AP still hasn't rendered any of the images.

In other words, the images are listed in the Layers list, but the canvas is blank.

See the screen shot attached :-

- there is no 'canvas' displayed

- you can see I've used the Σ method - the Live stack thumbnail is fully white, but the canvas is blank

- de-selecting all images except one still doesn't display anything

CPU usage = 10% right now - Memory pressure = 62%, Memory efficiency = 34% - so it does look like AP is still busy.

What I'm asking AB to do is not that unusual - astrophotography uses lots of large images - but I haven't managed to produce anything from this set of images.

 

Cheers,

 

Gary

AP : 1.10.0 251

OS : Mac 10.14.6

[Didge]

Hi Chris,

Some more feedback for you ...

I set up Affinity (10.1.0 250) to import and align a stack of 698 Tiffs last night.

AP got 1/2 way through the import dialog - then seemed to get stuck.

I thought AP might be busy processing something, so I just left it, expecting it to be done by morning.

Unfortunately, the progress bar had moved at all, but AP hadn't frozen - it was still using CPU and Activity Monitor indicated it was still active ...

but had to FORCE QUIT because AP didn't respond to any menu commands.

 

By the way ...

My main interest is astrophotography - can we create a sub-topic for 1.10 beta for astrophotography-related questions only ?

AP v 1/10/0 251

OS Mac 10/14/6

[Chris B]

On 8/2/2021 at 2:07 AM, Gary.JONES said:

My main interest is astrophotography - can we create a sub-topic for 1.10 beta for astrophotography-related questions only ?

 

I think your best bet is just to create a thread in Feedback or Questions and let the comments snowball otherwise we'd end up with hundreds of sub-topics for everyone's interests. 

[Didge]

Thanks Chris,

Actually, I really think an Astrophotography sub-topic would be helpful - simply because its one of the main sub-features of Affinity Photo.

IMHO - developers would get a lot of valuable feedback and it creates focal point for thoughts on this aspect of Affinity Photo's capabilities .. that is the main reason I bought it after all

Gary