Strange artifacts when merging layers or exporting to jpg

[RobKlein]

Im working with a huge file, perhaps the problem, but anyway, its a graphic for a 20 foot x 20 foot trade show booth.

Its an image of water, and I'm piecing together this image at 200dpi. The file is 2-4 G, depending on the version of my save. Anyway, the file crashes Affinity photo upon save or when merging layers quite often, and sometimes when merging layers odd artifacts are produced that garble the image. These are also produced when exporting the file as a jpeg. Ive burned up so much time on this, not sure what to do. Ive also tried opening the file in Affinity Designer, and exporting and same thing occurs. please help!

Rob Klein water image with artifacts.xlsx

[Chris B]

Hey Rob, welcome to the forums,

Is there a file you can upload to this private Dropbox folder? 

https://www.dropbox.com/request/NFRzmaXDUamjLR7eyRkN

Can you provide your system specs along with how much hard drive space you have left?

[RobKlein]

Hey Chris, thanks for jumping in to help!

I'm running Windows 10 64 bit Intel Core i7-6600U CPU@2.60 Ghz on a Lenovo P40 Yoga with 16 GB DDR3 memory

the file is 2.15Gb. three layers, one is turned off, but please turn it on so you can see all 3 - this is what I am attempting to export for large scale print.

Ive tried every combination of performance video display (ntel HD520 on board vs Nvidia M500M) as well as hardwear acceleration on vs off

I have 54.5 GB free of 500GB total

I also have an external 5Tb drive, not sure if Affinity photo can use that as a swap disk if needed

I'll upload that file, takes a few minutes to open up, and to export.

its uploaded.

 

Thanks

Rob

 

 

[RobKlein]

Hey Chris,

I uploaded the file to the dropbox link you sent me. Let me know if you have any ideas on whats happening. 

The file would also sometimes create those junk colored areas at times when I was attempting to do a clone brush or an adjustment. 

[Chris B]

I have managed to reproduce the issue. I have no idea what is happening I'm afraid. I will pass this file to the developers. I will also poke it when I get chance to see if there's a workaround we can find so you can finish your project.

[RobKlein]

ok...so then its not the performance settings, or how much ram I have...Thank you for your help with this - my project is stressfully on hold, I've completed my work and now its just about exporting it.

I also tried TIFF and that didn't work. It also occurs if I resize the doc from 200dpi down to other resolutions, including 180, 175 160 and 150 dpi, and also occurs when saving it as jpg at high quality, down to 50% quality. I dont think Affinity Photo can hold all the data in memory at once, can you use a scratch disk so we can designate 5 or 10GB of space for that swapping, so that it can get through exporting the 2GB file? I believe that may work, but just a guess. normal files dont have this issue as they are much smaller, this file is only special due to its size...so its got to be something like this, trouble managing the large file size. 

I may try to cut the file in half physically so that I can downsample each half to 160dpi, and then assemble them afterward, and then export after making sure the seam is invisible. if that works, then thats another piece of evidence that AF cannot manage the large file size. 

let me know what you discover on your end - thanks!

Rob

 

[RobKlein]

i split the file in 2 halves. Kept it at 200dpi. exported the top half at jpg, and the problem persists

I split the file into 4 quarters, kept 200dpi, exported the jpg, still artifacts

I first down-sampled the file into 150 and 144 dpi, exported and still artifacts

I down-sampled the file into 150 and then split the file into quarters and still artifacts

so maybe its not about the file size...

 

[Chris B]

I managed to get a sucessful merge. I resized the document to 72 dpi with resampling on and it was spot on. Note - I did have OpenCL enabled in Preferences. If I disable it, I get the artefacts which is surprising.

Can you check to see if OpenCL is on in Preferences?

Are you working to a spec for the dpi? Where is this ultimately going? If it won't be visible up-close I'd go with 72 dpi

A colleague managed to merge at the native 200 dpi with OpenCL enabled. If you need it, we can try sending over the merged image as whatever format you want at the 200dpi.

[RobKlein]

72 dpi wont work, this needs to be 150 dpi or better, preferably 200 dpi. The client already rejected an earlier 96 dpi verson, said it was too blurry and I agree it needs to be a higher res, 150 or better. 

Did you try exporting at 150 or better with OpenCL? 

I tried multiple times with Open CL turned on and off, and got the artefacts every time, irrespective of that setting.  I was also turniing on and off View Quality bilinear vs nearest neighbor, dither gradients, and use precise clipping. 

What did your colleague do to get the 200dpi to work besides OpenCL?  what was his system spec for reference? Did he or she use bilinear vs nearest neighbor, dither gradients, precise clipping? what about the retina rendering, and renderer? 

If you can send that to me, that would be great. send me a link, thank you!

Edited June 30, 2021 by RobKlein
added some details

[RobKlein]

heres a crazy idea, what if this is about the auto file recovery period? i have mine set to 3 minutes, what if the file is attempting to save the file usign auto recovery, while it is exporting...normally the save doesnt take over 3 minutes, but in this case it does...maybe its coming across a this competing issue, and then it gets better, and then it does this a 2nd time, and maybe thats why it has 2 zones of artefacts...?im restarting and retrying with the auto recovery period set to the max timeframe to see what happens

[RobKlein]

can you send me that file from your colleague?

I have to complete my project! 

 

[RobKlein]

On 6/30/2021 at 6:20 AM, Chris B said:

I managed to get a sucessful merge. I resized the document to 72 dpi with resampling on and it was spot on. Note - I did have OpenCL enabled in Preferences. If I disable it, I get the artefacts which is surprising.

Can you check to see if OpenCL is on in Preferences?

Are you working to a spec for the dpi? Where is this ultimately going? If it won't be visible up-close I'd go with 72 dpi

A colleague managed to merge at the native 200 dpi with OpenCL enabled. If you need it, we can try sending over the merged image as whatever format you want at the 200dpi.

please send me your colleague's successful merged file at 200dpi, that would be very helpful, thank you.

[RobKlein]

please send that file!!! or the link...

JPG at highest quality 200 dpi would be great.

thank you. 

[Chris B]

Sorry for the delay - I'll get this over to you as soon as possible. I'm just waiting for them to see the request to send it over to me and then I'll upload it for you.

[Chris B]

Two of us have spent a considerable amount of time this moring trying to export this image. Unfortunately, no matter what format we try, the app will either crash or give us rendering artefacts.

If you zoom into the document and pan around at 100% (1:1) you will see artefacts. All the formats we exported had artefacts in the exact same place - something has happened to these images at some point to cause the corruption.

I'm going to test your theory about the autosave.

[RobKlein]

You mentioned your colleague managed to merge at the native 200 dpi with OpenCL enabled. If you need it, we can try sending over the merged image as whatever format you want at the 200dpi....can you send that file? 

I thought by splitting the file into halves or quarters each file would be smaller and I could export those files separately into jpgs and then as a jpg combine them, as the total file size would be significantly smaller, but still there is the problem. 

 

[RobKlein]

I have a new idea...I can work on the file, and sometimes, as I work on it, it starts to create artifacts. So then I exit, restart, and re-open and continue my work. Its more than just the export filter, as this occurs when working in the file as well. 

Perhaps there is something about the native affinity photo file format for keeping track of all pixels that works under normal circumstances, but when stressed by too many pixels, perhaps it cannot hold all of that together and it cannot keep track of everything.,..and renders it out improperly. This happens after 2 layers are merged, or when using a paintbrush, or a stamp tool, or applying an adjustment such as HSV, or exporting. In the file format, is there a limit that is set too low? sorry if this makes no sense, I used to program many years ago, and perhaps today this means nothing....

[RobKlein]

I tested something... loaded up the original file before I edited it, to see if I can convert that 300 dpi image of about 24" x less than 24"to 20' at 150 dpi, and then exported that to a jpg 95% quality before any real editing, to see if the artefacts are created. This removes the various Affinity photo operations as culprits if it works. It worked. 

so now I will place that image back over the other file, deleting the bad areas, to see If i can now export that...of course theres a lot of work remaining to rebuild the file so its visually seamless, looking like real water...

[Chris B]

On 7/2/2021 at 4:14 PM, RobKlein said:

You mentioned your colleague managed to merge at the native 200 dpi with OpenCL enabled. If you need it, we can try sending over the merged image as whatever format you want at the 200dpi....can you send that file? 

The merge was successful, however on export we saw artefacts. Also, viewing the document at 100% after merge and before export, there were artefacts. For some reason, I had not thought to zoom to 100% for the 1:1 pixel ratio to see what was going wrong. I guess because the file was so huge. The only clue I had was that no matter what version of the app or operating system we used, the corruption was visible in the exact same places—the corruption was not even at pixel-level but instead covered a group of pixels.

On 7/2/2021 at 4:20 PM, RobKlein said:

Perhaps there is something about the native affinity photo file format for keeping track of all pixels that works under normal circumstances, but when stressed by too many pixels, perhaps it cannot hold all of that together and it cannot keep track of everything.,..and renders it out improperly.

Possibly. We thought they were NaN pixels at first but seeing as we could colour-pick them, that bebunked that theory. I suspect it's potentially a resource issue as well. I tried on my capable PC with an i9, RTX 3080, 32GB RAM and 500GB of free SSD storage and it still failed and was brought to its knees. 

On 7/2/2021 at 8:47 PM, RobKlein said:

so now I will place that image back over the other file, deleting the bad areas, to see If i can now export that...of course theres a lot of work remaining to rebuild the file so its visually seamless, looking like real water...

I am curious to see how you got on. Just as an FYI, I tried editing an image in Photoshop at 24ft x 24ft and 200dpi and the app was consistently crashing (this was on my iMAc). Affinity Photo managed to open it but I could not reproduce the artefacts in either app. If I recall, we are well within the 64,000 px x 64,000 px limit of a JPEG for export so I think Affinity is to blame for the corruption somewhere. I would love to be able to reproduce it from scratch. 

Now I have some extra info and I've exhausted everything I can do, I will ask the developers to see if they can identify what went wrong.

[RobKlein]

So I rebuilt the file...

I went back to the original photo that was about about 2feet x 1.3 feet at 300 dpi and converted it to a 4' wide file at 150 dpi, and then created a seamless repeat on all edges. Lots of work, but by using a repeat, (Distort > Affine) I was able to create a nice seamless pattern without a huge file size. I felt the only way was to keep the total image size much smaller, and since les than 150dpi or so would not be great, it had to be a repeat. In the end it came out pretty good I think. Then the last step was to save the repeat at 20 feet x 20 feet...thats 36,000 x 36,000 pixels, for a trade show booth, and this time the file size ballooned to a mere 754 MB. Huge, but nothing like the 2GB plus file sizes I was getting before. No artifacts this time! Just in case you are curious, I attached a reduced file here, this one is only 3000 x 3000 pixels, or 10" x 10" at 300 dpi

thanks for your help, and feel free to send me an email once you feel the developers have figured out what happened, or if this is a bug, once its been fixed, I appreciate that!

robertklein.biz[at]gmail.com

and feel free to pass on my name and email for anyone needing to build a seamless water pattern or other pattern

all the best, Rob 

[Chris B]

Hi Rob,

Thanks for the update. That looks absolutely fantastic! Well done.

I'd love to know what caused the corruption so hopefully we will find out.

I just edited the email address so you don't get crawled by spambots but it's fine to leave it here. 

[RobKlein]

Thanks Chris! I appreciate your help. 

Now I know when the file needs to be very large, into the multiple Gigabytes, strongly consider a repeat. 

When I think back to what happened, Its possible I did an adjustment (HSV or brightness contrast, or other adjustment) and then merged that with the pixel layer, and later when copying that area to extend the pattern, that may have caused the problem. sort of a remnant of a merged adjustment. who knows...

all the best, Rob