Affinity Designer file is corrupted suddenly

[Pšenda]

8 minutes ago, Andy05 said:

This could actually become some serious issue, if Affinity sells his software "for professionals"

I share the same opinion.

 

The fact that Serif does not take a "responsible" approach to this and has not been dealing with it for years is really bad - this should be a top priority.

[Grealake Mack]

thanks so much @carl123 and @Pšenda i will save locally now and watch my pixel layers

[R C-R]

1 hour ago, LondonSquirrel said:

Can anyone explain why using an external drive + Affinity is bad? (Apparently).

From what the devs have said, it is because unlike most graphic apps Affinity usually does not read all of a file into memory at once, or save all of it back to disk on every save (because changes are serialized in some unspecified way).

[Andy05]

31 minutes ago, R C-R said:

From what the devs have said, it is because unlike most graphic apps Affinity usually does not read all of a file into memory at once, or save all of it back to disk on every save (because changes are serialized in some unspecified way).

That's basically it, I think. Incremental saving of files has enormous speed benefits as less data needs to be written. But on the other side, it's riskier as even the slightest "hickup" will cause trouble in contrast to a completely save/load procedure (and this method produces bigger file sizes, hence "save as" usually creates smaller files than just "save" in apps like Affinity's).

Complete/non-incremental save may also create corrupt data, but in that case the program or the OS even usually instantly notices this and asks a user to re-save or gives a warning at least. Unfortunately, if Affinity would create some security features into their save routines, the speed benefit would be gone.

But as it's now, Affinity's save procedure seems to be overly error-prone when saving to external (or net) sources. Unfortunately, lots of devices are getting smaller and smaller for more portability, so one has to rely on external drives or network drives. In particular since working with tons of images can become quite demanding in terms of drive space needed.

[v_kyr]

6 minutes ago, Andy05 said:

Incremental saving of files has enormous speed benefits as less data needs to be written. But on the other side, it's riskier as even the slightest "hickup" will cause trouble in contrast to a completely save/load procedure (and this method produces bigger file sizes, hence "save as" usually creates smaller files than just "save" in apps like Affinity's).

Depends on if files are saved incrementally into just one and the same file here (as Affinity does), or if they are spread over several contiguous / consecutive files (like major backup apps do).

14 minutes ago, Andy05 said:

Complete/non-incremental save may also create corrupt data, but in that case the program or the OS even usually instantly notices this and asks a user to re-save or gives a warning at least. Unfortunately, if Affinity would create some security features into their save routines, the speed benefit would be gone.

Probably, it's also more difficult/laborious to handle security for the later (overall incremental saving into one and the same file), since all will need to be operated in some buffered file streams.

22 minutes ago, Andy05 said:

But as it's now, Affinity's save procedure seems to be overly error-prone when saving to external (or net) sources. Unfortunately, lots of devices are getting smaller and smaller for more portability, so one has to rely on external drives or network drives. In particular since working with tons of images can become quite demanding in terms of drive space needed.

That's right, working directly on/with external drive sources here enhances the risks of data lost due the incremental single file format nature. Especially on unstable, or not very fast net connections then.

[R C-R]

5 minutes ago, LondonSquirrel said:

Accessing an external drive is really no different from an internal drive as far as the OS is concerned.

I think it is not so much local external drives that are the problem but remote/cloud/networked ones since they are potentially subject to latencies, instabilities, etc. that local drives under the direct control of the OS are not.

I suppose it is also possible that for directly connected external drives there could be similar factors involved depending on the port type or maybe file system differences but that is just a guess.

[Pšenda]

1 hour ago, LondonSquirrel said:

But why does this only seem to apply with external drives and not internal drives too?

See my previous link:

 

 

[R C-R]

14 minutes ago, LondonSquirrel said:

The actual process of writing a file to disk is the same on external or internal drives.

What if they are not the same type of drive or do not use the same type of connection? If nothing else, I would expect the write time to differ greatly for a fast internal SSD vs. a slow HDD attached to a USB port shared with other peripherals.

[v_kyr]

7 minutes ago, LondonSquirrel said:

That is entirely abstracted away for 99% of developers. If using C++, the use of iostream and fstream takes care of that. It is the job of the OS to handle how files are written. As a programmer you don't care if your files are on SD cards, internal SSDs, USB attached SATA drives, or anything else - you can use exactly the same commands to write a file. It is the job of device drivers to present the hardware to the OS. Most developers would not go near device drivers because they don't need to. ...
...

That's right for file io from the programming context here, where let's say, you write to some specific drive, which in turn before is already mounted by the OS and thus whose filesystem is presented to you for r/w access.

For other more driver programming related tasks, there are other libs which allow to communicate with USB-devices, like for example:

• libusb
• libusb API function overview
• libusb-win32
• MS WinUSB
• USB in a Nutshell (Making sense of the USB standard)

[R C-R]

1 hour ago, LondonSquirrel said:

That is entirely abstracted away for 99% of developers.

But does that mean the process carried out by the OS is always the same regardless of the interface or storage media type?

[Andy05]

6 hours ago, LondonSquirrel said:

Has it been mentioned here that users with corrupted files are just yanking out the cable? Or even removing the cable at all?

I think this is a very, very good question! Affinity files can become quite large. And it make take a while longer than a user would think for all the buffers being flushed onto the external target drive. I know a lot of people, who pull out the USB sticks immediately after copying a file, which could cause problems as the file might not have been saved completely yet. The bigger the saved file, the higher the risk of doing something like that, I suppose.

So, it really might have been the case that people are done with today's work, save, power off the drive—too early—and do something different just to find out that their file got corrupted when they try to load it next time.

[R C-R]

2 hours ago, LondonSquirrel said:

According to the various threads, Affinity does something more complex than that. I would expect them to do so. They probably do some form of serialisation and chunking to get better performance.

The developers have explained several times over the years that they use some form of serialization but not the details of how it works, which they consider a proprietary trade secret. They have hinted that the method used is unconventional, perhaps unique, but little more other than at some point a threshold is reached when the entire file is rewritten. 

[v_kyr]

3 hours ago, LondonSquirrel said:

You as a programmer would include this library and make use of its system calls. You don't need to know what it does.

Of course I need to know what it is intended to do, otherwise it's useless for me and not reusable at all, if I don't know what certain function of a whole API/framework/library do offer and are meant for in their related context. Further if something is buggy inside you too should have some understandable knowledge of what and how certain routines/algorithms there do, since otherwise you won't be able to analyze, distinguish and find any related bug for fixing at all.

[v_kyr]

1 hour ago, LondonSquirrel said:

This is wrong for anything that is closed source. You as a Windows or MacOS programmer, in more than 99% of cases, do not have access to the source code for the file system, for the drivers, for the OS. You are relying on somebody else to provide you with reliable APIs and system calls. You are relying on any return values, errors, and so on.

That's right for closed source and OS related service/APIs here, my point was here more that you still have to know in general what and why you use certain APIs then.

1 hour ago, LondonSquirrel said:

Do you as a programmer who wants to write a file to a disk want or need (or even care) about how the file gets from memory to disk, how an inode (or similar) is allocated, how the blocks on the file system are allocated, how one block is linked to the next block, how the permissions are set? ...

Yes usually you do at least for how to write files out, since there are differences in how to write a file back to disk, if in a plain serialized unbuffered manner, or via buffered streams then etc. Just look at the differences several C and C++ IO-functions offered over time here from past to yet. Though you usually don't care that much about the low level system inode/blocks handling here, as far as you don't suppose to write some disk formater and partitioning system tools.

 

1 hour ago, LondonSquirrel said:

The standard C library on the open source operating systems is very widely used. I bet that most programmers have not looked at the source code to see what it does behind the scenes. They want to write a file, not care about how that file ends up on disk. That is why these libraries exist in the first place.

Well I can't speak for all or other programmers here, but I for my part once intensively studied "Advanced Unix Programming - by Richard Stevens" in order to see how certain Unix system services and system related functions are implemented (...especially the differences between BSD and System V here) and have to be used at all. Learned a lot from that book and the Unix system docs & man pages about certain system library calls. If you are going to write some more low level tools for a specific OS you need to know a few things more about it und it's underlaying mechanisms!

1 hour ago, LondonSquirrel said:

Perhaps I was not clear in the line that you have quoted. Yes, you need to that calling function xyz() does something. You don't need to know, in most cases, how it does it. That is what I meant by 'what it does'. If you use C++ and iostream, do you in most cases need to know or care what it is actually doing? The answer is no.

I know what you mean and for general or most usual/common programming tasks that's all right, but there are also programming situations, where you have to go and dig deeper into the materia, especially if something is not behaving as it should or is expected to be, or if you encounter serious performance or stability issues within your specific programming context.

 

1 hour ago, LondonSquirrel said:

We don't know the exact circumstances of how these Affinity files are getting corrupted. It could be user error. 

I pretty much doubt that any user intentionally disconnects a USB drive or network connection here before files have been completely written. Also most direct & permanently connected USB drives should be gracefully handled by the OS (some modern USB 3.2 devices do offer for example a read of ~420 MBps and write of up to ~380 MBps speeds nowadays), even if the device before has before been send to sleep for saving overall electricity by the OS. - The overall problem is IMO more related to the specific Affinity file format and it's write handling, which is vulnerable to external connections.

[v_kyr]

42 minutes ago, LondonSquirrel said:

I have seen that happening many times. Indeed, there is something more subtle: if I eject one of my USB drives, the icon disappears from the desktop but the lights on the drive itself keep flashing for about 10 seconds. I doubt it is safe to unplug the cable until these lights have stopped. I only notice these lights because the drive is right in front of me.

Then it's for an USB device probably due to write caching ...

Quote

Write Caching is the process whereby a device does not immediately complete writing a file, but instead caches some part of it to complete at a later time. When a USB storage device is inserted into a computer, data can be both written onto it and read off it. In order to improve its system performance, a computer may utilise its fast, volatile memory (RAM) to collect write commands sent to the USB storage device and ‘cache’ them until the slower, external storage device can be written to later. Other applications are then able to run faster, without having to wait for the external storage device to process the data write-requests.

Whilst this process increases system performance, there is also an amplified risk of data loss due to system failure, loss of power or unsafe removal of the external storage device. If the external drive is removed before the device flushes the cache and completes the whole write operation, then the file may not be readable. Naturally, the bigger the file size, the longer this process takes.

Windows’ default settings actually disable ‘Write Caching’ to allow for the quick removale of external devices without needing to use the ‘Safely Remove Hardware’ icon. However, as sticklers for performance, we would advise making use of this function.

The risk of power and system failures is sufficiently small so as to justify the increased performance benefits, and properly ejecting the flash drive by selecting the appropriate eject option prior to removal should cause the cache to flush and be written to the drive successfully.

USB related write caching can be one of the problems here with huger Affinity files, as far as the device isn't used permanently connected!

[ronald618]

Oh my. I'm working on a project, working and saving to an external drive. 

As I work I save everything to a Samsung T7 drive connected to my Mac M1. All of my files rest there.

So far I've had no trouble opening files. Nothing corrupted. But am I taking a risk?

I'd appreciate any opinion on this.

 

 

 

[R C-R]

38 minutes ago, ronald618 said:

So far I've had no trouble opening files. Nothing corrupted. But am I taking a risk?

Yes, there is a risk, at least if you are opening your files directly from the external to work on them, instead of copying them to the internal to work on them, & then after you have saved & closed them, copying them back to the external (& deleting the copy on the internal if you want).

[ronald618]

Thank you R C-R

I work on my files directly from the hard drive. As I do with every other program.

My advice to Serif: quit with the slick advertising and adding new features that few people need.

Fix the bugs and quirky performance issues first..

 

 

[Rooy AD]

On 3/10/2021 at 10:10 AM, Rooy AD said:

@carl123 file is working. thank you so much

@carl123hi bro, i've problem again. my file is corrupted. can you help me
thank you so much

Zakat.afdesign

[carl123]

21 hours ago, Rooy AD said:

@carl123hi bro, i've problem again. my file is corrupted. can you help me
thank you so much

Zakat.afdesign 79.85 MB · 2 downloads

"Recovered" document is attached below

Problem was with the very bottom layer where you had a white rectangle and you had clipped an image called "Ricardo-gomez-angel...." inside it.

Deleting that rectangle (and thus the image) resolved the problem - you just need to add that image back.

Note: Your document is very large for a one page design (66MB in attached file). You have a 57MB image off the canvas not being used. I would suggest you delete that image if not needed and do a File > Save as... to bring the file size down to a more representative/manageable size

PS Not all corrupt Affinity documents are recoverable, strongly suggest you backup to different file names as your work progresses

 

 

zakat-recovered.afdesign

[Pšenda]

49 minutes ago, carl123 said:

PS Not all corrupt Affinity documents are recoverable, strongly suggest you backup to different file names as your work progresses

I suggested Serif's automatic indexing of the previous/backup file name. Maybe they implement it sometimes.

 

 

[mifeje]

Hi, this is the second week in a row that i get a corrupted file of affinity designer. I'm a professional and that is taking my time because i'm in need of making presentations twice. My old files are getting corrupted too and i don't know why. If i need to access them in the future to make som edits i wont be able. Thats too bad. The old ones are stored in a External Hdd but the last ones aren't and they are corrupted.

[emmanaty]

Hi  @carl123, Can you help me with this file? It's also corrupted.

Thank you ;D!

MapaResultados.afphoto

[carl123]

1 hour ago, emmanaty said:

Hi  @carl123, Can you help me with this file? It's also corrupted.

Thank you ;D!

MapaResultados.afphoto 92.56 MB · 0 downloads

I have no idea what the document is supposed to look like (is it a texture?) so I can't say for sure if everything has been recovered

But you will be able to open the attached

PS I'm out for the rest of the day so if not fully recovered, start a new thread, attach the document and one of the support staff should see it

 

MapaResultados-recovered2.afphoto

[Swano]

Hi ! I also have the same problem
Can you try to help me please ?

Thank you ! 

Notice anglais principal.afdesign