Jump to content

Affinity Designer file is corrupted suddenly


Recommended Posts

8 minutes ago, Andy05 said:

This could actually become some serious issue, if Affinity sells his software "for professionals"

I share the same opinion.

 

The fact that Serif does not take a "responsible" approach to this and has not been dealing with it for years is really bad - this should be a top priority.

Affinity Store: Affinity Suite (ADe, APh, APu) 1.10.5.1342.
Windows 10 Pro, Version 21H1, Build 19043.1586.
Dell Latitude E5570, i5-6440HQ 2.60 GHz, 8 GB, Intel HD Graphics 530, 1920 x 1080.
Dell OptiPlex 7060, i5-8500 3.00 GHz, 16 GB, Intel UHD Graphics 630, Dell P2417H 1920 x 1080.
Intel NUC5PGYH, Pentium N3700 2.40 GHz, 8 GB, Intel HD Graphics, EIZO EV2456 1920 x 1200.

Link to comment
Share on other sites

1 hour ago, LondonSquirrel said:

Can anyone explain why using an external drive + Affinity is bad? (Apparently).

From what the devs have said, it is because unlike most graphic apps Affinity usually does not read all of a file into memory at once, or save all of it back to disk on every save (because changes are serialized in some unspecified way).

Affinity Photo 1.10.5, Affinity Designer 1.10.5, Affinity Publisher 1.10.5;  2020 iMac 27"; 3.8GHz i7, Radeon Pro 5700, 32GB RAM; macOS 10.15.7
Affinity Photo 
1.10.5.280 & Affinity Designer 1.10.5 for iPad; 6th Generation iPad 32 GB; Apple Pencil; iPadOS 15.0.2

Link to comment
Share on other sites

31 minutes ago, R C-R said:

From what the devs have said, it is because unlike most graphic apps Affinity usually does not read all of a file into memory at once, or save all of it back to disk on every save (because changes are serialized in some unspecified way).

That's basically it, I think. Incremental saving of files has enormous speed benefits as less data needs to be written. But on the other side, it's riskier as even the slightest "hickup" will cause trouble in contrast to a completely save/load procedure (and this method produces bigger file sizes, hence "save as" usually creates smaller files than just "save" in apps like Affinity's).

Complete/non-incremental save may also create corrupt data, but in that case the program or the OS even usually instantly notices this and asks a user to re-save or gives a warning at least. Unfortunately, if Affinity would create some security features into their save routines, the speed benefit would be gone.

But as it's now, Affinity's save procedure seems to be overly error-prone when saving to external (or net) sources. Unfortunately, lots of devices are getting smaller and smaller for more portability, so one has to rely on external drives or network drives. In particular since working with tons of images can become quite demanding in terms of drive space needed.

»A designer's job is to improve the general quality of life. In fact, it's the only reason for our existence.«
Paul Rand (1914-1996)

Link to comment
Share on other sites

6 minutes ago, Andy05 said:

Incremental saving of files has enormous speed benefits as less data needs to be written. But on the other side, it's riskier as even the slightest "hickup" will cause trouble in contrast to a completely save/load procedure (and this method produces bigger file sizes, hence "save as" usually creates smaller files than just "save" in apps like Affinity's).

Depends on if files are saved incrementally into just one and the same file here (as Affinity does), or if they are spread over several contiguous / consecutive files (like major backup apps do).

14 minutes ago, Andy05 said:

Complete/non-incremental save may also create corrupt data, but in that case the program or the OS even usually instantly notices this and asks a user to re-save or gives a warning at least. Unfortunately, if Affinity would create some security features into their save routines, the speed benefit would be gone.

Probably, it's also more difficult/laborious to handle security for the later (overall incremental saving into one and the same file), since all will need to be operated in some buffered file streams.

22 minutes ago, Andy05 said:

But as it's now, Affinity's save procedure seems to be overly error-prone when saving to external (or net) sources. Unfortunately, lots of devices are getting smaller and smaller for more portability, so one has to rely on external drives or network drives. In particular since working with tons of images can become quite demanding in terms of drive space needed.

That's right, working directly on/with external drive sources here enhances the risks of data lost due the incremental single file format nature. Especially on unstable, or not very fast net connections then.

☛ Affinity Designer 1.10.5 ◆ Affinity Photo 1.10.5 ◆ OSX El Capitan

Link to comment
Share on other sites

Many thanks for the info, R C-R, Andy05, and v_kyr.

It would seem then that the design of the file format + the saving method are not very robust. It's some form of file chunking.

But why does this only seem to apply with external drives and not internal drives too? All disk access is handled by the operating system (whether MacOS, Windows, UNIX). OK, you can write directly to the disk if you know how and if you jump through some hoops. It's useful for some specific cases, but I don't see it being useful here. Accessing an external drive is really no different from an internal drive as far as the OS is concerned.

Link to comment
Share on other sites

5 minutes ago, LondonSquirrel said:

Accessing an external drive is really no different from an internal drive as far as the OS is concerned.

I think it is not so much local external drives that are the problem but remote/cloud/networked ones since they are potentially subject to latencies, instabilities, etc. that local drives under the direct control of the OS are not.

I suppose it is also possible that for directly connected external drives there could be similar factors involved depending on the port type or maybe file system differences but that is just a guess.

Affinity Photo 1.10.5, Affinity Designer 1.10.5, Affinity Publisher 1.10.5;  2020 iMac 27"; 3.8GHz i7, Radeon Pro 5700, 32GB RAM; macOS 10.15.7
Affinity Photo 
1.10.5.280 & Affinity Designer 1.10.5 for iPad; 6th Generation iPad 32 GB; Apple Pencil; iPadOS 15.0.2

Link to comment
Share on other sites

1 hour ago, LondonSquirrel said:

But why does this only seem to apply with external drives and not internal drives too?

See my previous link:

 

 

Affinity Store: Affinity Suite (ADe, APh, APu) 1.10.5.1342.
Windows 10 Pro, Version 21H1, Build 19043.1586.
Dell Latitude E5570, i5-6440HQ 2.60 GHz, 8 GB, Intel HD Graphics 530, 1920 x 1080.
Dell OptiPlex 7060, i5-8500 3.00 GHz, 16 GB, Intel UHD Graphics 630, Dell P2417H 1920 x 1080.
Intel NUC5PGYH, Pentium N3700 2.40 GHz, 8 GB, Intel HD Graphics, EIZO EV2456 1920 x 1200.

Link to comment
Share on other sites

50 minutes ago, Pšenda said:

See my previous link:

Sure, I know about buffered IO on storage. But buffered IO also occurs on local disks. The actual process of writing a file to disk is the same on external or internal drives. It is not specific solely to USB drives. Has it been mentioned here that users with corrupted files are just yanking out the cable? Or even removing the cable at all?

If, as mentioned in your link, USB drives are not unmounted cleanly then I expect there will be problems.

I'm following this post as i have long had an interest in files, file systems, IO, etc. 

Edited by LondonSquirrel
reason for following post.
Link to comment
Share on other sites

14 minutes ago, LondonSquirrel said:

The actual process of writing a file to disk is the same on external or internal drives.

What if they are not the same type of drive or do not use the same type of connection? If nothing else, I would expect the write time to differ greatly for a fast internal SSD vs. a slow HDD attached to a USB port shared with other peripherals.

Affinity Photo 1.10.5, Affinity Designer 1.10.5, Affinity Publisher 1.10.5;  2020 iMac 27"; 3.8GHz i7, Radeon Pro 5700, 32GB RAM; macOS 10.15.7
Affinity Photo 
1.10.5.280 & Affinity Designer 1.10.5 for iPad; 6th Generation iPad 32 GB; Apple Pencil; iPadOS 15.0.2

Link to comment
Share on other sites

48 minutes ago, R C-R said:

What if they are not the same type of drive or do not use the same type of connection?

That is entirely abstracted away for 99% of developers. If using C++, the use of iostream and fstream takes care of that. It is the job of the OS to handle how files are written. As a programmer you don't care if your files are on SD cards, internal SSDs, USB attached SATA drives, or anything else - you can use exactly the same commands to write a file. It is the job of device drivers to present the hardware to the OS. Most developers would not go near device drivers because they don't need to. 

If you copy a file to an exFAT formatted SD card, to an USB-attached HFS+ formatted hard drive, and to an internal APFS formatted SSD, you as a user do not do anything different. But you, the user, have just executed a program to do it. All of the different low level commands for writing the blocks are handled by the OS and the drivers, and the firmware on the drives themselves. Do you, as a user, worry that your SSD is 20 times faster than your SD card? No, you just accept it. Sure, the time it takes to write a file varies but there is nothing a programmer can do to speed up physics. They can, if they really wanted to, slow down writes. But that wouldn't really gain anything.

The means of checking that data really is written to storage (whether internal, external, remote) varies and can be problematic. Indeed, in your linked post the problem that the OS reports a file being written, when it is still in a buffer on a hard disk (it really does not matter if it is externally attached or internal), is a serious one. 

Link to comment
Share on other sites

7 minutes ago, LondonSquirrel said:

That is entirely abstracted away for 99% of developers. If using C++, the use of iostream and fstream takes care of that. It is the job of the OS to handle how files are written. As a programmer you don't care if your files are on SD cards, internal SSDs, USB attached SATA drives, or anything else - you can use exactly the same commands to write a file. It is the job of device drivers to present the hardware to the OS. Most developers would not go near device drivers because they don't need to. ...
...

That's right for file io from the programming context here, where let's say, you write to some specific drive, which in turn before is already mounted by the OS and thus whose filesystem is presented to you for r/w access.

For other more driver programming related tasks, there are other libs which allow to communicate with USB-devices, like for example:

☛ Affinity Designer 1.10.5 ◆ Affinity Photo 1.10.5 ◆ OSX El Capitan

Link to comment
Share on other sites

1 hour ago, LondonSquirrel said:

That is entirely abstracted away for 99% of developers.

But does that mean the process carried out by the OS is always the same regardless of the interface or storage media type?

Affinity Photo 1.10.5, Affinity Designer 1.10.5, Affinity Publisher 1.10.5;  2020 iMac 27"; 3.8GHz i7, Radeon Pro 5700, 32GB RAM; macOS 10.15.7
Affinity Photo 
1.10.5.280 & Affinity Designer 1.10.5 for iPad; 6th Generation iPad 32 GB; Apple Pencil; iPadOS 15.0.2

Link to comment
Share on other sites

6 hours ago, LondonSquirrel said:

Has it been mentioned here that users with corrupted files are just yanking out the cable? Or even removing the cable at all?

I think this is a very, very good question! Affinity files can become quite large. And it make take a while longer than a user would think for all the buffers being flushed onto the external target drive. I know a lot of people, who pull out the USB sticks immediately after copying a file, which could cause problems as the file might not have been saved completely yet. The bigger the saved file, the higher the risk of doing something like that, I suppose.

So, it really might have been the case that people are done with today's work, save, power off the drive—too early—and do something different just to find out that their file got corrupted when they try to load it next time.

»A designer's job is to improve the general quality of life. In fact, it's the only reason for our existence.«
Paul Rand (1914-1996)

Link to comment
Share on other sites

10 hours ago, R C-R said:

But does that mean the process carried out by the OS is always the same regardless of the interface or storage media type?

To 99% of developers it does not matter. Unless you are writing programs for manipulating file systems, checking file systems, backup tools for file systems, and so on, it is unimportant to you. It is squarely in the field of OS and device driver programmers, not desktop GUI application developers. You don't even need to know. That is the whole purpose of having APIs and system calls for writing files.

By way of an example, look at the simple 'cp' command found on MacOS. The MacOS UNIX tools are derived from the *BSD world. I've looked at the source code for 'cp' on a FreeBSD machine. It has no special hardware-specific/filesystem-specific code. It used stdio.h You can see this if you check which libraries it links to:

jre@bsd12a:~ % ldd /bin/cp
/bin/cp:
libc.so.7 => /lib/libc.so.7 (0x800250000)

It uses the standard C library, nothing else. This is akin to iostream and fstream in C++. I am saying that the process of writing files (which cp inevitably does) by itself has no special hooks built-in for different types of storage or file systems. It doesn't need to. How the files are written to disk is handled by the OS and the drivers. The actual process will certainly differ by hardware type and file system type. But you, as an end user, and 99% of developers do not care about it.

'cp' is a blunt instrument. It copies a file and writes it somewhere else. The whole file, every byte. According to the various threads, Affinity does something more complex than that. I would expect them to do so. They probably do some form of serialisation and chunking to get better performance. Here's why: On small files, let's say 20MB, writing the whole file would be trivial and quick. It would be complete before you carried on with your drawing. If you have a larger file, let's say 500MB, that process could take several seconds on a SATA disk (there are plenty of these still around). If you have a 1GB file, you could be there waiting a long time before you can resume your drawing.

Let's go a bit further. Let's say you have a 500MB drawing and you add one straight line somewhere (a tiny change). You save it. Should the whole file, 500MB of it, be written again from scratch? Or should it take into account this minor change and write only this part? I would guess that Affinity have taken this sort of thing into account and for performance purposes write only that change to the file. It is certainly a valid way of doing things, but it does need to be done carefully with lots of checking.

Checking that files have been correctly written is definitely in the province of every programmer. It is true that you can get a system call return from the OS that says more or less 'there was no error', but the file has not yet been written to storage safely. There are ways round this. In particular, databases which are ACID compliant do this as a matter of course. Simply calling fsync() does not guarantee that a file is definitely written to disk. Look at this snippet from man page for fsync on a Mac:

     Note that while fsync() will flush all data from the host to the drive (i.e. the "permanent storage device"), the drive
     itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order
     sequence.

Affinity's programmers will know all about this.

I could go on about this but it won't help. Suffice to say that anyone doing anything 'clever' with IO needs to be clever with their error checking. 

Link to comment
Share on other sites

12 hours ago, v_kyr said:

That's right for file io from the programming context here, where let's say, you write to some specific drive, which in turn before is already mounted by the OS and thus whose filesystem is presented to you for r/w access.

For other more driver programming related tasks, there are other libs which allow to communicate with USB-devices, like for example:

I am aware of the libusb in the UNIX world. Again, somebody has abstracted the task of writing files to specific devices for you. You as a programmer would include this library and make use of its system calls. You don't need to know what it does.

Link to comment
Share on other sites

7 hours ago, Andy05 said:

pull out the USB sticks immediately after copying a file, which could cause problems as the file might not have been saved completely yet.

There are lots of things to consider with external drives. Take this scenario:

  1. You are in a virtual coffee shop (because everywhere is closed) and nobody can see you sipping your latte.
  2. You are working on your laptop with an external hard drive. Your external hard drive has the file you are working on.
  3. You have finished this piece of work and hit the save button.
  4. You immediately shut the lid and your laptop goes to sleep.
  5. You pull out the cable of your external hard drive.

I would be willing to bet a large sum of money that your file has not been safely written to disk.

The thing is, we don't know the exact circumstances in which these files have become corrupted. As said, I'm just here as I have long had an interest in file systems in general.

Link to comment
Share on other sites

2 hours ago, LondonSquirrel said:

According to the various threads, Affinity does something more complex than that. I would expect them to do so. They probably do some form of serialisation and chunking to get better performance.

The developers have explained several times over the years that they use some form of serialization but not the details of how it works, which they consider a proprietary trade secret. They have hinted that the method used is unconventional, perhaps unique, but little more other than at some point a threshold is reached when the entire file is rewritten. 

Affinity Photo 1.10.5, Affinity Designer 1.10.5, Affinity Publisher 1.10.5;  2020 iMac 27"; 3.8GHz i7, Radeon Pro 5700, 32GB RAM; macOS 10.15.7
Affinity Photo 
1.10.5.280 & Affinity Designer 1.10.5 for iPad; 6th Generation iPad 32 GB; Apple Pencil; iPadOS 15.0.2

Link to comment
Share on other sites

3 hours ago, LondonSquirrel said:

You as a programmer would include this library and make use of its system calls. You don't need to know what it does.

Of course I need to know what it is intended to do, otherwise it's useless for me and not reusable at all, if I don't know what certain function of a whole API/framework/library do offer and are meant for in their related context. Further if something is buggy inside you too should have some understandable knowledge of what and how certain routines/algorithms there do, since otherwise you won't be able to analyze, distinguish and find any related bug for fixing at all.

☛ Affinity Designer 1.10.5 ◆ Affinity Photo 1.10.5 ◆ OSX El Capitan

Link to comment
Share on other sites

35 minutes ago, v_kyr said:

if something is buggy inside you too should have some understandable knowledge of what and how certain routines/algorithms there do

This is wrong for anything that is closed source. You as a Windows or MacOS programmer, in more than 99% of cases, do not have access to the source code for the file system, for the drivers, for the OS. You are relying on somebody else to provide you with reliable APIs and system calls. You are relying on any return values, errors, and so on.

Do you as a programmer who wants to write a file to a disk want or need (or even care) about how the file gets from memory to disk, how an inode (or similar) is allocated, how the blocks on the file system are allocated, how one block is linked to the next block, how the permissions are set? In nearly all cases the answer is 'no'. File systems are a specialised subject. I recall more than a decade ago that IBM said there was only a few people in the world who could really create one, and they knew this as they tried to hire every one of them.

The standard C library on the open source operating systems is very widely used. I bet that most programmers have not looked at the source code to see what it does behind the scenes. They want to write a file, not care about how that file ends up on disk. That is why these libraries exist in the first place.

Perhaps I was not clear in the line that you have quoted. Yes, you need to that calling function xyz() does something. You don't need to know, in most cases, how it does it. That is what I meant by 'what it does'. If you use C++ and iostream, do you in most cases need to know or care what it is actually doing? The answer is no.

Anyway, we are probably getting off topic. We don't know the exact circumstances of how these Affinity files are getting corrupted. It could be user error. 

Link to comment
Share on other sites

1 hour ago, LondonSquirrel said:

This is wrong for anything that is closed source. You as a Windows or MacOS programmer, in more than 99% of cases, do not have access to the source code for the file system, for the drivers, for the OS. You are relying on somebody else to provide you with reliable APIs and system calls. You are relying on any return values, errors, and so on.

That's right for closed source and OS related service/APIs here, my point was here more that you still have to know in general what and why you use certain APIs then.

1 hour ago, LondonSquirrel said:

Do you as a programmer who wants to write a file to a disk want or need (or even care) about how the file gets from memory to disk, how an inode (or similar) is allocated, how the blocks on the file system are allocated, how one block is linked to the next block, how the permissions are set? ...

Yes usually you do at least for how to write files out, since there are differences in how to write a file back to disk, if in a plain serialized unbuffered manner, or via buffered streams then etc. Just look at the differences several C and C++ IO-functions offered over time here from past to yet. Though you usually don't care that much about the low level system inode/blocks handling here, as far as you don't suppose to write some disk formater and partitioning system tools.

 

1 hour ago, LondonSquirrel said:

The standard C library on the open source operating systems is very widely used. I bet that most programmers have not looked at the source code to see what it does behind the scenes. They want to write a file, not care about how that file ends up on disk. That is why these libraries exist in the first place.

Well I can't speak for all or other programmers here, but I for my part once intensively studied "Advanced Unix Programming - by Richard Stevens" in order to see how certain Unix system services and system related functions are implemented (...especially the differences between BSD and System V here) and have to be used at all. Learned a lot from that book and the Unix system docs & man pages about certain system library calls. If you are going to write some more low level tools for a specific OS you need to know a few things more about it und it's underlaying mechanisms!

1 hour ago, LondonSquirrel said:

Perhaps I was not clear in the line that you have quoted. Yes, you need to that calling function xyz() does something. You don't need to know, in most cases, how it does it. That is what I meant by 'what it does'. If you use C++ and iostream, do you in most cases need to know or care what it is actually doing? The answer is no.

I know what you mean and for general or most usual/common programming tasks that's all right, but there are also programming situations, where you have to go and dig deeper into the materia, especially if something is not behaving as it should or is expected to be, or if you encounter serious performance or stability issues within your specific programming context.

 

1 hour ago, LondonSquirrel said:

We don't know the exact circumstances of how these Affinity files are getting corrupted. It could be user error. 

I pretty much doubt that any user intentionally disconnects a USB drive or network connection here before files have been completely written. Also most direct & permanently connected USB drives should be gracefully handled by the OS (some modern USB 3.2 devices do offer for example a read of ~420 MBps and write of up to ~380 MBps speeds nowadays), even if the device before has before been send to sleep for saving overall electricity by the OS. - The overall problem is IMO more related to the specific Affinity file format and it's write handling, which is vulnerable to external connections.

☛ Affinity Designer 1.10.5 ◆ Affinity Photo 1.10.5 ◆ OSX El Capitan

Link to comment
Share on other sites

9 minutes ago, v_kyr said:

there are differences in how to write a file back to disk, if in a plain serialized unbuffered manner, or via buffered streams

Yes, I'm aware of that. If you roll your own way of writing bytes to disk you will need to know about it. But that is not the same thing as knowing 'everything' about a file ends up on disk.

I read, or rather dipped in quite regularly, several of Stevens' books. And the Design and Implementation of the 4.4 BSD Operating System. Let's not boast towards each other.

13 minutes ago, v_kyr said:

I pretty much doubt that any user intentionally disconnects a USB drive or network connection here before files have been completely written.

I have seen that happening many times. Indeed, there is something more subtle: if I eject one of my USB drives, the icon disappears from the desktop but the lights on the drive itself keep flashing for about 10 seconds. I doubt it is safe to unplug the cable until these lights have stopped. I only notice these lights because the drive is right in front of me.

I would agree that if Serif are doing something 'clever' that would the first place to check for any possible errors. 

Link to comment
Share on other sites

42 minutes ago, LondonSquirrel said:

I have seen that happening many times. Indeed, there is something more subtle: if I eject one of my USB drives, the icon disappears from the desktop but the lights on the drive itself keep flashing for about 10 seconds. I doubt it is safe to unplug the cable until these lights have stopped. I only notice these lights because the drive is right in front of me.

Then it's for an USB device probably due to write caching ...

Quote

Write Caching is the process whereby a device does not immediately complete writing a file, but instead caches some part of it to complete at a later time. When a USB storage device is inserted into a computer, data can be both written onto it and read off it. In order to improve its system performance, a computer may utilise its fast, volatile memory (RAM) to collect write commands sent to the USB storage device and ‘cache’ them until the slower, external storage device can be written to later. Other applications are then able to run faster, without having to wait for the external storage device to process the data write-requests.

Whilst this process increases system performance, there is also an amplified risk of data loss due to system failure, loss of power or unsafe removal of the external storage device. If the external drive is removed before the device flushes the cache and completes the whole write operation, then the file may not be readable. Naturally, the bigger the file size, the longer this process takes.

Windows’ default settings actually disable ‘Write Caching’ to allow for the quick removale of external devices without needing to use the ‘Safely Remove Hardware’ icon. However, as sticklers for performance, we would advise making use of this function.

The risk of power and system failures is sufficiently small so as to justify the increased performance benefits, and properly ejecting the flash drive by selecting the appropriate eject option prior to removal should cause the cache to flush and be written to the drive successfully.

USB related write caching can be one of the problems here with huger Affinity files, as far as the device isn't used permanently connected!

☛ Affinity Designer 1.10.5 ◆ Affinity Photo 1.10.5 ◆ OSX El Capitan

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...

Important Information

Please note there is currently a delay in replying to some post. See pinned thread in the Questions forum. These are the Terms of Use you will be asked to agree to if you join the forum. | Privacy Policy | Guidelines | We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.