Corrupted file — I need help.

[konstantnnn]

Guys I need help. 

Affinity Photo on my Windows PC was opening a file, and I edited it completely fine — then I went and saved it. The program seemingly saved and closed the document and everything was fine. I transferred the file on an external drive and left it there for a couple of months.

The photo was a part of hundred other documents I was collecting to use on a huge composite for a school final. Now when I attempt to open a file on my PC or iPad, it shows me an error that the Application can’t open the file because it’s not “valid”. 

 

https://drive.google.com/file/d/19gyGMDY5A7Fz1eWU7PC3hLcXIu4xXzqo/view?usp=sharing

[Dan C]

Hi @konstantnnn,

Sorry to hear you're having trouble - I've forwarded this thread to our developers for further investigation, hopefully they will be able to repair this file - I'll be sure to respond here as soon as I hear back from them

[Xzenor]

I don't know what happened.. but this is weird.

 $ file Z004.afphoto
Z004.afphoto: Linux-8086 executable, unmapped zero page, impure, A_PAL

while an ordinary file is just data

$ file square.afphoto
square.afphoto: data

 

[walt.farrell]

There's about 4KB of zeros at the start of that file, that look suspicious to me.

[Xzenor]

30 minutes ago, walt.farrell said:

There's about 4KB of zeros at the start of that file, that look suspicious to me.

Exactly.. suspicious is what it is.

[Ben]

If the start of the file has been overwritten with zeros, then it will be missing vital data and can't be fixed.

[konstantnnn]

I had many other files on my external drive… should i not work on them directly from affinity photo but move them to my desktop from the drive?

[Dan C]

8 minutes ago, konstantnnn said:

should i not work on them directly from affinity photo but move them to my desktop from the drive?

We always recommend avoiding working directly from external drives, as this can cause corruption in your Affinity files if access to the drive is lost (such as an accidental unplug).

My sincerest apologies that we can't recover this file in this instance!

[konstantnnn]

Thank you all very much for trying.

 

:))

[Pšenda]

6 minutes ago, Dan C said:

as this can cause corruption in your Affinity files if access to the drive is lost

Sorry, but if the access to the external drive is lost, the whole file is "damaged/destroyed", so it is not a well-designed interface and manipulation of working files.
If the concept of accessing working files does not allow file consistency techniques (something like the ACID for database files), then working with files on external drives should be disabled.

It is not possible to rely on the fact that the user knows (does not know - he has nowhere to find out) that he should first transfer the files to the local disk.

[konstantnnn]

Apsolutely. Working on files off USB drives should be disabled. I thought affinity could bring the document on the macbooks hard drive in the background without me moving it over, kinda like what photoshop does

[Ben]

@Pšenda This is not the issue.  We have all the checks possible to allow saving files to any storage medium (in fact, more than most software).  The issue lies in the way that USB attached storage buffers data to later be written to physical storage.  We will have finished writing the file with no error messages coming back from the Operating System.  The *actual* write to the physical hardware happens at a later time that is out of our control - there is additional management of the hardware performed by device drivers, etc.  This is one of the reason why you MUST dismount external drives BEFORE disconnecting them.

We rarely (if ever) see these kind of corruptions happening on local storage, and that is why we advise people to work on local storage (which is also faster).  Attached storage is good for duplicated backups, and we recommend that also.

If a file fails to write during our saving process, we have checks and mitigation for that - as well as disconnected network shares, changes in access/permissions, out of space, etc.  What we have no control over is what happens to the file after the saving process completes.

[Pšenda]

Thank you @Ben, but this only confirms that Affinity should "disable" working with data files on external drives, because the correct writing of data is beyond your application control.
Or at least a warning dialog when opening data from an external drive "your data may be irretrievably destroyed".

[Xzenor]

33 minutes ago, Pšenda said:

Thank you @Ben, but this only confirms that Affinity should "disable" working with data files on external drives, because the correct writing of data is beyond your application control.
Or at least a warning dialog when opening data from an external drive "your data may be irretrievably destroyed".

Your operating system (Well windows anyway) gives a warning that you should never just pull out usb storage devices because it might corrupt the data.
I'd say that's enough. If you're too stubborn to follow that advice then that's your own fault.

Every software has this. Microsoft Applications have the same problem. Adobe software has the same problem. Even notepad has this issue! None of them give any message or warning.
It's not an application issue, it's the operating system (which actually DOES give a warning).  So it's not the application's responsibility to fix it.

[Pšenda]

6 hours ago, Xzenor said:

Every software has this.

No, not really - I recommend studying something about this problematics (for example, the mentioned ACID).

[Xzenor]

I think you're comparing apples to oranges with that ACID comparison. We're not working with a database here. But that's jut my opinion.

 

[Pšenda]

23 minutes ago, Xzenor said:

We're not working with a database here.

A database application is "software" like any other, database files are files like any other, and can be stored on local or external drives - and of course without the risk of damaging and destroying the contents in case of problems such as power fail or loss of access while writing data.

That's why I mentioned this as an example of an approach to data storage - that's just my opinion, which you tried to convince me was wrong, and that destroying files when writing data to external drives is commonplace. No, it really isn't - we already have 21st centuries.

[Blende21]

File corruption by pulling an USB device without unmounting it is a known issue. A file is not a database management system, where an operation can wait for confirmation before moving on to the next access. This is by no means standard behavior of a DBMS because it sucks in performance, but used on critical data where integrity is the first concern, over speed.

No warning can anticipate the moment in which I (wrongfully) decide to pull a flash device or SSD out of the USB connector while it is in operation. There is no strategy to prevent this action, which may lead to a loss of data.

Personally I keep photos for sorting and editing on a large USB SSD after downloading them, and would not accept software to prevent this. It is my own responsibility to handle it correctly.

 

[thomaso]

12 hours ago, Ben said:

If the start of the file has been overwritten with zeros, then it will be missing vital data and can't be fixed.

Ben, I'd like to understand the structure better. – At what moment get the zeros written? By what application?

Does Affinity write them when it opens the file, e.g. with the plan to replace the zeros when closing the file?
If not, wouldn't the file remain in its state like before opening?

And what is different in a file which can't be opened by the user's Affinity but can be restored by developers?
As with this file: https://forum.affinity.serif.com/index.php?/topic/121665-affinity-photo-doesnt-recognize-the-file/&do=findComment&comment=665735

[walt.farrell]

17 minutes ago, thomaso said:

If not, wouldn't the file remain in its state like before opening?

This is a guess at a possible scenario, based on situations I've seen on mainframe computers. If you were very unlucky:

1. An application could initiate a Write operation, and the system (device driver) could initiate the data transfer to the disk drive.
2. At that point, the disk drive is prepared to write the data to the physical disk, and expects the data to flow down the wires with certain timing.
3. If you were to pull the USB cable at that instant, or if the system were to die, the disk drive is still expecting the data to arrive, and has already started writing it to the physical disk.
4. If the data doesn't arrive (because of the pulled cable, or system crash) then zeros get written instead of the expected data.

[Pšenda]

26 minutes ago, Blende21 said:

because it sucks in performance, but used on critical data where integrity is the first concern, over speed.

So let Affinity give the user a choice - whether he prefers speed over data security.

 

33 minutes ago, Blende21 said:

File corruption by pulling an USB device without unmounting

??? Did anyone talk about that here? As far as I know, an accidental/unexpected loss of access to the file was mentioned here, not a user-controlled disconnection of the external drive.

 

32 minutes ago, Blende21 said:

No warning can anticipate the moment in which I (wrongfully) decide to pull a flash device or SSD out of the USB connector while it is in operation. There is no strategy to prevent this action, which may lead to a loss of data.

??? Did anyone talk about that here?

[Blende21]

Maybe read the thread more carefully before continuing with posting:

There is no such „choice“, because we are talking about file corruption, not a database design. USB drives only allow for a few file systems, and none of them is proofed against an unconnect while writing. 

DanC & Ben talk about data loss through an accidental unplug. They explain what probably happened, and why it can’t be avoided.

As I understand it, before writing stuff into the cell of a flash drive, it is set to zero first (which is one of the two allowed values of a cell in a binary system). In the next step, all cells to take a „1“ are flipped. When this following operation is not executed, the „0“ values remain, and the File is corrupted. If the zeros are in the files header, it can become entirely illegible, if in a later sector maybe only damaged.

This operation is done by the controller on the USB device itself, not by any software running on the computer to which the USB device is connected. If you unplug during R/W, the USB controller looses power and stops execution wherever it was at that moment.

[thomaso]

11 minutes ago, walt.farrell said:

If the data doesn't arrive (because of the pulled cable, or system crash) then zeros get written instead of the expected data.

Again, what app is writing the zeros? – If the cable is pulled * then it can't be the computers OS nor Affinity but the USB disk system only. Don't you think it would be so clever to store 'old' data in memory until the 'new' are written, to be able to replace the zeros with the stored memory in case the new data don't arrive? – But how would it be able to end this process in either way if the whole situation occurred because the disk got suddenly unplugged from data AND from energy? Would it mean that the zeros get written in advance, kind of long time before the disk got unplugged?

* Here it was not unplugging; the OP wrote "I transferred the file on an external drive and left it there for a couple of months (...) Now when I attempt to open a file (...)".

However, your and my theory sound quite speculative. That is why I asked @Ben as an Affinity developer.

[walt.farrell]

26 minutes ago, thomaso said:

Again, what app is writing the zeros? –

No app is writing zeros, in my hypothetical scenario. It's purely a hardware effect because the hardware is expecting to receive data and never does.

1. The app tells the system (device driver) that it wants to write the first 4KB of the file.
2. The device driver signals the disk drive to accept 4KB of data and sets the writing position to the start of the file.
3. The disk drive prepares to write data at that location, prepares to receive it, and starts receiving.
4. The device driver starts sending the data.
5. The data never arrives (cable was pulled, system crashed, ...) so the disk drive sees zeros.
6. Disk drive writes what it saw: zeros.

There's basically, in that scenario, a point somewhere between steps 3 and 5 where the disk drive is committed to writing data, and if it doesn't get data it (the disk drive) corrupts the contents of the file.

[carl123]

13 hours ago, Xzenor said:

Your operating system (Well windows anyway) gives a warning that you should never just pull out usb storage devices because it might corrupt the data.

 

 

"The times they are a changing" - Bob Dylan

"Windows 10 now lets you pull out USB drives without ‘safely removing’ them" - thenextweb.com

"Users that insist on writing Affinity Files to USB drives should disable write caching on those drives to lessen the risk of corrupt files" - me

 

Reference:

https://thenextweb.com/microsoft/2019/04/09/windows-10-now-lets-you-pull-out-usb-drives-without-safely-removing-them/