Corrupted file — I need help.

[carl123]

4 hours ago, walt.farrell said:

1. The app tells the system (device driver) that it wants to write the first 4KB of the file.
2. The device driver signals the disk drive to accept 4KB of data and sets the writing position to the start of the file.
3. The disk drive prepares to write data at that location, prepares to receive it, and starts receiving.
4. The device driver starts sending the data.
5. The data never arrives (cable was pulled, system crashed, ...) so the disk drive sees zeros.
6. Disk drive writes what it saw: zeros.

7. Device driver rereads the 4KB data from disk and verifies data is correct
8. If data is incorrect, resend the data (retries 3 times)
9. If data is still incorrect activate "Error protocol 639" 

 

[Xzenor]

2 hours ago, carl123 said:

"The times they are a changing" - Bob Dylan

"Windows 10 now lets you pull out USB drives without ‘safely removing’ them" - thenextweb.com

"Users that insist on writing Affinity Files to USB drives should disable write caching on those drives to lessen the risk of corrupt files" - me

 

Reference:

https://thenextweb.com/microsoft/2019/04/09/windows-10-now-lets-you-pull-out-usb-drives-without-safely-removing-them/

I stand corrected. Write caching is no longer the default.

However, we don't know the settings, nor the exact OS (could be windows 7 or 8 as well)

[Xzenor]

On 8/7/2020 at 5:18 AM, konstantnnn said:

Affinity Photo on my Windows PC was opening a file, and I edited it completely fine — then I went and saved it. The program seemingly saved and closed the document and everything was fine. I transferred the file on an external drive and left it there for a couple of months.

I think we all missed this important part.... AP was not writing to usb storage.

[konstantnnn]

Exactly. And when iPad opens a file, it never writes directly, only imports to affinity photo.

[konstantnnn]

I was working on a file that I began on my iPad, using iCloud synced it to my PC and finished working on my PC. (Win10 ver2004). 

I saved it to the desktop, and i noticed that the thumbnail was not generated. I just thought “windows bug probably” and I moved the file to my external drive I was keeping all my documents in.

[konstantnnn]

23 hours ago, konstantnnn said:

I had many other files on my external drive… should i not work on them directly from affinity photo but move them to my desktop from the drive?

I asked this for future measure only, I realized that I have actually always saved stuff to the desktop first and then transferred to the external drive/usb.

[Blende21]

Anybody can decide where to keep a file, and from where to open it. The risk involved has to be balanced with the workflow needs. If you have a lot of pics, there is no economical way to store them all on your main drive.

If you run a desktop computer, and do not have a UPS applied, data loss can occur as well on an internal drive when electrical current is lost.

There is no need to make wild changes to software - just run a good backup strategy.

[walt.farrell]

8 hours ago, carl123 said:

7. Device driver rereads the 4KB data from disk and verifies data is correct
8. If data is incorrect, resend the data (retries 3 times)
9. If data is still incorrect activate "Error protocol 639" 

 

Yes, but if (a) the device has been removed (cable pulled) or (b) the system has crashed, then steps 7-9 cannot be performed.

And, generally, I doubt that every write is verified by rereading. that really hurts performance. But I don't know PC (or Mac) protocols well enough to say for sure.

[Xzenor]

3 hours ago, walt.farrell said:

I doubt that every write is verified by rereading. that really hurts performance.

Well, to be fair, USB stick performance is mostly terrible.

[R C-R]

1 hour ago, walt.farrell said:

And, generally, I doubt that every write is verified by rereading. that really hurts performance. But I don't know PC (or Mac) protocols well enough to say for sure.

Long ago, when Macs ran on System 7 or 8, HDD's were attached using SCSI protocols & the OS relied on cooperative multitasking to run multiple apps. Back then, write verification via read-back was an available option for some HDD's. It could be enabled using certain 3rd party software, jumpers, or as a factory configuration. It totally clobbered performance so it was rarely used for anything other than for critical backup storage & the like. 

IIRC, the way this worked was the drive had to wait until the data was transferred from its onboard write buffers to the disk, then read it back to its read buffers, then compare that to what was in the write buffer. Only then could it flush its buffers & signal back to the OS (or app?) that it was ready to accept more data. Cooperative multitasking & other factors made this even slower that it sounds.

I am not certain but I think once Apple started using PATA & for internal HDD's & supporting ATAPI for external storage devices, write verification all but vanished from the Mac world, in part because it greatly complicated read-ahead/write-behind & other performance enhancing features HDD's were adopting in that era.

Of course, these days everything is much more complicated because data can be stored directly on locally attached devices like HDD's, SSD's, flash ("thumb") drives, multiple kinds of SD cards or even certain cameras; or remotely on LAN devices or 'in the cloud' via WAN. So not only are there many different protocols there are also many different file systems to consider.

With all that in mind plus what @Ben said about the OS level limitations, it seems obvious that write verification is totally impractical. Adding a 'performance vs. safely' choice of some kind to the Affinity desktop apps would be possible but giving users enough info to make an informed choice probably isn't practical.

Something that might be useful is a warning that pops up whenever a user tries to open or place a file that is not locally stored explaining that this is not recommended with a 'proceed at your own risk' disclaimer, but I suspect there might be technical issues we are not aware of even with that.

[carl123]

9 minutes ago, R C-R said:

Something that might be useful is a warning that pops up whenever a user tries to open or place a file that is not locally stored explaining that this is not recommended with a 'proceed at your own risk' disclaimer, but I suspect there might be technical issues we are not aware of even with that.

In a commercial environment all data tends to reside on centralized servers (networked drives) for security and consolidated backups reasons.

Having an app telling you to "proceed at your own risk" on every new save is never going to happen. It would scare the life out of the users 😄

Then again, any app that can't function correctly with (properly configured and maintained) network drives will never make it into a commercial environment.

[Blende21]

What is the result of warnings and checkboxes ?

The user gets interrupted in his workflow, does not think about the warning (the more words it uses to explain stuff the less it will be read) and clicks it away anyhow. The informed user does not need it, the uninformed will not be stopped, but both will be annoyed by this super-nanny behavior.

And all this because somebody wants to use a USB flash device to store his data ? C‘mon ...

[R C-R]

52 minutes ago, carl123 said:

In a commercial environment all data tends to reside on centralized servers (networked drives) for security and consolidated backups reasons.

For backups maybe, but these days in particular there are many commercial environments where for one reason or another it is routine to download documents to local storage to work on them, for example when a reliable network connection is going to be unavailable when it is time to work on them.

Regardless, my point is simply that considering the OS limitations @Ben mentioned, there is no 100% risk free way to work on a document in Affinity, even if it is stored locally. That should scare users enough to either implement a good backup strategy of their own or to take best advantage of one that already exists in their environment, commercial or otherwise.

[Blende21]

No backup, no mercy ...

[R C-R]

19 minutes ago, Blende21 said:

And all this because somebody wants to use a USB flash device to store his data ? C‘mon ...

It is not just because of the type of storage device, USB flash drive or otherwise. It is also because of how Serif has implemented the native Affinity file format to improve performance/responsiveness, minimize memory use, & so on.

Reworking that at this point in the development would very nearly be like starting over.

[Xzenor]

5 hours ago, carl123 said:

 

Then again, any app that can't function correctly with (properly configured and maintained) network drives will never make it into a commercial environment.

Well, I'm not sure if you heard about this commercial mail client that doesn't support archive files to be stored on network drives.. it's fairly well known. It's called Microsoft Outlook.