Lens EXIF data deleted?

[wps]

Hello, I'm a new user of Affinity 1.6.  One issue I've run in to is that when I modify an image in Affinity, save it and then load it in to another editor (in this case DXO Optics Pro) the lens data is no longer there. It was there in the image before being edited by Affinity

If I reload it in to affinity, it shows it.  I also tried this in another editor and again it won't show the lens data in the exif.  How does Affinity change the exif  lens data during a save so that it recognizes it but other programs do not?

 

Thanks,

WPS

[wps]

Any help?

[v_kyr]

On AF Win or Mac? - For what file type RAW/TIFF/JPG and manufacturer cam type?

12 hours ago, wps said:

...

If I reload it in to affinity, it shows it.  I also tried this in another editor and again it won't show the lens data in the exif.  How does Affinity change the exif  lens data during a save so that it recognizes it but other programs do not?

If AP can show up the lens data afterwards it probably is still there in some context of the image. Try to use a tool like ExifTool in order to inspect if and where the lens data might reside. Aka if it is under some vendor specific Exif metadata section, or maybe has been moved into some new tagged Exif section etc. - In short, use a tool which will list ALL available Exif infos, even proprietary vendor ones in order to find out!

[R C-R]

13 hours ago, wps said:

One issue I've run in to is that when I modify an image in Affinity, save it and then load it in to another editor (in this case DXO Optics Pro) the lens data is no longer there.

Since no other apps besides the Affinity ones can open native format Affinity files (ones with the .afphoto or .afdesign extension), I assume instead of 'save & load' you mean export the Affinity document to some other format other editors can open. If so, one of the export options in AD/AP allows you to remove the metadata on export. (It is in the "Embed Metadata" checkbox in "More" window for File > Export & similarly in the Export Persona.) If you uncheck that option, the all metadata will be stripped from the exported file, including lens data.

 

This will not affect the Affinity format file, so if you open that file again in the Affinity app after exporting it to another format, the metadata will still be there. If you confuse the two files, this would explain what you are seeing.

[wps]

17 minutes ago, R C-R said:

Since no other apps besides the Affinity ones can open native format Affinity files (ones with the .afphoto or .afdesign extension), I assume instead of 'save & load' you mean export the Affinity document to some other format other editors can open. If so, one of the export options in AD/AP allows you to remove the metadata on export. (It is in the "Embed Metadata" checkbox in "More" window for File > Export & similarly in the Export Persona.) If you uncheck that option, the all metadata will be stripped from the exported file, including lens data.

 

This will not affect the Affinity format file, so if you open that file again in the Affinity app after exporting it to another format, the metadata will still be there. If you confuse the two files, this would explain what you are seeing.

Yes, I did mean export.  That option is checked and the only data that is removed (or moved, in this case) is lens data.  Everything else is there.

 

Thanks,

WPS

[R C-R]

1 minute ago, wps said:

Yes, I did mean export.  That option is checked and the only data that is removed (or moved, in this case) is lens data.  Everything else is there.

What do you mean by "moved"?

[wps]

The lens data is still there as AP can see it but DXO Optics can no longer see it after it has been exported by AP.  DXO Optics could see it before export by AP.

[R C-R]

1 minute ago, wps said:

The lens data is still there as AP can see it but DXO Optics can no longer see it after it has been exported by AP.  DXO Optics could see it before export by AP.

Are you saying AP can see the lens metadata in the exported file, or in the .afphoto one? What is the file format of the exported version?

[wps]

35 minutes ago, R C-R said:

Are you saying AP can see the lens metadata in the exported file, or in the .afphoto one? What is the file format of the exported version?

AP can see the lens data in the exported (jpg) file.  DXO Optics can see all metadata except the lens data after it has been exported by AP.

 

[wps]

2 hours ago, v_kyr said:

On AF Win or Mac? - For what file type RAW/TIFF/JPG and manufacturer cam type?

If AP can show up the lens data afterwards it probably is still there in some context of the image. Try to use a tool like ExifTool in order to inspect if and where the lens data might reside. Aka if it is under some vendor specific Exif metadata section, or maybe has been moved into some new tagged Exif section etc. - In short, use a tool which will list ALL available Exif infos, even proprietary vendor ones in order to find out!

Thanks, I'll give it a try.  This was AP on Win with a jpg export.

 

[v_kyr]

On Win you can also test more comfortably with the IrfanView or XnViewMP image viewers/converters (instead of DxO here) what's listet as a whole inside the exported JPG Exif data.

[wps]

I think I have discovered the issue.  DXO optics identifies the lens for correction by making use of the exif field 'MakerNote'.   AP doesn't include that field on export.  Solution is to use DXO optics first and then AP.  I prefer the lens corrections in DXO optics which is why I was using both programs. 

Anyway to get AP to include that exif field on export?

 

Best,

WPS

[v_kyr]

Ah Ok, so you probably found the cause of all evil (see also this about the Exif Makernote tag). Maybe that AP doesn't restructure/rewrite the whole initial Exif data structure when exporting/converting to other file formats and thus strips out some portions of the initial Exif infos. - However, the AP devs should be able to tell you here more concrete what they finally do, how they handle Exif data when exporting!

[R C-R]

14 minutes ago, wps said:

I think I have discovered the issue.  DXO optics identifies the lens for correction by making use of the exif field 'MakerNote'.   AP doesn't include that field on export. 

That may have something to do with this from the Exif Wikipedia article:

Quote

 

The "MakerNote" tag contains image information normally in a proprietary binary format. Some of these manufacturer-specific formats have been decoded:

{...}

Unfortunately, the proprietary formats used by many manufacturers break if the MakerNote tag is moved, i.e. by inserting or editing a tag that precedes it. The reason to edit to the Exif data could be as simple as to add copyright information, an Exif comment, etc. In some cases, camera vendors also store important information only in proprietary makernote fields, instead of using available Exif standard tags. An example for this is Nikon's ISO speed settings tag.

 

EDIT: The link @v_kyr provided explains this much better.

[wps]

2 hours ago, v_kyr said:

Ah Ok, so you probably found the cause of all evil (see also this about the Exif Makernote tag). Maybe that AP doesn't restructure/rewrite the whole initial Exif data structure when exporting/converting to other file formats and thus strips out some portions of the initial Exif infos. - However, the AP devs should be able to tell you here more concrete what they finally do, how they handle Exif data when exporting!

Thanks for the link, very helpful.  Now if I can get AP developers to take a look at this.

 

WPS

[R C-R]

3 hours ago, wps said:

Now if I can get AP developers to take a look at this.

I am reasonably sure they already know about this, but as the link @v_kyr provided explains, the real problem is camera makers are using the Makernote tag for many different kinds of metadata that should be in other metadata tags that are not encoded in some proprietary format that require reverse engineering to decode, and/or break if the Makernote tag must be moved because another tag is added or inserted before it in the file's embedded metadata.

 

IOW, Serif/Affinity did not create the problem, but it they would have to devote significant time & resources to fix it. This almost certainly is part of the reason why apps that do all the necessary reverse engineering & are coded to prevent the tag from breaking when a preceding tag is edited cost more.

[wps]

Actually, a couple of the free editing programs I just tried keep the MakerNote tag intact and none of my paid but less expensive programs (L.R., Corel, Photomatix) break it. AP is the only editing program I have that strips out the data.

[R C-R]

2 hours ago, wps said:

Actually, a couple of the free editing programs I just tried keep the MakerNote tag intact and none of my paid but less expensive programs (L.R., Corel, Photomatix) break it.

Did you try editing or adding any of the tags that precede the MakerNote one in these apps?

[wps]

9 minutes ago, R C-R said:

Did you try editing or adding any of the tags that precede the MakerNote one in these apps?

I didn't add or edit the actual tags in AP or any of the other programs.  I assume the only change to the tags would be the name of the program doing the editing. All I need AP to do is to pass along the tag as it exist when the image is loaded. All the other programs I have tried do this.

[R C-R]

5 minutes ago, wps said:

I didn't add or edit the actual tags in AP or any of the other programs. 

According to the links posted in this topic, doing that is what breaks the MakerNote tag.

[v_kyr]

Well most vendor specific Makernote Type specification formats themself are nowadays identified and have been reverse-engineered. So it's more a problem of reading and writing those back in a consistent manner when dealing with Exif data, since with binary data when changed you have to keep track of correct offsets, byte lengths and used byte order (endianness) etc. of hex IDs. Some better tools which are specialized in it (dealing explicitly with Exif data here), can also attempt to fix the incorrect offsets when they determine such issues or are forced to check for such issues.

However, most image software reuses some other third party tools for these purposes here, meaning they just (re)use some third party libs/code which deals with this. Similar as using third party tools/libs for dealing with RAW files, reading/writing image file formats, applying the lens data corrections, or other things. So they rely for certain functionality also on third party code they don't wrote and maintain themself, thus there might be still bugs and/or side effects when dealing with certain functionality here.

So it's possible that AP restructures the Exif Makernotes differently or incorrect when exporting JPGs. This usually could be inspected with a HeX editor, when searching after a concrete known makernote tag inside the image bytecode and looking for certain offsets. Or they do strip those tags generally before exporting, etc. etc. - There are many possibilities and thus instead of speculating here, it's best the Affinity dev team will then tell for sure!

 

[R C-R]

1 hour ago, v_kyr said:

Well most vendor specific Makernote Type specification formats themself are nowadays identified and have been reverse-engineered.

I am not so sure that is true. Consider the table from the link. From what I can tell, it only covers jpeg files generated by the listed cameras, so the tags in the innumerable RAW format outputs might be different. The table also doesn't show anything decoded, just raw ASCII & hex values for the first 20 (of how many?) characters in the tag.

 

1 hour ago, v_kyr said:

There are many possibilities and thus instead of speculating here, it's best the Affinity dev team will then tell for sure!

Judging from your reading and writing those back link, 'automagically' fixing these "particularly insidious" write-back problems is iffy at best, so "operator intervention" is required, meaning some technical knowledge and/or experimentation will be needed. If not done correctly, "some" (how much?) of the MakerNote information may be lost permanently.

 

So I suspect if the dev team posts any comments about it, they basically will say that the 'breakage' issue is a very difficult one to solve, regardless of the available third party libs/code the apps might use.

[v_kyr]

15 minutes ago, R C-R said:

I am not so sure that is true. Consider the table from the link. From what I can tell, it only covers jpeg files generated by the listed cameras, so the tags in the innumerable RAW format outputs might be different. The table also doesn't show anything decoded, just raw ASCII & hex values for the first 20 (of how many?) characters in the tag.

That's just a short comprehensive info there about (6624 unique camera models by 96 different manufacturers) and due to space and page table representation limits it is listed in that first 20 bytes form. If you want to know the whole big picture, you can dive into the tools code and there you can see how they handle it for different image file formats, vendor cams and so on. Or read through the tools website (there the supported file types and other informations etc.), in the hope that the website is keeped up to date and thus maybe represent the actual state.

48 minutes ago, R C-R said:

Judging from your reading and writing those back link, 'automagically' fixing these "particularly insidious" write-back problems is iffy at best, so "operator intervention" is required, meaning some technical knowledge and/or experimentation will be needed. If not done correctly, "some" (how much?) of the MakerNote information may be lost permanently.

Such things might can happen in certain case when dealing with... 

"Problems like this may be caused by image editing software which doesn't properly update offsets in the MakerNotes when rewriting an image. These offsets are used as pointers to reference tag values and structures within the metadata, and errors like this may lead to missing or incorrect values for some MakerNotes tags. In many cases, ExifTool will detect this type of problem and issue a warning like this when reading (or an error when writing)".

...and...

"...This is usually the appropriate choice if this problem was caused by editing the image with other software."

However, a bunch of RAW converters and photo imaging software handles Exif metadata correctly and is also capable to read and write back makernote parts the right way here, so Affinity should do too!

1 hour ago, R C-R said:

So I suspect if the dev team posts any comments about it, they basically will say that the 'breakage' issue is a very difficult one to solve, regardless of the available third party libs/code the apps might use.

Well honestly that would then again be another pretty meaningless answer like often seen before here. So I hope they have a better technically qualified explanation for this!

[wps]

1 hour ago, R C-R said:

 

 

So I suspect if the dev team posts any comments about it, they basically will say that the 'breakage' issue is a very difficult one to solve, regardless of the available third party libs/code the apps might use.

It's apparently wasn't difficult to solve for the other programs I tried, even Irfanview, a free viewer with some limited editing function, will export the tags correctly. And, as I stated, of all the programs I have tried, AP is the only one that doesn't do this correctly. 

[R C-R]

4 hours ago, v_kyr said:

If you want to know the whole big picture, you can dive into the tools code and there you can see how they handle it for different image file formats, vendor cams and so on. Or read through the tools website (there the supported file types and other informations etc.), in the hope that the website is keeped up to date and thus maybe represent the actual state.

 

OK, for example dive into the whole big picture for Nikon at http://owl.phy.queensu.ca/~phil/exiftool/TagNames/Nikon.html. Search the page on "unknown," look for the question marks, or note what it has to say about Nikon LensID Values:

Quote

The Nikon LensID is constructed as a Composite tag from the raw hex values of 8 other tags: LensIDNumber, LensFStops, MinFocalLength, MaxFocalLength, MaxApertureAtMinFocal, MaxApertureAtMaxFocal, MCUVersion and LensType, in that order.

 

Or consider what he has to say at http://owl.phy.queensu.ca/~phil/exiftool/writing.html about writing tags:

Quote

 

Currently, ExifTool can write most of the EXIF tags that anyone could reasonably want to change (but some tags are protected because they describe physical characteristics of the image that you can not change with ExifTool, eg. Compression). Also, all of the GPS, IPTC and XMP information and most of the MakerNotes information can be edited. This gives you great power, but with great power, comes great responsibility...

 

It is possible for you to write nonsense into a file, which could cause other image readers to throw up their hands in despair and refuse to read the image. For this reason, it is best to always preserve the original copy of your image file. The "exiftool" script does this for you automatically by renaming the original file and always working on a copy.

 

That page also includes other info about the problems & pitfalls of writing metadata tags, even with tools based on an ExifTool backend.

 

So it seems pretty obvious that if we do dive into the big picture it becomes very clear that there is nothing simple about this, & that the issue is in fact a very difficult one to solve.