Preflight Error "Data Merge sources need updating" Won't Go Away

[Anklebuster]

I'm using Affinity Publisher, Windows build 1.10.4.1198.

I'm using XLSX as a data source. It hasn't been changed recently.

Preflight panel shows the warning and a Fix button, but it doesn't seem to work.

I have no idea if my manipulations caused some kind of disconnect. But here are the things I did to create a template:

• Initially, I began the whole process in my main document, "Puzzles"
• I created a page (not Master) and used the Data Merge Layout tool to create a "Sudoku" grid
• After learning more about Data Merge, I decided I should have a separate .AFPUB template
• I copy/pasted the page from "Puzzles" to a new document
• I belatedly realized that I should be using a Master page
• I copy/pasted from the normal page in my new document to Master A
• This is key: I was concentrating on the Master page and totally forgot to delete the Data merge Layout from the normal page
• The source field on the normal page was called "Solution", then I changed it to "Puzzle" before I copied it to Master A
• Meanwhile, I kept tweaking the Master page and renamed the field from "Puzzle" to "N" to help get rid of an Overflowing text frame error
• After each tweak, I would save the project as a template.
• Every time I generated the merged document and tried to export it to PDF, Preflight complained about three errors
• I finally removed the old Data Merge Layout from the normal page. I resized the text frame in the upper cell to handle the Overflowing text error

Maybe all that copying, pasting and renaming confused the software. If not, then perhaps the preflight check is not seeing the XLSX as being unchanged.

Edited February 11, 2022 by Anklebuster
typo

[Dan C]

Hi @Anklebuster,

Sorry to hear you're having trouble! I'd like to request a copy of your .afpub file and the data merge document, so that I can investigate this further for you.

Can you please upload this to the below link for me:

https://www.dropbox.com/request/lY9IrGZBa3zjJvENUmBX

Once uploaded, please reply here to let me know. Many thanks in advance

[Anklebuster]

Hi Dan,

Thanks. The two files have been uploaded. It's really no trouble, because the merge works. I look forward to the results of your investigation.

 

Cheers,

 

Mitch

 

 

[PixelEngineer]

On 2/10/2022 at 9:23 PM, Anklebuster said:

I'm using Affinity Publisher, Windows build 1.10.4.1198.

I'm using XLSX as a data source. It hasn't been changed recently.

Preflight panel shows the warning and a Fix button, but it doesn't seem to work.

I have no idea if my manipulations caused some kind of disconnect. But here are the things I did to create a template:

• Initially, I began the whole process in my main document, "Puzzles"
• I created a page (not Master) and used the Data Merge Layout tool to create a "Sudoku" grid
• After learning more about Data Merge, I decided I should have a separate .AFPUB template
• I copy/pasted the page from "Puzzles" to a new document
• I belatedly realized that I should be using a Master page
• I copy/pasted from the normal page in my new document to Master A
• This is key: I was concentrating on the Master page and totally forgot to delete the Data merge Layout from the normal page
• The source field on the normal page was called "Solution", then I changed it to "Puzzle" before I copied it to Master A
• Meanwhile, I kept tweaking the Master page and renamed the field from "Puzzle" to "N" to help get rid of an Overflowing text frame error
• After each tweak, I would save the project as a template.
• Every time I generated the merged document and tried to export it to PDF, Preflight complained about three errors
• I finally removed the old Data Merge Layout from the normal page. I resized the text frame in the upper cell to handle the Overflowing text error

Maybe all that copying, pasting and renaming confused the software. If not, then perhaps the preflight check is not seeing the XLSX as being unchanged.

 

On 2/11/2022 at 5:09 AM, Dan C said:

Hi @Anklebuster,

Sorry to hear you're having trouble! I'd like to request a copy of your .afpub file and the data merge document, so that I can investigate this further for you.

Can you please upload this to the below link for me:

https://www.dropbox.com/request/lY9IrGZBa3zjJvENUmBX

Once uploaded, please reply here to let me know. Many thanks in advance

Hello, I created my own Data Merge example and simulated the problem and was able to re-create this scenario:

I used the Data Merge Layout tool to create the box where the data would be imported from the records in the .xlsx document.

Verifying the preview text, I then went to change the record in the .xlsx document and noticed preflight warning me that "Data Merge sources need updating" in the preflight. When I clicked the "Fix" button nothing happened. When I clicked the "Update" button in the Data Merge dialog box, nothing happened. I also wondered what went wrong? 

The way to prevent this from happening is to make sure that you click "Save" in your .xlsx document first before clicking "Fix" button or "Update" button since it is pulling this data from another source's saved state. Preflight is smart and noticed the change live, but the "Fix" and "Update" buttons will not work until you first save the .xlsx document. Generally when working with databases, the source should be saved first before updating your document. After doing this everything worked.

Note:

Please note that this was an observation above is based on a scenario that I created from my end having the .xlsx document open "live" while I was working on it on my own version of a Data Merge document.

 

[Anklebuster]

Hey PixelEngineer,

Thanks for weighing in on this. I'm glad things worked out in your scenario. In my case, I had not made any changes to the workbook recently.

Your answer makes me wonder if AP "forgets" to reset its timestamp (I'm assuming that's how it tracks whether a data source has changed.)

In other words, if you never edit the workbook after attaching it do the AP document, you will never get the warning; and, if you edit it once,

Data Merge never fixes its point of reference.

 

[PixelEngineer]

On 2/11/2022 at 3:22 PM, Anklebuster said:

I had not made any changes to the workbook recently

You do have a point there. This is something that definitely needs to be looked at.

 

Update: see posting dated Monday, 2/14/2022, regarding this.

[PixelEngineer]

Image snapshot source: Affinity Publisher Help Data Merge

The following will only cover Data Merge errors that I've come across and how I approach them.

For "Data Merge Sources need updating", this is how I'm using the Fix button for now.

It seems that clicking "Fix" will invoke a check to see if there are any fixes that need to be made from changes in an updated and "Saved" database source.

After clicking "Fix", If it finds that a field has changed, a Preflight error description (Field . . . not found in source. . .),will populate below this warning and identify the name of fields that are not matching with your source database. 

Clicking the "Fix" button will not magically update a changed field. To solve these issues you need to:

1. Uncheck 'Preview with record' in the Data Merge dialog box to reveal your field names in your Affinity Publisher document.
2. Double click the error, it will navigate to the field and highlight it for you.
3. With the  old "tag" field highlighted, navigate to the ‘Fields’ tab to insert your replacement field by double clicking on it.

'Fields' tab - View > Studio > Fields, under 'Data Merge - [your file name]' double click the replacement field name. If you have preflight set to 'Live' you will see those errors disappear immediately.

In your Preflight, if you ever get an error that says "No Enabled Data Source. .”, then this means that you need to check 'Merge Enabled' in your Data Merge dialog box so that your data can merge properly.

After fixing all your "red" errors you can then merge your data to a new document where the data source will disconnect upon conversion and you will no longer see the "Data Merge Sources need updating" in your preflight area.

In your template, I suppose you can look at the “Data Merge Sources need updating" to serve as a reminder that you must always update your document if you change anything in your source database and check for errors. The manual has some more detail under Data Merge, Fields, and Preflight.

 

[Anklebuster]

@PixelEngineer I actually did encounter the Field not found in source early on.  I was using a rather long label for single-digit data (Sudoku numbers) so I did what your post suggests to fix it.

I haven't tried a TXT source, yet. Have you had any issues with "Data Merge sources need updating" with file types other than XLSX?

[PixelEngineer]

On 2/12/2022 at 10:11 AM, Anklebuster said:

I haven't tried a TXT source, yet. Have you had any issues with "Data Merge sources need updating" with file types other than XLSX?

 

Yes, Anklebuster, I have. I tested .csv which is a type of text file and it works the same way. 

I would suggest the following:

Perhaps the constant warning in Preflight is being misinterpreted regarding its purpose within a user's workflow. 

After working with several merge routines, it seemed to make sense to have the warning and fix button always visible. It’s possible that a user needs to understand what it means and how to use it. Preflight warnings and errors can be disabled by editing its profile. 

Observations:

The 'Fix' Button does many things:

1. It updates the document with any existing records that have been modified or added by the user from their "saved" source database. It seems that these checks have to be invoked manually by design, after doing some reading in the manual. Having the 'Fix' button always present, is a very handy feature to have without needing to open up the Data Merge modal dialog box to do the updating, while also serving as a reminder.
2. The 'Fix' button also triggers the built in "Data Merge Fields" Preflight profile while modifying a document to check and see if any fields have been modified in relation to the source database. It seems that triggering the check manually via the 'Fix' button is by design. Any potential warnings, however, get triggered automatically upon opening the document. The fields will not update automatically as the program cannot tell what the user's design intent is. 

 

For me, it makes total sense when I look at it this way:

"Data Merge sources need updating" (Constant reminder to check and update your records before generating a merge). Use the Fix button to do this.

Tool tip of this "flyout" message: "Checks if a data merge source file has been modified since the last update" same as described above, (click the fix button to check and update if the data merge source file has been modified). Its a manual process. 

Keep in mind that Preflight is a tool is designed to help you with your workflow. It there to provides information based on a  user's requirements. You can always go into the options and modify its alert level or turn it off. 

So far any warnings or errors that I have come across make sense. Even those that can occur after generating a perfectly good document.
A couple of months ago, I also didn't understand what this message meant and how to get rid of it. But now I see things differently. 

Sometimes what looks like a bug may not be a bug, especially when learning how to use certain features within a program. When I first started working with the Data Merge tool it seemed "buggy" at first. As I gained more experience using it, I realized that that all I needed to do was learn how to use it properly. 
Hope this helps.🙂

BTW, 7:17 minutes into following video is an interesting tip that may help you with one of the issues you encountered.
Source: YouTube Affinity Publisher Channel
Affinity Publisher - Data Merge Layout Tool

 

[Anklebuster]

Hi PixelEngineer,

While your interpretation may be valid, I have two issues that preclude my accepting the perspective of "Always on...Always helpful", if I may paraphrase. First take a look at these snapshots:

merged.png shows a red bulb, highlighting a perfectly reasonable No Data Source... error. The newly generated document has no connection to the data. Since it is based on a template that does contain the field in a Master Page, I understand and accept that error.

template.png shows the document that is used to create merged. The bulb is green.

If the Preflight were to be an assistant, it would show red, letting me know that I may have to update the source, if I understood you correctly.

The second issue I have is that the Fix button never responds to my "click" to fix Data Source needs update, but reacts to my "click" on all other errors where I have had the option to use it, and preflight messages go away, turning the bulb green again. At the very least, that would be inconsistent UX.

As for the video tip: I only have one field and once I changed the name to the single letter, N, I removed the overflow issue.

Overall, this fix button is very minor. I'm preparing other layouts to take advantage of all the cool stuff I learned from Elaine Giles.

 

Cheers,

 

Mitch

 

 

 

[PixelEngineer]

Hello,

I understand the confusion.

Your points are valid.

The one thing that does appear confusing, is that with other warnings, after clicking fix button the warning will disappear. However, due to its nature in how it checks data from external database, this Preflight feature would be an exception to the rule.

This warning paired with the fix button seems to work a little differently than the others, since I noticed that the two Preflight profiles that deal with Data Merge are linked together in the Preflight settings. The fix button also seems to "ping" the external document, checking for mismatches by comparing fields. It is very important to note that the fix button does not update 'fields' information when you see warnings pop up. Those warnings belong to another Preflight profile and those updates are done by using another method. If there are field errors, they will show up prompting the user to repair them. Once the 'field' errors are corrected using the proper method by clicking the field value in the fields tab menu, pressing the fix button again will "ping" the external document to compare and see if there is a mismatched between the database and the document. If 'fields' matched, the 'field' errors will go away and the records will update when pressing the fix button. It is very important to understand the dynamics of the workflow process using this Preflight feature, while keeping your "bearings" (orientation) of different elements that you are working with. The manual states that external documents are checked manually by design. The benefits of updating manually make sense as it would result in the application working smoothly since using other methods can cause a great performance hit. This is why I am drawn to this conclusion that the fix button needs to be available at all times to check for these updates.

Keeping on topic,

The button is working since it does check and update the records table in the Affinity Publisher document.

The fix button checks your database and updates the records table in the document that you are working on. You can see this in action. It works every time under the condition where the fields have not changed. That is the fix.

Try it:

1. Add one new record to your database and make sure you save it. (I missed saving on occasion)
2. Open Affinity Publisher, look at the amount of records you have in the Data Merge dialog box under source before doing anything.
3. Click the fix button and you will notice that one more record was added. You can confirm that it works by looking records readout in the Data Merge dialog box. That's the fix. You have now updated your table of records successfully within the document.

 

In the following paragraph, when I mention 'field' and 'records', this pertains to the actual structure of the  database:

To further clarify the fix button only updates 'records' and checks for existence of 'field' errors but never fixes changed 'fields' by design.  If a field was changed, an error will appear. The user then needs replace old field with the new field properly (using the correct methods) as described in my previous post. After updating the field,  fix button will then be able to update the records below it accordingly. It works as designed.

You mentioned:

" . . . highlighting a perfectly reasonable No Data Source... error. The newly generated document has no connection to the data."

Yes, you are correct, the data source disconnects upon conversion when you merged to a new document. The information is useful as it is informing the user of the document's current state. If a user doesn't like the red error warning they can change it to yellow or disable it.

Regarding the green light I don't know what your settings are but here is one possible scenario:

Looking at your snapshot I could’ve assumed that your Preflight warnings were off for Data Merge. But I found that this is not necessarily the case.

Without looking at your actual document, I was able to create a situation where I had all my Data Merge Preflight warnings active and it showed green. How did this happen? Because I used the wrong workflow. Was it a bug . . . no. What happened in the simulation is that I updated my fields by typing over them which is an incorrect way to update fields. Typing over a field erases its properties. The errors didn't show up because Data Merge Preflight was specifically designed to find errors by matching fields between a document and the data source. It had nothing to match, so it didn’t see an error. This is why there was a green light, user error ironically. Preflight basically acts as a checklist but it is still up to the user to do the navigating by establishing a proper workflow in a given situation. 

Go to the menu Text > Highlight Fields. If the field tags are not highlighted in this mode then they are not real fields. This feature is very helpful.

The above scenario works same way in Microsoft Word's mail merge. User's just have to learn how to use the feature properly. One thing is for sure just like other database programs, there is a lot to learn. 

Please keep in mind that these are simply observations base on what I have observed reading the manuals and working with several merge routines. 

 

 

 

 

[Anklebuster]

Hi PixelEngineer,

Thanks for your insights. I have seen the record indicator change, so I know that works. In fact, I also know the limits of data merge, since I mistakenly tried to add 535 Sudokus to my document! LOL

When you say reading the manuals, did you spring for the Affinity workbooks? I have all three, though I've only cracked the Publisher workbook open. It is definitely a worthwhile purchase. Couple that with Elaine Giles' YouTube channel and we can become experts (or at least proficient) in short order.

Cheers,

 

Mitch

 

[PixelEngineer]

On 2/15/2022 at 2:04 PM, Anklebuster said:

Hi PixelEngineer,

Thanks for your insights. I have seen the record indicator change, so I know that works. In fact, I also know the limits of data merge, since I mistakenly tried to add 535 Sudokus to my document! LOL

When you say reading the manuals, did you spring for the Affinity workbooks? I have all three, though I've only cracked the Publisher workbook open. It is definitely a worthwhile purchase. Couple that with Elaine Giles' YouTube channel and we can become experts (or at least proficient) in short order.

Cheers,

 

Mitch

 

Hi Anklebuster, 

Regarding the three Affinity Publisher books, I didn't get around to purchasing the set yet. I do plan on getting a copy soon. And yes there is a lot of good information out there. These programs are packed with so many useful features.