Data Merge | Colecandoo!

Data Merge “Did you know” Part One

March 22, 2021 Leave a comment

Regulars to the site will know that many of my articles relate to InDesign’s Data Merge feature. Given the amount of tutorials already available online elsewhere concerning basic tutorials for Data Merge, the Colecandoo site focuses more on articles about Data Merge in relation to scripts, GREP styles, or advanced techniques.

But there is a middle-ground that hasn’t been covered in many Data Merge tutorials, nor here on Colecandoo, so over the next two articles, I will attempt to bridge that gap and highlight some lesser known issues that can become a problem if users aren’t initially aware of them.

Can’t package the data or links used in the data

When InDesign packages an INDD file, it will save a copy of the file and copy any links used in the document into a Links folder, and any fonts used (within licensing restrictions) to a Document fonts folder.

However, this does not extend to the source data of a Data Merge file, nor any links that the source data may refer to.

PDF made from merge is different to regular PDF

I have written about this before but ultimately when exporting a PDF directly from Data Merge, it makes a variety of PDF that is similar but not the same as a usual PDF, as the following options cannot be chosen.

The ability to merge to an interactive PDF
The page range (not the record range)
Spreads
Create Tagged PDF
Create Acrobat Layers
Hyperlinks

I’ve speculated why this might be the case in this article but until this addressed, it is a consideration to be aware of.

Headers with the same name

If the headers in a database are exactly the same name, InDesign’s Data Merge will add a sequential number after the first instance of the field name to make a distinction between the field names.

Colons can cause weird issues in the header

This featured briefly in my creative pro article “Troubleshooting data merge errors” but in short, colons used in field names can cause one of two dialog boxes when used in particular circumstances. Thankfully, if a colon appears at the start or end (or both) of a field name, the data will import without any issues, but if a colon is within the field, then a dialog box with the words “Generic extended parser error” appears.

If there are two or more colons in the field name (neither at the start or end of a field name), a dialog box that says “not well formed” appears.

UTF-16 is the format built for Data Merge

InDesign’s Data Merge is designed with UTF-16 text in mind. However, CSV and TXT files exported from programs such as Microsoft Excel usually export to UTF-8.

This is usually fine for most circumstance in English, but can cause problems when:

using an alphabet other than the Roman alphabet;
the data contains punctuation or characters that may not be available via UTF-8

Excel does have an option to export to UTF-16 and it is worth using. The option is here when exporting via Excel:

In part two of this Did You Know series, we will look at other lesser known phenomenon, such as:

Data is there even if link is missing
Merge fields can be removed via the hyperlinks panel
Shift clicking during the import does not show options
The benefits of linked images in the same folder as the source text file

If you have any lesser-known Data Merge behaviours that you think would easily make this list, please feel free to mention them in the comments.

Filed under Handover, Housekeeping, VDP Tagged with Adobe, Data Merge, InDesign, issues, troubleshooting

Data Merge to Single Records Pro: Now Available

November 30, 2019 1 Comment

Since 2016, Colecandoo has provided the free version of the Data Merge to Single Records script for Adobe InDesign – a script that allows single records to be exported from Data Merge with unique filenames available from the Data Merge database itself. This improves Adobe InDesign’s default – naming each file Untitled-N and is only available for InDesign files, not PDFs.

On that note, the PRO version of this script is now available!

This script improves upon the free original by:

Exporting to various additional file formats, such as interactive PDF, EPS, PNG, JPG, direct to print, or PDF via InDesign first;
Add a primary key to either the start or the end of a filename;
When exporting to certain file formats – the ability to run a user-selected additional script before the export.

The original Data Merge to Single Records script offered by Colecandoo remains free and can be downloaded from the scripts page.

Filed under PDFs, Scripts, VDP Tagged with Adobe InDesign, Data Merge, eps, indd, InDesign, interactive pdf, Javascript, jpg, pdf, png, script, variable, Variable Data, VDP

Of interest: New Data Merge techniques and quote bug

June 1, 2017 2 Comments

In the latest Colecandoo Youtube episode, four Data Merge specific features are covered, namely:

Adding faux-returns to a data merge field to split over lines, and subsequent limitations of this technique;
Using GREP styles to swap a character for a glyph during a Data Merge;
Highlighting field codes so that they are easier to see when not showing live data; and
A bug that occurs when a double-quote is at the start of any field in a Data Merge text file.

Faux returns within a field

The faux-returns technique is written about elsewhere, so rather than spoil their presentations, please read the articles directly from the appropriate sources:

I’m a fan of this trick, but emphasise that this is a workaround rather than a long-term solution, given that formatting is limited and there are more appropriate ways of accomplishing this task such as dedicated plug-ins or an XML workflow.

Swap characters for glyphs

Daniel Solis also features a clever trick to swap phrases with glyphs during a Data Merge that uses both GREP styles and ligatures. Again, rather than simply repeat the technique, please see his original video here.

A similar method can be employed using Indiscripts’ Indyfont script, but rather than swapping phrases with glyphs, will swap single characters.

Highlight Field Codes

The video also shows a method for highlighting field-codes when Data Merge is not in the preview mode. It relies on the [Basic Paragraph] style using a GREP style that contains a large highlight, and that any other styles in the document are based off of the [Basic Paragraph] style. It also means the document has to be styled correctly.

I’d demonstrated this technique following a real-world example of a live file where fields were very hard to see, and the file had to have an offset shell printed with variable data printed afterwards, so making sure the shell had no variable data on it was crucial. Using this technique would make finding the field codes much easier to see.

Double Quote bug

Also featured in the video was an issue that once again arose from a real-world example where a customer had provided a database that had double-quotes at the beginning of fields, but no closing quotes, resulting in rather unusual results.

Filed under GREP, Scripts, VDP Tagged with Data Merge, GREP, InDesign, Javascript, script, Variable Data, VDP

Better Infographics for Data Merge with Chartwell Bars

June 26, 2016 1 Comment

While speaking at the 2016 PEPCON in San Diego along with Co-presenter David Creamer on the topic of Data Publishing, I presented an older tip that allows shapes to change size based on numerical values that appear in Data Merge. The tip requires the Chartwell typeface, particularly the Chartwell bars font. I’d mentioned at the time that while it was a novel tip, I didn’t have a practical purpose for it. I’d also mentioned in my presentation about using knockout groups in InDesign to hide information and had demonstrated it using my “Parkway Drive” demonstration where it is used to hide parts of a sign that changes size, but again felt there should be a better use of this tip.

However, it was on my 15 hour flight from Los Angeles to Melbourne where I thought of a new and much more practical purpose – creating infographics. I also thought about getting some sleep, but that was a fool’s errand!

Once I arrived home, I tested out the theories I had during the flight, and while the results were mixed, I was happy with what had been achieved.

Ultimately, I have created three techniques for anyone making infographics. In all instances, I’ve colored the chartwell bars font as black so that the technique can be demonstrated, but in application the type (and its spacer) would be given the color “none”:

1 – Infographics as scaleable shapes.

This uses the method described in an earlier indesignsecrets.com article that I have written. Rather than rewrite the tip, the link to that article is here. The point of difference is that the shape being transformed into an infographic is what is being scaled.

There are some drawbacks to this method.

First, the shape has to allow the chartwell bars font to expand from the left to the right without getting caught on any part of the shape, so not every shape will work. Bottles that were used in the example were fine because they meet this criteria.

Second, there is a lack of precision, especially concerning low numbers as the graphic scales. This appears to be because there is a minimum size that the graphic can shrink to.

2 – Data that is pasted into a vector

This method works the same way with the exception that the data is in a rectangular shaped textframe that is pasted into the target shape, and also given a 2mm spacer object to allow low figures to be presented. The 2mm spacer is a 2mm square that is an inline object before the figures in the chartwell font.

For anyone wondering why such an odd technique was used to add 2mm to the frame, I had tried using a 2mm left align or a 2mm inset space in the shape itself but these presented issues.

3 – Hiding an image underneath

This works the same way as method 2 with the exception that rather than being pasted into the graphic, it is pasted above the graphic. An additional anchored object that is larger than the infographic is then pasted after the figures in chartwell bars and given very specific values in the anchored object dialog box, along with being given the fill color of paper and a multiply effect of 0% from the effects panel. The frame with the values is then grouped with the infographic that is underneath and the “knockout group” checkbox is ticked.

To make the effect more impressive, an “after” graphic is added that is the same size as the infographic but has different properties to make the difference in the values clear to the reader.

Moving forward

By itself, these techniques aren’t that impressive if creating one-off graphics, but if preparing infographics for variable data (whether for a catalogue or direct mail) I’m sure that readers will find these methods quite useful. These are not the only infographics tricks I have recently discovered, so watch this space.