Data Merge “Did You Know” Part Two

This is the second article in the Data Merge “Did You Know” series. If you’ve not read the first article, be sure to do so here. Carrying on from the first article, let’s dive into more lesser-known Data Merge behaviours of Adobe InDesign.

Data is there… even if the link is missing

If you ever have to share a Data Merge file with anyone else but do not want to share the data with them but instead only give them the base InDesign file, note that simply removing the link doesn’t remove the data.

In Part One, I wrote that InDesign doesn’t package the source file of a Data Merge… but that doesn’t mean the data isn’t there. Take this example of a packaged InDesign file used for a Data Merge. While opening the document, there is a missing link warning.

Once the document opens, I can then see in the links panel that there is a warning next to this text file that was used as the source.

If I right click on the link, I can’t unembed the text file as it simply isn’t linked

However, if I go to the Data Merge panel and check the preview box, I can see the records that were in the unlinked text file.

If the file didn’t have fields added to the page, I can also add those fields to the page and check the preview button on and the data will appear for these records.

It also merges correctly to PDF and InDesign files.

Be aware of this if you ever have to package a Data Merge file to others whom you do not wish to provide unredacted data.

Merge fields can be removed via the hyperlinks panel

A Creative Pro article referred to this as “ghost hyperlinks” but it is a great way of solving issues where a newly provided data merge source file can’t be previewed because of a mismatch of source names.

By opening the Hyperlinks panel, it is possible to see the fields that InDesign is using for the data merge as they are within this panel, though they aren’t obvious at first glance.

If one of the hyperlinks is double clicked, it will reveal the field that is being referred to.

From here, a hyperlink can removed, thereby replacing the field codes in the document back to regular text.

Shift clicking during the import does not show options

If the show import options checkbox is toggled off when placing an image, it is possible to perform a “one-time” request to show the import options without clicking on the checkbox. This is done by holding shift and then clicking Open.

But this doesn’t work with Data Merge. A similar option is available when selecting a data source, though holding shift and clicking Open will simply open the document – the Show Import Options checkbox has to be checked if it needs to appear. Hopefully this is a bug that is eventually fixed.

Put linked images in the same folder as the source text file…

While it is possible to add images to a data merge project by supplying its link in the source file, I so often see users put the complete file path of the link being used in the field.

If the images are filed in the same location as the text file, the only item that needs to be added here is the name of the file.

However, this means the links need to be in the same folder as the source text file.

…or use relative syntax as well

It is also possible to use syntax that is relative to the folder where the database is. Take the following folder structure.

To link to these images, the database needs to use syntax for the previous folder and then folders above. That will look like this

..: is syntax for go back a folder, whereas / plus the folder’s name is syntax to look into that particular folder.

I hope you found this short series useful, and if you have any Data Merge “did you know” tips, please feel free to submit them either in the comments, or contact me via my contact page.

Data Merge “Did you know” Part One

Regulars to the site will know that many of my articles relate to InDesign’s Data Merge feature. Given the amount of tutorials already available online elsewhere concerning basic tutorials for Data Merge, the Colecandoo site focuses more on articles about Data Merge in relation to scripts, GREP styles, or advanced techniques.

But there is a middle-ground that hasn’t been covered in many Data Merge tutorials, nor here on Colecandoo, so over the next two articles, I will attempt to bridge that gap and highlight some lesser known issues that can become a problem if users aren’t initially aware of them.

Can’t package the data or links used in the data

When InDesign packages an INDD file, it will save a copy of the file and copy any links used in the document into a Links folder, and any fonts used (within licensing restrictions) to a Document fonts folder.

However, this does not extend to the source data of a Data Merge file, nor any links that the source data may refer to.

PDF made from merge is different to regular PDF

I have written about this before but ultimately when exporting a PDF directly from Data Merge, it makes a variety of PDF that is similar but not the same as a usual PDF, as the following options cannot be chosen.

  • The ability to merge to an interactive PDF
  • The page range (not the record range)
  • Spreads
  • Create Tagged PDF
  • Create Acrobat Layers
  • Hyperlinks

I’ve speculated why this might be the case in this article but until this addressed, it is a consideration to be aware of.

Headers with the same name

If the headers in a database are exactly the same name, InDesign’s Data Merge will add a sequential number after the first instance of the field name to make a distinction between the field names.

Colons can cause weird issues in the header

This featured briefly in my creative pro article “Troubleshooting data merge errors” but in short, colons used in field names can cause one of two dialog boxes when used in particular circumstances. Thankfully, if a colon appears at the start or end (or both) of a field name, the data will import without any issues, but if a colon is within the field, then a dialog box with the words “Generic extended parser error” appears.

If there are two or more colons in the field name (neither at the start or end of a field name), a dialog box that says “not well formed” appears.

UTF-16 is the format built for Data Merge

InDesign’s Data Merge is designed with UTF-16 text in mind. However, CSV and TXT files exported from programs such as Microsoft Excel usually export to UTF-8.

This is usually fine for most circumstance in English, but can cause problems when:

  • using an alphabet other than the Roman alphabet;
  • the data contains punctuation or characters that may not be available via UTF-8

Excel does have an option to export to UTF-16 and it is worth using. The option is here when exporting via Excel:

In part two of this Did You Know series, we will look at other lesser known phenomenon, such as:

  • Data is there even if link is missing
  • Merge fields can be removed via the hyperlinks panel
  • Shift clicking during the import does not show options
  • The benefits of linked images in the same folder as the source text file

If you have any lesser-known Data Merge behaviours that you think would easily make this list, please feel free to mention them in the comments.

Export many PDFs at once… plus security

A recent question on Reddit’s InDesign subreddit was whether two PDFs could be exported at the same time from the same document, but have two different properties – one with trims and one without. The answer is yes, but via a custom script written for the task.

I use such a script on a daily basis so that I can prepare a PDF for client proofing via email; and a separate PDF that has trim and crops that is sent directly to a hot-folder that prints it for me.

I’d submitted my script as a solution (that can be downloaded from the scripts page), but then realised that this concept was not a new idea. Ariel Walden over at ID-Extras had already written a similar script within a blog post of his own.

Similarly, Peter Kahrel’s Batch Convert script can perform the same task, with the added advantage that it can also do this for all open InDesign documents;

Or if no documents are open, a specified folder (and subfolders if desired) of InDesign files.

Can’t make these secure

One feature that all three scripts have in common is that the exports are based on the PDF presets available on the user’s machine. One feature that can’t be added to a PDF preset is security – this can only be done when a request to export the document is made, as security settings aren’t saved into PDF presets.

This is a problem if there are lots of documents that need to be exported with security settings as it requires the user to enter the security details each time a PDF is exported.

I’ve made an additional script

For this purpose, I thought I would make a script that not only makes several PDFs, but can also add password security to one version. The script can be downloaded from the scripts page.

When the script is run, it will generate two PDFs using different PDF export settings, but one will have the suffix “_secure” added to the filename, and a dialog box will appear once the export is finished:


The script can also be adjusted by opening the script in any text editing application and making the necessary changes, such as.

Use the same password for every document

Look for the line

    openDocumentPassword = myPassOpen; // requires a password to open the document

and change the myPassOpen to the desired password in quotations. For example:

    openDocumentPassword = "OpenSesame"; // requires a password to open the document

Similarly, do the same thing for the line underneath, making sure that the open password and edit password are not the same.

    changeSecurityPassword = myPassWrite; // requires a password to change the document

change to

    changeSecurityPassword = "EditSesame"; // requires a password to change the document

then search for the lines;

and swap the forward slashes in the lines around so that the lines now read like this.


Only require a password to edit the document

Look for the following line:

    openDocumentPassword = myPassOpen; // requires a password to open the document

and add two forward slashes to the start of the line.

//    openDocumentPassword = myPassOpen; // requires a password to open the document

Adding two forward slashes to a line in a javascript tells the script to ignore the rest of the line and go to the next line of code.

Don’t show the “done” message

The default script has a dialog at the end for showing what the opening and editing passwords are, but if you want to edit the script so it makes a PDF that applies security to edit the document but does not provide the password (e.g. for the purpose of handing PDFs over to parties who may seek to deconstruct them in other applications) then make the adjustment mentioned a moment ago to restrict passwording to editing only, and then search for the lines;

and swap the forward slashes in the lines around so that the lines now read like this.


Add more PDF exports

Look for the line

app.activeDocument.exportFile(ExportFormat.pdfType, File(resultsFolder + "/" +".indd")[0] + ".pdf"), false, "[High Quality Print]");

make a copy of the line and make the appropriate changes:

  • Replace the “[High Quality Print]” to the desired PDF preset exactly as it is written in the PDF export dialog box and put it in quotes. For example, if your PDF preset is called My Export then type “My Export”
  • Replace the “.pdf” with a suffix that denotes that this is an additional PDF. For example, if the pdf is a high res print, perhaps replace this with “_hi-res.pdf” so that the resulting file has _hi-res.pdf at the end of its filename.

Otherwise if you are after specific changes to the script to suit your needs, contact me via the contact page.

Things to know about the script

Opening and editing passwords must be different

One condition of preparing a secure PDF from Adobe InDesign is that the password required to open the PDF must be different to the password to edit the PDF, so if editing the script to replace the randomly generated password to a known one, the opening and editing passwords must be different. If the passwords are the same, the PDF will be made without security.

PDF Standard in the preset must be set to “None”

PDFs that use a PDFX standards can’t have security applied to them as the security panel of the PDF export box is greyed out, preventing security to be applied. The standards dropdown box in the desired PDF preset must be set to None.

Only password security is applied

When exporting a PDF from InDesign, only password security can be applied, unlike Adobe Acrobat’s choices of security that it can offer (as shown below).

While password security may deter or prevent a layperson from editing the PDF, the security can be broken through some effort. Several websites offer services where users can drag and drop a PDF to the site, and within moments the PDF will have the PDF password removed.

Similarly, there are desktop applications that can also be purchased to remove the security (as one of their many features), such as PDFsam Visual.

Outlining the problem… text outlining

From time to time, I will prepare PDF artwork for third party providers and then note that their specifications indicate “Convert all text to outlines” (also known as converting to curves or paths). But why do some third parties recommend this practice?

The PDF is opened in software other than Acrobat

For commercial printers, PDFs are usually imported into Raster Image Processing (RIP) software that will impose and trap the artwork for their printing methods. However, not all providers work this way and may need to open the PDF in applications other than Adobe Acrobat. For example, a third party that prepares cutting formes may open the file in Corel Draw or a CAD application that supports its CNC software.

This means that as the file opens, the application may ask for fonts not available to the third party.

This can be exacerbated if the PDF is opened not only in a different application than Adobe Acrobat, but also a different alphabet and writing system. Converting the type to outlines maintains the appearance of the type without requiring the font to be present.

Other reasons that text is converted to outlines

So special effects can be applied

InDesign, Illustrator and Photoshop can apply interesting special effects to vector objects, but not all of those effects can be applied to live type. The solution is to convert the type to outlines, thus converting the type to vector shapes that can have the desired effect applied.

To prevent editing by third parties

Limited editing is possible within PDFs using either Acrobat’s own editing tools or using plugins such as Enfocus Pitstop Professional. These tools can allow last minute alterations to text so long as the text is type and not converted to outlines.

Locking the PDF with password protection isn’t an option as this can prevent the file from being placed into layout software or RIP software for output, so the password is then required to unlock the file. PDF password protection is also somewhat breakable, with many websites offering services where PDFs can be uploaded, and then unlocked and then downloaded without the password protection. There are also PDF editing and viewing applications such as PDF Sam that allow for decryption of PDFs.

Even without the Enfocus Pitstop plug-in, it is possible to open PDFs in Adobe Illustrator or Affinity Publisher and then – if the fonts are available – make the necessary alterations… though converting type to outlines will prevent this.

To circumvent the font EULA

A client may have acquired a font that has allowed for screen use only and prohibits embedding in a PDF, preventing the font from appearing correctly in the PDF. A way around this is to convert the type to outlines in the native application prior to PDF export, though it is worth noting that the End User Licence Agreement (EULA) of the font may forbid this workaround, so it is worth reading the font EULA.

That doesn’t mean it should be done!

There are issues that arise from converting type to outlines. Dov Isaacs – Principal Scientist for Adobe Systems – has a brilliant PDF that details this (and much more) but the basic takeaways concerning type to outlines are:

  • Increased filesize that takes forever to download or view onscreen
  • Smaller typefaces do not render as well
  • May potentially breach the font’s EULA

In addition, there are other issues such as:

  • Potential issues with fonts where type overlaps itself (it can knock out holes in the joins)
  • If the conversion from type to outlines has been done in the native application and then accidentally saved and closed, this means the type will no longer be live in the native application.
  • It can prevent or hinder minor type alterations being made in a PDF submitted for print.
  • Text (as outlines) that has special effects applied (as described earlier) may not always be able to have the same effect applied to live type. This can create issues with variable data campaigns where the effect needs to be applied to a text variable.
  • It can make it difficult to identify the font used, as the font’s information is no longer in the PDF and the only other way to identify the font is visually or with apps such as what the font, adobe capture, or identifont.
  • The conversion is usually a one-way conversion. There is a fantastic Adobe Illustrator plug-in from Astute Graphics called Vector First Aid 2 that – in some circumstances – can convert outlines back to type, but it isn’t a magic bullet (though definitely worth a look).

If your hand is forced…

In a perfect world, I’d only deal with providers that fully supported PDF/X-4 files. Unfortunately, not all providers do, and occasionally our hands will be forced into providing PDFs specifically as the provider has requested, which may mean converting text to outlines. Rather than doing this in the native application (e.g. InDesign or Illustrator) there is a great way to quickly convert all type to outlines using an Adobe Acrobat Preflight that is detailed over at CreativePro.

%d bloggers like this: