Data Merge: Multiple Record Madness

There have been plenty of posts on the Adobe InDesign Forum lately concerning issues that users are having with data merge, particularly Multiple Record Data Merge (MRDM) projects. These projects could be “stepping up” artwork onto a larger sheet for trimming (e.g. imposing many business cards onto one sheet) or preparing catalogues for example.

Before rushing into an FAQ of issues concerning MRDM, here is a step-by-step of preparing a MRDM to ensure the minimum of fuss:

1) In a new document, go to the Master page and create the static items that are to appear on every page.


2) Create a new master page BASED ON the master page created in step 2 and in this page, add the items that will be variable, but prepare the set up as if it were for one record only.

(for those familiar with Data Merge, the variable can indeed be on a regular page. The difference is that MRDMs created with variable items on a regular page will not allow the source data to be linked)



3) From the Data Merge palette, choose “Select Data Source” and select the text file that will be used for the merge.

4) Populate the variable placeholders using the fields from the Data Merge Palette, being sure to that the placeholders of the text are visible in their own frames. Once satisfied with its appearance, select “Create Merged Document”


5) A new dialog box appears. In the tab “Records”, make sure that the Records per Document Page dropdown says Multiple Records. Then from the tab “Multiple Record Layout”, set the appropriate margins of the artwork. If using the preview, note that the position of the first record may have moved – this is normal. Change the margins to the margins that were in the static layer, and layout the records as appropriate. Click OK once ready to proceed



6) A progress dialog will appear. After a moment the document will be created either with or without an overset text warning.



There are several things that should be noted:

  • The margins in the InDesign file are irrelevant for where the variable data starts – this is determined by the Data Merge Panel;
  • That during the construction of the file, it is only possible to view the data 1-up. To preview multiple records, this can only be done from the  Create Merged Document from the Data Merge panel.

Common complaints when preparing a Multiple Record Data Merges:

Each page has the same record repeated, so page 1 has record 1 many times, page 2 has record 2 many times, etc.

Likely that many variable placeholders were created and populated thereby filling the page, instead of one series of placeholders for one record only. MRDM works by creating one record and then using the MRDM panel to allow the next records to be inserted based upon details here (e.g. distance between the records, direction of flow of the records)

MRDM records aren’t merging where I want them to merge (e.g. off by half a millimetre or more, or off by miles)

Data merge uses the margins within the MRDM dialog box, NOT the margins in the active document. There is an additional glitch that offsets the starting position by fractions of a millimetre. This may seem insignificant but there is no reason that the software should not place the text to precise measurements.



Data merge has a further glitch if the document was initially created at one size but was then resized (regardless of orientation or size). When merging, the starting offset appears to be where the original size page margins would have been. The following example shows the same file but just made landscape.



This glitch persists despite further resizing, adjusting of margins, or saving as IDML and reopening. The only solution is to cut and paste from the defective document into a brand new document.

When MRDM merges to PDF, the first record does not appear, and instead of the records displaying one after the other, only the first record placeholders appear and is repeated through the document.


This only occurs when exporting directly to PDF. In this instance, the variable data portion has to be on a regular page, and any items that are to remain static need to be on the master page.



The MRDM preview is correct, but the resulting InDesign file has only one record per page. Once the view is changed to see the entire pasteboard, it is clear that the other records are on the pasteboard.



Likely because the variable and static data is on the same master page. Instead, the variable data portion has to be on a regular page, and any items that are to remain static need to be on the master page. If linking to the text is essential, the workaround is to put the background on a master, and then make a second master that is BASED ON that background and put the variable data on that layer. Refer to part 2 of the tutorial at the start of this post for more information.

Other commonly asked merging questions:

Upon selecting data to import, a dialog box says “The data source has one or more empty field names. Please fix the file or select another file”.


The header row has fields that don’t have names. Can happen if the field names have no headers – the header row has to have names in it. Can also happen if an excel file is saved as a txt/csv file but it saves more columns than required. These excess columns need to be deleted in excel before attempting to import the data again.

The text has picked up the formatting of the line underneath.

This has to do with the “Remove Blank Lines For Empty Fields” options in the Content Placement Options portion of the Data Merge palette. This issue has been discussed elsewhere on Colecandoo.

The records are all in their own frames, instead of one long flowing frame like this:


Data Merge works in this fashion, it doesn’t have a “next record” feature that Microsoft Word has in its “Mail Merge” feature. There is a workaround in the form of a script created by Loic Aigon that will do this; or by merging to a new file and then using a script such as Ajar production’s merge textframes that will thread the text so it run into one text frame, but if the data is likely to change then this procedure will have to run all over again. XML workflow is better for this kind of project.

How do I prepare a multiple record merge where records specific to one field are on the same sheet?


This can’t be done from InDesign “Off of the shelf”. Data Merge, whether single or multiple records, can only handle one to one database relationships. It can’t merge items such as itemised invoicing for a client database where the database contains varying record lengths for each client. This is an example of “one to many” database relationship. InDesign can only handle “one to one” database relationships.

How to I preserve carriage returns from my Excel database into Data Merge?

Can’t be done. Data Merge only works with txt or csv files, so ultimately the Excel file has to be saved as one of these formats. A return (whether soft or hard return) in the txt/csv file indicates the end of the record and the start of the next, whether intentional or not. One trick is to substitute the returns with a character that is unlikely to use in normal type (e.g. the “pipe” symbol = | ) and once merged into the file, use find/change to replace the | with a carriage return.

The drawback is that this method cannot be used if merging directly to a PDF as the intermediate step of removing the pipe symbols cannot be done. Another drawback is if the data file for the merge is replaced, the merge will have to be done again. XML is a better choice for this task as carriage returns, non-breaking spaces and other special characters can be preserved.

How do I remove unintentional line breaks (whether soft or hard returns) from my Excel database before using Data Merge?

Dozens of answers exist on the many Microsoft Excel forums that may/may not work within the Windows OS, but on the Mac OS there is no reliable answer that works within Excel itself. Anyone who can answer to this question should feel free to post it in the comments. Best answer so far was again from Loic Aigon’s Blog.

My data is not importing properly (e.g. Records are starting where they are not supposed to, characters are corrupting, etc)

Could be a variety of reasons depending on how the data was prepared, how the fields are separated (whether comma or tab). The data should be checked by finding the corrupted record in InDesign, then comparing the data file in a text editor such as UltraEdit or TextWrangler to see what is happening.

Breaking up is hard to do… InDesign files into individual PDFs that is!

Several forums dedicated to InDesign advice have recently been asked the following question: “How to split an InDesign file into single page PDFs”.

Splitting a large PDF into single page PDFs is possible via the extract pages feature of Acrobat 9 and up. The resulting pages are then extracted to the same file location as the original PDF but contains an underscore and page number in the filename.


Splitting a large PDF into fixed page lengths (e.g. singles, doubles etc) is possible via the Split Document feature of Acrobat X and up. This also provides limited control concerning the name and location of the resulting split PDFs, as well as other ways of splitting the PDF (e.g. filesize or bookmarks).


So splitting a large PDF into smaller PDFs is possible via Acrobat. However, the brief was “How to split an InDesign file into single page PDFs”.

By default, there is no way to do this directly from InDesign without a script. However, there are at least four scriptable solutions available as of this moment:

  1. Scripter Loic Aigon produced a script called Custom Export – an InDesign javascript that behaves in a similar fashion to the Split Document feature of Acrobat, but without leaving InDesign.
  2. (updated 5 November 2013) Scripter Dmitry Lapaev offers three scripted solutions, but of these there are two that will be of most use to those who intend to output to print: the first is Quick Export to Adobe PDF  (see this link here) and the next is Batch Export to PDF (see this link here).
  3. Yet another javascript with a more sophisticated interface is Scott Zanelli’s Page Exporter Utility that had been discussed on this blog before. Read more about the script here.
  4. Fellow wordpresser Macgrunt has also produced an applescript that allows the export of single page PDFs from one InDesign file. While it does not have a user interface, it certainly does the job. Read more about his script here and read his related blog posts concerning renaming of files.

UPDATE 2014/1/18: There are also paid solutions that can accomplish this task. One such solution is PDF Bee by Chris Paveglio. This has not been tested by Colecandoo, nor is this a paid endorsement.

So the question is answered… right? Yes and no. Yes, it is possible to export to individual or smaller page PDFs, but the naming of the files could be better.

Using an example of business cards that have been data-merged to a new InDesign file, the brief is now to produce PDFs with filenames that reflect the names of the people on the business cards. Using the earlier solutions, the files would still need to be renamed afterwards. So how is this done?


Loic has another script called PDF Export Cropper. This script is much more flexible than the previous scripts in that files can be split according to more variables, and the file naming is more flexible. To demonstrate, an example single sided business card has been created. The PDFs are to be named based upon the name of the person, so the field that holds the client’s name has been assigned a special paragraph style that is used nowhere else on the card – in this instance, the style “clientname” has been assigned.


The file is then merged to a new InDesign file. Once the new file is created, the PDF Export Cropper script is run.


A new user interface appears. In this example, within the “Choose Identifier” portion of the interface, the appropriate paragraph style has been chosen. The filename is to be the name of the resulting paragraph style, so all that is left to do is click Export.


And voila! The PDFs are split and named based on the client name that appears on the business card.

The only downside – that the cards can only be one page (that is, if the business cards were double-sided, Loic’s script would not work).

UPDATE 2014-01-14: Loic’s scripts are currently being revised and at the time of writing this update are unavailable. See his post here.

SOLUTION TWO: Hans Haesler

German Swiss scripter Hans Haesler has a similar script to Loic’s script. Sadly for me, it is in German, so I can’t understand the user interface that the script creates. A link to the script and a brief how-to-use for Anglophones is available here.

SOLUTION THREE: Via Adobe Acrobat

Unlike the previous two solutions, this solution requires the data merge file to be merged to one large PDF. It also requires that the field from the database that contains the names to become the future filenames is called PartnerHQ_Id. From here, an Acrobat action has to be run. The action is available from here:

First, the action will have to be loaded by opening Acrobat, select the File Menu, Action Wizard, Edit Actions. From the new window, select Import and then navigate to the downloaded split files.sequ file.


Next, create a new folder and copy the PDF to merge and the csv or txt file that was used for the data merge into that folder.


Once this is done, open the PDF to be split using Adobe Acrobat and from the File Menu, select Action Wizard, split file


A new window will open, and the file that is already opened should be listed in the window. Click Next.


The actual javascript to be run will open as a window. Click OK.


At this stage, the script presents an error. Click Close.


Voila! The PDFs are renamed based on the client names. The folder also contains the original PDF and the database files.


So there are at least three solutions to this brief.

I’m gonna knock you out, my printer didn’t knock you out…

An earlier post “To Overprint or not to Overprint, Black is the question” explains how the colour labelled [Black] in InDesign behaves, and when solid black ink should and should not knock out of the colours behind it.

Paying attention to this advice and applying it to artwork should result in a good printed reproduction, correct? While the answer should be yes, there is one more level of control of black appearance and overprints, and that is in the hands of the printing company and their output software.


Let us look at this following example:


This card is set up for a Black plus spot output for an offset press. The Black is only overprinting on the text as misregistration would be noticeable here, but the Black elsewhere is knocking out so that the colour does not look muted through the yellow.

However, despite best intentions and checking the separations both in InDesign and Acrobat, the card has printed like this (effect is exaggerated for the screen):


So what has happened? The separations were correct, they were checked in both InDesign and Acrobat! It turns out that the Raster Image Processor (RIP) software that the commercial printer uses to image the design onto the printing plates has its own settings. Here are some example screenshots from AGFA’s Apogee X system and Fuji’s XMF system respectively about the overprinting of black:



In both screenshots above, the respective RIP software CAN honor the settings that were in the initial PDF and not apply its own preferences, but in the instance of the business card, the RIP settings overrode the PDF settings and chose to overprint all instances of 100% black, regardless what swatches were chosen in InDesign.


Using the same artwork, the card was printed via a colour copier, but this time the result was as follows:


So what happened here? The while the solid black looks good, where the black in the top line meets the vignette looks rather weak, and there are is a lighter black around the travel agent. What is going on?

Again, the RIP software has manipulated the artwork with unintentional results. Unlike printing directly to a desktop printer, most digital printers will print to a RIP where the file can be imposed, colour adjusted and printed in whatever order the prepress operator sees fit.

Using the EFI Fiery RIP, there is a little-known feature of the RIP that changes the way black is displayed that can produce unexpected results, and that is in the color settings dialog box and it is “Pure Black On”.

Screen Shot 2013-06-03 at 2.55.28 PM

This setting takes every instance of 100K and ramps the colour to a “super black” as opposed to using the black toner only. Again, this setting can be changed, but when this card was printed, the defaults were unchanged resulting in this unwanted appearance.

This setting only applies to vectors and text AFTER the PDF is flattened into postscript. This is visible where the rich black abruptly changes to the muted black. One look at the flattener preview in InDesign confirms that areas of flat black in that image were as a result of the flattening.



This small example shows how changing the client’s intended black overprints can have unwanted consequences. For prepress operators it is a wake-up call to make sure that the RIP defaults will maintain the clients’ expected results; and for designers or publishers it is worth understanding that even the treatment of black overprint is an important topic.

Adding pages features to Acrobat

The pages panel of Acrobat has many features such as rotate, extract, crop and split, but is missing several features that many users would appreciate as standard:

  • Break readers spreads
  • Break printers pairs
  • Reverse page order
  • Interleave two files
  • Duplicate pages in a 1-1-1-1-1-1-2-2-2-2-2-2 sequence
  • Duplicate pages in a 1-2-1-2-1-2 sequence

Breaking spreads and pairs

Breaking readers spreads has been covered on this blog several times so won’t be discussed in this post, but breaking printers pairs has proved to be a little more challenging and an Acrobat-based solution remains to be found.

Reversing and interleaving

Solutions have however been found for the last four of the bullet points. One solution solves two of the missing page features, namely reversing page order; and interleaving two files. The script can be found in post 8 on the following thread:

Once added to the javascripts folder of Acrobat, two new items are added to the Edit menu: Collate and Reverse. The collate option allows another file to be interleaved into the already open PDF. The Reverse function simply reverses the page order of the already open PDF.

An alternative solution for users who want to reverse order lots of PDFs at once may want to make an Acrobat action. From the file menu, select Action Wizard, and Create New Action.


In the new window, click on the “More Tools” panel in the left hand column and choose “Execute Javascript”. The window that should now look similar to this, click on the “Options” button within the Execute Javascript step and type the following in the next window that appears:

for (i = this.numPages - 1; i >= 0; i--) this.movePage(i);


From here, click Save. A prompt will appear asking for a name and description of the action.

Once an action has been made, it can be called upon by navigating to the File menu, Action Wizard, and then selecting the action.


Actions made in this fashion will prompt for a file or files. Once selected, the action will then run and prompt for a save location.

Duplicate pages in a 1-1-1-1-1-1-2-2-2-2-2-2 sequence

Again, this solution is provided in the form of an Acrobat action. It effectively follows the same steps as creating the reversing action but with one important difference – to tick the checkbox “Prompt User”. The script for this action can be found here:


When this action is run, the usual prompt for a file will appear. However, there will be a pause where the javascript console will once again appear. It is at this point the amount of duplicates are typed in.


Duplicate pages in a 1-2-1-2-1-2 sequence

This solution is almost identical to the solution above, except that the script for the action is different. The script can be found in the following forum post:

Other solutions

The action wizard feature of Acrobat is often overlooked. Apart from running custom javascripts, the wizard allows general Acrobat functions such as extract images as tiff, or recognise OCR text to run. Prior to the release of Acrobat XI reader, anyone using Acrobat X reader could not comment or mark-up a PDF unless the file had been specifically saved from Acrobat with commenting enabled. Doing this via the menus is tedious, especially if there are many files enable, but using an action this was quite simple. It was also simple to set the initial views so that a client will see the PDF to fit height, as spreads with the cover turned on.

Having these options available while making the PDF from InDesign would solve this issue permanently… perhaps in future updates of InDesign CC.

Anyone interested in looking for more script based actions may want to have a look at this site and this site.


Get every new post delivered to your Inbox.

Join 56 other followers

%d bloggers like this: