Drupal 7

Getting content into Drupal with Migrate : Montréal (v.0.4)

This year I'm presenting on Migrate at Montréal DrupalCamp! It will be the best one so far! I refactored the presentation yet again.

Recently I have published a new module, migrate_webform, that is a good example of the items in the talk. I also refactored commerce_migrate_ubercart to work with migrate version 2.6, which is the one with the amazing new UI config.

My company, Kafei Interactive, is a sponsor this year. We also sponsored Toronto DrupalCamp. I welcome Git tip if you want to buy me a coffee.

File migrate_montreal.odp600.7 KB
PDF icon migrate_montreal.pdf584.13 KB

Getting content into Drupal with the Migrate module (Vancouver v.0.3)

This is the third version of my migration presentation, now with much more code samples and information on how to use migrate_d2d. This time it was presented in Vancouver for the Pacific Northwest Drupal Summit.

Update: on slide 24 migration.inc file should be "kafei.migrate.inc".

Migrate all the things!

This is a presentation I did for the 2013 Toronto DrupalCamp.

Migrate allows you to bring content into your new Drupal site from a variety of data sources including past versions of Drupal. In this session we will talk about why you should consider using Migrate over other methods such as upgrading Drupal, using feeds, node export, copy and paste, etc. We will look at what data sources can be imported, how to create migration classes and where to find starter templates you can use to build your own custom migrations.

At the end of the presentation we walked through my previous blog post for code examples and more detail. http://www.verbosity.ca/working-drupals-migrate-module

EDIT: the prepareRow and prepare functions had their names inverted on the slide titles.

Working with Drupal's Migrate module

UPDATE: if you're interested in this article you should check out the slides from my 4th migrate presentation. The registration system in migrate has changed ("automatic" registration is gone) and I've explained it all in better detail in the slides. I even cover the pathway of the variables if you're new to OOP. ...and yes, I'm available for support and/or workshops.

I have had the opportunity to work on two Drupal sites that involved the Migrate module. I have in the past worked fairly extensively with the Feeds module to do this type of thing. Feeds is a great and powerful module, but at a certain point you will probably find that you want to use Migrate.

Why use Migrate rather than Feeds?

  • Rollbacks. The migrate module makes it damn easy to rollback and re-import.
  • Rules in code. Any changes you do to your input file are stored in a coded template.

The downside as you have probably already guessed is that you're going to spend a lot of time writing custom code here. On the bright side, there isn't that much complicated stuff once you know what everything is trying to do.

Example Migrate Templates

These are the ways that do not involve using much code. The reality is that you *will* be doing code with almost any Migrate project. Before you get started the best things you can do are (1) read the manual on Drupal.org, and (2) find a reference template that is close to what you want to do.

There is one additional thing... the beer and the wine templates. These are part of the migrate_example package that is part of the main migrate module. Review these, the code is commented specifically for this purpose.

Also check out the following as possible starting points:

Additionally, there is a Wordpress conversion template, and ones for Typo3 and PhpBB. The Wordpress one has a UI and works quite flawlessly (though you will need to remap your image locations somehow).

Migrate and the Future of Drupal

Some people have suggested using Migrate as the main way to upgrade between versions of Drupal. My guess is that my team is going to use it a lot for D7->D8 migrations when the time comes, and I suspect that it will be the default way for D9.

Setting Up Your Dev Environment and Migrate Basics

One thing about Migrate that you should know is that some of the queries that will run will be enormously expensive processor-wise. Especially when things aren't working correctly. I find it is best to work from a local environment rather than a dev server until you have a working template. Additionally, it helps to have a process monitor showing the activity on the machine so you can see when things go wrong.

Another tip that you will need to know is the command drush_print_r($variable); in your PHP code that will help you to analyze the objects referenced in your code and to see what structure Drupal is expecting when you put stuff into $row->something; (prepareRow) or $entity->field_name[][0]['value']; (prepare).

Familiarize yourself with the following commands, as it is recommend to use drush to do your migration:

drush ms
drush mi $migration
drush mr $migration

You will probably need some debugging tools as well:

drush_print_r($var); - this command does what you would expect - dumps a variable to the command line. You put this in your PHP code.

drush_print_r($query->__toString()); - this will dump the MySQL query that is generated by Drupal's dynamic query engine. If you have one template that won't run, this can come in handy. You can adapt the output of this to run in MySQL directly - useful if you need to confirm that the query is causing memory problems. You can prepend the keyword EXPLAIN to the query generated for additional info from MySQL.

Anatomy of a Migrate template

The migrate module needs some basic definitions so that it will show up on the Migrate page, which is a tab on the Content list page. Most of the time "registration" of the migration classes is "automatic" but I have found that it is useful to use the tab on that page to force it to look for migrations.

In the case of Commerce Ubercart Migration module, there is also a settings tab which will appear for that. You will want to configure it to connect to the existing data set and it will create a bunch of "migrations" to show on the main Migrate page.

When your migrations are all listed, you can click one and see what fields are mapped and which are not. This is where you're going to spend a lot of your migrate time.

Typically you will have a custom module with the following:

  • custom.info file
  • custom.module file, blank... unless you want to use it for something*
    • *an optional API definition if not using automatic-registration
  • something.inc, your first migration, listed in your files in the info file
  • another.inc, your second migration, also listed in files of info file

Within your .inc files you will need to know about the following components which are almost always going to be in your migration:

  • class CommerceMigrateUbercartNodeMigration extends DynamicMigration class definition setting what migration will be used as a base. Then the following sub-components:
    • public function __construct(array $arguments); // initialize the migration, connect to source and grab the settings! Contains a reference to what will be used to map the migration (keeps track of changes to IDs)
    • public function prepareRow($row); // preprocess function, DOES NOT RUN FOR XML IMPORTS
    • function prepare($entity, stdClass $row); // last chance to do stuff. Can run SQL queries here, first param is always the target node/order/entity fully expanded, so put in your languages setting and delta values!
    • function complete($entity, stdClass $row); // rarely needed if you did things right. Can be used for doing post-migration tasks but could complicate your rollbacks, so be sure to test!

Initializing the Migration

With Migration templates you're dealing with Object Oriented Programming, not a staple in our Drupal PHP diet yet. If you're familiar with OOP you'll be right at home. Let's start out with a definition:

class CommerceMigrateUbercartNodeMigration extends DynamicMigration {
  // An array mapping D6 format names to this D7 databases formats.
  public $filter_format_mapping = array();

  public function __construct(array $arguments) {
    $this->arguments = $arguments;
    $this->description = t('Import product nodes from Ubercart.');
    $dependency_name = 'CommerceMigrateUbercartProduct' . ucfirst($this->arguments['type']);
    $this->dependencies = array($dependency_name);

    // Create a map object for tracking the relationships between source rows
    $this->map = new MigrateSQLMap($this->machineName,
        'nid' => array(
          'type' => 'int',
          'unsigned' => TRUE,
          'not null' => TRUE,
          'description' => 'Ubercart node ID',

There is a lot of stuff going on here. Firstly, we're extending a base class. That means this will apply to any objects of that class. So we can just assume that if the base class Migrate is running, that this class will run as well. What follows is a constructor, which what is going to run at startup.

The last part of this code block is the most interesting: $this->map. Here is how Drupal is going to track the objects in your migration. It is your key for each item in your import - and it is going to BREAK if you are trying to use XML as your import source. Then you're going to have to dig around in your DB and change that type from int to something else.

Connecting to Source(s)

The most exciting part about this part of the migration template is that you have many options. In this case we're looking at a D6->D7 migration, so in our case D6 is our source! Migrate takes care of the actual connection to the database so that part is easy, you can point it right at production to grab the data. As noted above, XML can also be your source(s). When I say source(s) I mean possibly plural... the example code provided will let you cycle through many XML files during the import process. Potentially really handy if your exports were run in batches.

Note, that the switch statement in the start of this module is related to the UI of Migrate, where the author of this module has made a switch available. Most migrations don't provide a settings form for Migrate but this one does.

    // Create a MigrateSource object, which manages retrieving the input data.
    $connection = commerce_migrate_ubercart_get_source_connection();

    $query = $connection->select('node', 'n');

      case 'd7':
        $query->leftJoin('field_data_body', 'fdb', 'n.nid = fdb.entity_id AND n.vid = revision_id');
        $query->leftJoin('field_data_field_campaign_id', 'cid', 'n.nid = cid.entity_id');
        $query->fields('n', array('nid', 'type', 'title', 'created', 'changed'))
              ->fields('fdb', array('body_value', 'body_summary', 'body_format', 'language'))
              ->fields('cid', array('field_campaign_id_value'))
              ->condition('n.type', $arguments['type'])
      case 'd6':
        $query->leftJoin('node_revisions', 'nr', 'n.nid = nr.nid AND n.vid = nr.vid');
        $query->leftJoin('filter_formats', 'ff', 'nr.format = ff.format');
        // ... and so on ...
    $this->source = new MigrateSourceSQL($query, array(), NULL, array('map_joinable' => FALSE, 'skip_count' => TRUE)); // connect!

You probably noted the D7 and D6 options have different mappings. Take note. If you're going to extend one, do the one you intend to use.

When your main Migrate page stops working...

Eventually if you start adding fields your query will silently break, and will force Migrate's listing page to hang. Usually you can still access the page for the specific migration but since the purpose of these pages is to help guide your clients through the process this is a severe deal-breaker. Fortunately, it is easily solved one of two different ways. Either (1) don't bother counting how many tasks are to be done or (2) specify a basic query that can be tabulated quickly.

Here is how you do that:

$this->source = new MigrateSourceSQL($query, array(), NULL, array('map_joinable' => FALSE, 'skip_count' => TRUE));

In your source you specify skip_count as a parameter and you set it to TRUE. Now you're going to have an X where the number of items to be migrated was... but you can at least get to that page. Whew...

Adding each of your fields

This is where start getting fun... or really boring. Your pick. We now have to go through each field that will be brought into the database and create a relationship with the place where we want it to go.

Each field in your migration is going to have the following components:

  • A field in your target content type to hold the incoming data
  • An extension to the $query object to perform a LEFT JOIN if your source is in a table that is not in the base query
  • An extension to the $query object's fields to SELECT by column name
  • An extension to $this to connect the old field to the new field
  $query->leftJoin('content_field_location', 'loc', 'n.nid = loc.nid');

  $query->fields('n', array('nid', 'type', 'title', 'created', 'changed'))
             ->fields('loc', array('field_location_value'));

  $this->addFieldMapping('body', 'body_value') // add semicolon if no options
         ->arguments($arguments)  // optional
         ->defaultValue(''); // optional

This wouldn't be Drupal without some "gotchas". As I was adding fields in each of the migrations I have done eventually the main Migrate page stops listing out my migrations. This was REALLY aggravating and I spent at least a full week debugging this issue across the two migrations I have done. The solution? Stop counting how many things you are going to process (see the section above about using skip_count when connecting to the source.

It probably goes without saying... but test EVERY field after you have defined it. This is where you'll start finding weird stuff in your data. Unless, like nobody before you, your previous database is perfection in every way. Highly doubtful. As you discover problems in your code you can deal with them in processRow or process functions.

Processing the Data

Now that we have all of our fields coming into Drupal and our Migrate page has no errors on the main list and additionally no errors on our specific migration it is time to process the data! Most of the time this will not be necessary, but sometimes you must change things as they import so it will fit the new data structure.

  public function prepareRow($row) {
    // Transform body format to something we can use if it's not already.
    if (!filter_format_exists($row->body_format)) {
      $row->body_format = $this->transformFormatToMachineName($row->body_format);
    $row->temp_data = "somejunk"; // will be passed to prepare function then dropped
    $row->field_text = "XXX"; // sets the value of a text field to the string "XXX" (works for many simple fields)

Within your migration class you will probably always have a prepareRow function. This is where you can make some changes, or alternatively, reject a certain row before it gets imported by return FALSE;. This step DOES NOT RUN for any XML import you do, not even to carry over temp data. Gotcha!

  function prepare($entity, stdClass $row) {
    $entity->field_text['fr'][0] = 'YYY'; // manually set field_text French version (if using entity_translation), for delta 0 (the first instance if allowing multiple values) to YYY

The prepare function is your second chance to change things using the entity pre-rendered, fully expanded into Drupal's typical array syntax. If you're doing XML processing, this is where you need to make any transformations. You can also grab any data you setup in prepareRow... or just run SQL queries here. Be forewarned... if you do SQL queries here it is generally better to load all of your data earlier on (in $this->query) so that you're not hammering your database with requests or blowing out the memory on your server. Those things are hard to detect.

  function complete($entity, stdClass $row) {

The complete function runs at the end. In some cases it is necessary to make an update after the data has been imported. For example: if you are importing line items that are part of an order in Commerce, you will have a task here to update the order total after each line item has been imported. Migrate uses $this->map to track the progress and ID changes between all of your migrations. This function takes that into consideration.

So... sweet, we're already done and we don't even need to use the complete function! That was easy right? Sort-of. It is not always easy. It is a long time-consuming process to verify all the data is correct and to get this far. If you're considering taking on a Migrate project be sure to consider all the possibilities of problems that will come up with the old data. Quantify all of your content types and entities that will be migrated and every individual field contained therein. Expect that each field will need a certain level of review and base your estimates/calculations on that.

The All-Purpose Drupal Entity Translation Guide

Great news Drupalists! Fresh for 2013 is the new 1.0 version of the Entity module. That means we're on stable ground to build our entity-based sites that allow us to translate anything in Drupal. To celebrate the 1.0 milestone, we're finalizing and documenting our build recipe that we use to build multilingual sites.


entity and entity_token
i18n and i18n_menu


First enable the locale and the l10n_update modules. I put the l10n_update in there because I like to see the translated Drupal strings in all their glory.

Go to Configuration > Languages

Enable your second, third, or more additional languages. Drupal will fetch the strings for each language for the modules that you have installed using the l10n_update module.

Edit the English language, add a prefix "en". This will help us deal with English when we setup language negotiation.

Go to Configuration > Languages > Detection and Selection.

Enable path translation, and put it at the top. Then I would enable the browser setting as a fallback. This means that if a user with a French web browser comes to our site, we will give them FR content, unless they explicitly put a different language in the path. For this reason we don't need to worry about what "default" language the site has - the site will always default to the user's default.

Ok. We have languages and a way to manage which one we're using. Now we need an interface to do that.

Go to Structure > Blocks.

Enable the "Language switcher (user interface)" block. Put it somewhere you can get at easily so we can do some testing with it. If you need to change the text name for the language that appears in this block, you can go back to the Config > Languages page and update the title there.

At this point you've configured Drupal to do multiple languages, negotiate between them, and you have an interface to do that as a user. Additionally, Drupal translated itself by fetching all the strings from drupal.org for each module you have installed. As you add more modules, l10n_update will continue to pull down new updates to the text. So to your translators, Drupal will provide service in all the languages that you support. Handy.

Translating Content with Entity Translation

Enabling Entity Translation

Go to Structure > Content Types > Edit the type you want

Publication options: Enabled, with field translation.

Then go to Structure > Content Types > Manage fields for the type you enabled above

Edit the body field, scroll to the bottom, and check the box to enable translation. Repeat this step for every field that you wish to translate. Leave it unchecked if you wish to have the content synchronize all versions.

For the title field, get the title module. Enable it, then go to Structure > Content Types > Manage fields

Then choose the "replace" option for the title field. This will create a new field_title that behaves the same way as the traditional field, but with multilingual features. Edit this field. Remove the description text. Save.


Entity now fully supports revisions. Each update to a translation creates a new revision to the host node.

Rollbacks of versions are not separated by language, so if you must roll back, do it before someone adds new content in a different language. Rolling back will affect all languages that have been updated between the revisions.

Making Menus Entity Translatable

Entities can now have translatable menus. You can structure your entity-based menus two different ways. The entity_translation_menus module takes care of setting up a different menu entry for each version of a page. That is awesome. How you structure the menus are therefor up to you.

You can:

  • Create one menu that has all the language versions grouped together, or...
  • Create a copy of the menu for each version of the language, and allow editors to put the entry in any of those trees when they select where it goes

With the first strategy, if you go to Structure > Menus > Main Menu > List links you will see all the pages at once, but only on this page and when the editor puts something into the site (if you have big menus this could quickly become a big jumbly multi-language mess).

The second method means that when the user selects where they want to put the content into the menu, they must first find the right language menu and put it within the same language part of the site. In other words, the drop-down menu that lets you pick where the menu entry will be, is grouped by language.

This one is a workflow/editorial question I would consider depending on the size of the site and how many languages are being supported. In any case, the decision doesn't matter because you have only one option to configure:

Structure > Menu > Edit menu > Translate and localize (even if you are using it as fixed language)

Then, when you edit... remember to check off "Menu link enabled only for the English language" when you do the original version of the post. It would be really nice if that was set to be the default somehow, but I have not found any settings for this yet.

Entity Translated Paths

By default the pathauto module will do most of what you need here. It will generate aliases for each language version of a node. You can also enable the entity_tokens module to give yourself some additional translated tokens if you need to create some more complex URLs.

Since we are dealing with a multilingual site here, there is one additional module I have to tell you about: transliteration.

This module converts non-ascii characters to ascii for the URL. So rather than "catégories" becoming "cat_gories" we get "categories" as you would more logically expect.

Enable the transliteration module, then:

Configuration > URL aliases > Transliterate prior to creating alias. Done.

Date Formats

This applies more to specific field configurations, but it is likely something you are going to have to take a look at. If you go to the following configuration page you will find your options:

Configuration > Date and Time > Localize

You can also change the first day of the week in the site to change how calendars will be displayed. You can do that at Configuration > Regional Settings.

Translating Blocks

Use the i18n_blocks module. When you create a new block, you will need to specify what language it is. Usually you will want to create blocks with a fixed language, as otherwise you will need to search for the block content as a string. So far we've managed to stay away from Drupal's built-in translation engine, and I like that, because we should only really need to use it to translate strings in Drupal's interface.

Blocks are not the greatest with translations but they do the job. In some cases, it might be easier to just create a new content type and display the translated data with a View.

Solving your multilingual navigation issues with Entity Translation in Drupal 7

UPDATE: the D7 entity module has changed, and now has support for menus. See my latest post on the topic for more up-to-date information.

This blog post has been a long time coming. I have been using Entity Translation, Drupal 7's new interface to doing translations, for the better part of this past year. It has been an exciting adventure, but with every new way of doing things there are some dragons along the path. In this post, I think I have solved one of my last great challenges with it by using some new contrib modules, so I'm documenting my experience here so you do not have to have the same troubles.

Entity Translation, for those who are not aware, is Drupal's transition to translating fields rather than translating nodes. You can read more about it on Gábor's site on Drupal7 Multilingual if you wish (you've probably already seen this page if you were searching around and landed here).

So what is wrong with Drupal's navigation menus when you use Entity Translation?

  • By default, there is only one navigation menu configuration for the node - on the source translation only.
  • The modules relating to entity translation are very new - some of them even lack a user interface!
  • Menus become a difficult problem because they are not entities like everything else
  • "Language neutral" (aka 'LANGUAGE_NONE' or 'und') nav menu entries must be translated from the default language, even if the source node is not in that language, potentially making your translation tables very messy (not to mention this is complicated to setup).

What is the best way to address these issues?

This new module solves a lot of problems by allowing us to add a field to each node that we will use in place of Drupal's standard navigation "menu_links". It then generates Drupal's standard links managed by this module, so that you can use menus in the standard way.

There is one limitation which had me scratching my head for a couple days. If you set the menu field to "translatable" you can get the menu setting on all versions of the translation. This is fabulous. What doesn't work is taking the language from the translated entity and putting that language setting into Drupal's standard "menu_links" table. Not ideal right? Well, there is a workaround.

You see, it is possible to create a navigation menu in Drupal that is language-specific. So, if you take a step back and think about this, since each language version of an entity-node has it's own menu configuration, why not just localize the menu entirely? That way we don't care if the menu_link is language neutral. With this structure, as an example, the French editor when doing the menu settings on a node must only remember to place the French translation within the context of the other French postings. That is simple enough to explain to your users.

Other dragons? Other awesome things?

There are not that many issues I have found yet since this module generates real Drupal menu_links for the navigation menu you can use any modules that support using Drupal's nav menu system, but you might have to do it twice (or more, depending on how many languages you are supporting.

Additional things you might want to think about:

What about the future?

Eventually, when we get around to adopting Drupal8, menu navigation will probably be based on entities. So we're already inching toward the future by embracing entities for everything now!

UPDATE: You may also be interested in my new Entity Translation Tabs module!

UPDATE 2: Keep in mind you don't need many i18n modules with this, and that some i18n modules cause a conflict. Only "Block languages" (i18n_block) and the core i18n modules are necessary for this configuration.

Be sure to disable i18n multilingual select, i18n field translation, i18n path translation and i18n menu translation.

For your convenience:
drush pm-disable i18n_select i18n_field i18n_path i18n_menu


i18n_taxonomy is OK. i18n_taxonomy might cause troubles too, but I have not tested this yet.

I recommend the "Localize" option on your taxonomy, which requires you to put the default language version first, and the "title" module to replace the taxonomy term name interface. You may have to hook_form_alter your Views if you output your taxonomy as an exposed filter to put the translation.

I put the following into my hook_form_alter function to make taxonomy make sense:

  if ('taxonomy_form_term' == $form_id) {
    drupal_set_message('Taxonomy terms must be set in English first.  Then click "save and translate" and you can put the French version.  You can then manage existing translations by going to the term using the translate tab.');
  if ('views_exposed_form' == $form_id) {
    global $language;
    // this uses French, but you could abstract it a bit.
    if ($language->language == 'fr') {
      $result = db_query("select * from {field_data_name_field} where language='fr'");
      foreach ($result as $row) {
          $form['field_series_code_tid']['#options'][$row->entity_id] = $row->name_field_value;

Deploy Drupal's blocks on a different instance using Features without having to re-map your CSS

Get the fe_block module as part of Features Extra.

drush dl features_extra
drush pm-enable fe_block

Go to your blocks page. Configure any custom blocks you are planning to deploy. When you edit a custom block you will now have a "machine_name" variable. Give your block a machine name.

Now the block is available for Features export. You can now go to Features in your Structure menu and generate a new feature with the custom block(s) and their settings (placement in the theme, if it will be the same theme on the target system).

This is great so far, but how will we keep our CSS consistent when the block's ID will change? There is a block_class module but it does not export data to Features. Sad face... but wait! We have a solution:

We can do a custom block template in our theme to add the machine_name as a custom class for this block. So the machine_name for each block will now (1) enable the export and (2) get a class of the same name if the theme supports it.

Copy over your block.tpl.php file to your theme's templates folder.

If you have this file in your theme already, great! If you're using a starter kit theme, look in the starter kit's templates folder for this file, and copy it to templates/block.tpl.php in your theme. If you are coding your theme from scratch, grab the original block.tpl.php file from modules/block/block.tpl.php under the root of your site.

Now that you have added that file to your theme, using the name templates/block.tpl.php :

Find the line that defines the 'class' in HTML. It is probably really close to the top.

Put the following line before it. This code will add the machine name as a class while leaving the rest of the code in-place:
<?php $classes .= ' ' . _fe_block_get_machine_name($block->delta); ?>

Be sure to test this code before you put it live! Otherwise, get to work on your CSS using the machine_name defined by fe_block module! Enjoy never copying and pasting your blocks ever again!

Overriding Drupal7 node display with Views

Back in the Drupal6 days overriding node displays with a View was a pretty common thing for me but since using Drupal7 most of the time it is not necessary. That time has finally come, and I did a bunch of searching around for a better/automatic way but I did not find anything that did not involve over-engineering so I did it in my theme.

First, create a view, as a block (to not to have extra pages floating around).

  • Set the title to %1 to set your view title the same as the node
  • Do not use a pager
  • Display only one item
  • Add a Contextual Filter using the NID field, set the fallback behaviour to "hide"

Then, override the node.tpl.php file in your theme. If you do not have this file in your theme, find out which theme your custom theme is based on, and copy over the relevant file to the correct location for your theme.

In your node.tpl.php file, look for the following code:

print render($content);

...and replace it with a view_embed_view.

print views_embed_view('node','block_1', $node->nid);

In this example, the first parameter, node, is the machine name for my view, and block_1 is the first block in the view. When you are editing your view you will probably see both of these parameters in your URL. Finally, the last parameter is the node ID. This last parameter will probably be the exact same that you will use unless you are doing something truly crazy (like showing the wrong node on purpose? har har).

I only wanted to apply the view to nodes which use the 'article' content type so I combined the old behaviour and the new behaviour with a test for the content type:

if ($node->type == 'article') {
      print views_embed_view('node','block_1', $node->nid);
    else {
      print render($content);

There you have it! Now you can use Views to render your fields for a specific node type... and using Views you can customize your HTML as much as you like using the awesomeness of Views without the overhead of Panels or Display Suite.