Drupal Planet

Migrations on the road

As many people in the Drupal community know, I travel a lot. Today I'm in Paris having just completed a migration in-flight during my trip from Montréal. At the end of August I was in San José, Costa Rica where I presented both of the migrate talks my colleague Novella and I have worked on over the past year (unfortunately she could not make it as she moved to Detroit that week). It was fun to present talks I've worked with for so long... familiar. Low-stress. All the notes are now safely stored in markdown format on GitHub. Bliss.

Since they are both in an accessible format I have posted them here for others to access. Of coruse, particularly with the project management talk, the real fun in the presentation is the examples we cite as we talk through each issue, so you'll have to live without that. Here they are:

Migration Shotgun Tour (Montréal 2016, Costa Rica 2016) - in markdown and as HTML/JS slides.

Project Planning a Migration Project (Montréal 2016, Costa Rica 2016, Ottawa 2015) - in markdown and as HTML/JS slides.

...unfortunately there are no recordings of these talks. I tried to capture audio of one of them and did not realize the disk was full. :-/

Prior to these two talks I have done many different variations of these presentations at other camps and summits going back as far as 2013 when I presented commerce_migrate_ubercart to the Toronto Drupal community. It was my first-ever Drupal talk. Since then I became a maintainer of that module and done many talks and countless migrations since! I'm hoping to get back into Commerce migrations when 2.x hits beta, which I hear will be coming soon.

My next upcoming migrate talk will be a full-day training at BadCamp 2016 in Berkeley, California. It is already sold out (sorry!).

See you at DrupalCon Dublin?

Before BadCamp 2016 I will be in Dublin for DrupalCon... sorta! Specifically I will be at the Business Summit on Monday and the Friday Sprints, but NOT at the conference itself (ie, Tuesday/Wednesday/Thursday). Obviously I don't have a talk at the conference this time around... but I will still be in Dublin for that time so I may still see some of you at the parties. If you want to get lunch outside the conference venue or join me for some sight seeing let me know!

I'm on twitter (and yes DMs will ding my phone, even when I travel, so long as I'm paying attention to it). If it is work related, you can also reach out to me through the contact form at kafei.ca.

Drupal Sprint Cards

Announcing the sprint cards! My team created these cards as a condensed version of the DrupalCon Autsin mentor training. They summraize the top tasks needed for new contributors and coders and can be printed at any local print shop (usually in 12-24h) because we formatted them to be printed on standard business cards. So they are cheap to print and ready when you need them.

These cards have been seen at many Drupal events now and they get a good response wherever they go. The first sprint where they appeared was at the Jersey Shore (thanks again to the NJ team for bringing us there!) and they have since been at camps across North America and now at DrupalCon Amsterdam.

You can find them on our DrupalGive page, or use this handy PDF link.

Printing Specifications

Take this information to your local print shop.

  • Business card size (formatted for North America, so 3.5 × 2 inches... in metric that is 88.9 × 50.8mm [you may ask them to "scale" the PDF to your local size])
  • cardstock (thick paper)
  • matte (not glossy... sometimes people want to write on them)
  • double-sided
  • color
  • "no bleed" (extra cropping not necessary)
  • PDF file
  • quantity $x (most print shops have a special rate for different quantity, good to ask)

Migrating Drupal 8 in Europe

This week we're in Europe for DrupalCon Amsterdam! This is starting to feel suspiciously close to a beta so it is time to dive into Migrate again so you can start working on your new sites with real-world data. Let's begin!

What's up with migrate?

Migrate in the Drupal context means running a migration from the new Drupal 8 site and pulling data from an existing site. It replaces the old upgrade-in-place system that was used in prior versoins of Drupal. So do a fresh install of Drupal 8 and have an old Drupal 6 site on the same host. After you've logged into Drupal 8 you can connect to your Drupal 6 site.

How is Drupal 8 migrate different from Drupal 7 migrate (and migrate_d2d)?

In the older versions of Migrate the process invovled defining your field mappings and manually creating the new content types in the D7 system. This is no longer necessary.

Drupal 8 migrations will automatically create the needed content types and establish the mappings between the old and new fields by default. Therefore much less configuration is needed.

For times when it is necessary to customizae your migration you can use the included hooks or you can use the configuration schema to use the included plugins with your custom data set. The hooks go further than in D7, allowing you to alter data in the prepareRow stage without having to create a custom migration to deal with the data.

Migrate from Drupal 6? What about from Drupal 7?

Migrate frees us from the need to update each sequential version. You can leafrog versions! With Drupal 6 being close to end-of-life it is important to have a pathway for these users. For this reason, the D6 to D8 path was built first.

For Drupal 7: soon. This is now in-progress now that we are finalizing the Drupal 6 code.

Requirements

  • Drush! The UI is still in a contrib sandbox, so for now you must use Drush. The latest version - from Github.
  • Composer. It is needed to install Drush. Go into the drush folder and run "composer install". Already installed? "composer update"
  • D6 Database. Have it on the same host as the new Drupal 8 install.
  • D6 Files. Probably a good idea to be on the same host. Can be file path or http(s) address.
  • D8 Database. A new, empty database. Use the old Creating the database howto if new to this.
  • D8 Files. Check out the git repo... unless of course a beta becomes available. Then use that.

Using Drupal Tools

If you do not currently have a Drupal 8 install one route to get there is to use the Drupal Tools for your platform. It includes all the software you need and the correct versions. It is available for Linux, Mac, and Windows.

If you install the Drupal Tools package it is not necessary to install Git, Drush, or Composer.

The installation should be current as it includes a version of Drush which must be up-to-date. So if you had installed Tools before, double check the version if you have any trouble.

Install Drupal8

Using the database credentials you created, install your new Drupal 8 site.

If you need to rebuild your site remember to delete your ENTIRE files folder and old settings.php. Reinstalling without doing this step will cause problems.

Install Composer / Drush

Make sure you have run the Composer installer. You should be able to type which composer and get a result in your terminal.

Check out the latest Drush. Go into the Drush folder and run composer install.

Next time you git pull (or git fetch) in the Drush folder, make sure to run composer update.

Find an isuse to test

  • Go to the Drupal project issue queue and filter down by Component: migration system, Version: 8.x.
  • Pick an issue that is not already fixed.
  • If you are sprinting with lots of people, pick something further down the list so you are not working on the same thing as someone else.
  • Read the posts. If it is easy enough you think you can handle it, post a comment to say you are doing some work on it.
  • Post the results of your tests.

Time to Migrate

Put the manifest.yml file with the migrations you wish to run in your D8 site root. Then go there on the command line and run the following command, using your D6 database credentials.

You can install Drupal 8 at this stage. If you do, be sure to enable all of the necessary modules. For example, if you use the Book module, it is not enabled by default, so you should enable it now or your book nodes will simply become a regular content type.

When Drupal 8 is no longer in beta the manifest.yml file will not be necessary unless you are doing some custom work. In most cases all that will be necessary is to put in the database credentials and the system will run all the migrations that the system knows about by default.

You will find a manifest.yml file attached to many Migrate issues that will enable you to begin the migration. Here is a sample of what I am using... I have added together many different issues and I run them all at the same time:

To get the complete list of available migrations run drush config-list|grep migrate

# user
- d6_user
- d6_user_profile_field
- d6_user_profile_field_instance
- d6_user_profile_entity_display
- d6_user_profile_entity_form_display
- d6_profile_values:user
- d6_filter_format
- d6_user_role
- d6_user_picture_entity_display
- d6_user_picture_entity_form_display
- d6_user_picture_file
- d6_user_picture_field
- d6_user_picture_field_instance
# taxonomy
- d6_taxonomy_vocabulary
- d6_taxonomy_settings
- d6_taxonomy_term
# nodes
- d6_node
- d6_node_revision
- d6_node_type
- d6_view_modes
- d6_filter_format
- d6_field_instance_per_form_display
- d6_field_instance_widget_settings
- d6_field_formatter_settings
- d6_field_instance
- d6_field
- d6_field_settings
- d6_node_settings
- d6_cck_field_values:*
- d6_cck_field_revision:*
# taxonomy fields
- d6_term_node_revision
- d6_term_node
- d6_vocabulary_entity_display
- d6_vocabulary_entity_form_display
- d6_vocabulary_field_instance
- d6_vocabulary_field
# blocks
- d6_block
- d6_menu
# custom blocks
- d6_custom_block
- d6_filter_format
# book
- d6_book
- d6_book_settings
# files
- d6_file:
  source:
    conf_path: sites/assets
  destination:
    source_base_path: destination/base/path
    destination_path_property: uri

Now that you have created a manifest.yml file in the Drupal 8 root directory you can run the migration.

drush migrate-manifest --legacy-db-url=mysql://d6user:d6pass@localhost/d6 manifest.yml

If you are using Drupal Tools (Acquia Dev Desktop), and if you have also put your D6 site into Dev Desktop, you will need to specify the port number. You can find your database settings by creating an AcquiaDevDesktop terminal and typing drush status to get the exact settings for your D6 site. The result should look something like this:

drush migrate-manifest --legacy-db-url=mysql://drupaluser:@127.0.0.1:33067/drupal_6 manifest.yml

Note that the database user for D6 is called "drupaluser" and it uses the local IP address and port number rather than the host name. Again, run drush status if you are having trouble connecting to verify these values.

Results

After you have run the migration check your work. Did things do what you expected? Post the results of your findings to the issue queue you of the item you were working on.

Was the result successful? If so, post the result.

Did something fail? Post the result.

Post your results! Don't be afraid to comment on Drupal.org. If you provide examples of your tests you will help the migration path improve.

Rolling back (starting over)

  • It is possible to re-run the migration. This can be helpful if you forgot to run a component, or if you have new items in the source site that you would like to add to the Drpual8 site.
  • To completely "roll-back" you really need to reinstall Drupal 8. To do this you must do three things: (1) empty your database, (2) delete settings.php, (3) remove the files folder completely.

Files

Le sigh!

One problem you will run into at the moment is that using drush there is no way to set the path of the files directory from d6.

UPDATE: use the new manifest format in the code block shown above to set your file paths. It is the ONLY settings you will need in your manifest file.

The other solution is to use the migrate_upgrade contrib module (which will hopefully be part of 8.1). This module provides a URL at /upgrade which allows you to put database settings and file path. It does not let you select which migratiosn to run - they ALL run. So if you must test file migrations, this is the way (for now) but note that it is not as focused as the other testing methods.

Currently migrate_upgrade has some bugs! Look at the issue queue for the critical patches until the next update to migrate_upgrade is ready (at the moment the last version of that module dates back to June 2014).

Going further

Migrating multilingual data into Drupal 8

First thing, have a D6 site with multilingual things ready to go! This node has a file attached (see end of post) with a copy of D6 with some translated nodes and site information (see settings.php file settings to translate these variables). The file is a drush archive.

D6 "Requirements" at this time:

  • It must have CCK installed (2014-04-11)
  • Recommended to install in the non-English language if you wish to download the translations automatically (this isn't working in the UI yet 2014-04-11). You can also go download the translations manually later.

Setting up D8

Currently we do not work from the main branch as there are significant changes happening in this area of Drupal 8. Some things may be broken. You have been warned! Please test and update issue summaries where appropriate.

  • Clone the IMP migration sandbox and install Drupal 8 https://drupal.org/sandbox/chx/2105305
  • Enable migrate, migrate_drupal, content_translation, configuration_translation, locale/"Interface Translation", language (these last two are installed by default if you did not install in English)
  • Enable English as a language
  • Get the patch so you can bind to a second database in D8 https://drupal.org/node/2181775
  • Create a manifest.yml file in the root

Migration Mappings

Put this file int he root of your D8 site. If you did migrations to D7 with the latest versions of migrate and d2d, this file similar to your migrate.inc file registration array (but in a completely new format).

manifest.yml

# nodes
- d6_user_role
#- d6_user (not working yet - "does not meet requirements" 2014-04-11)
- d6_language_types # Patch needed 2014-04-11 https://drupal.org/node/2225293
- d6_language_negotiation # needs above patch
- d6_language # Patches needed 2014-04-11 https://drupal.org/node/2166875 and https://drupal.org/node/2234623

- d6_filter_format

- d6_node_type
- d6_node_settings # gets story content type
- d6_node:*
- d6_node_body_field

- d6_view_modes

You can find the complete list of migrations at core/modules/migrate_drupal/migrate.config.yml and core/modules/migrate_drupal/migrate.content.yml some (like d6_user are not yet working 2014-04-11).

Running the Migration

Then, on the command line, run your manifest file. I used a D6 database with no password because of the bug mentioned above.

drush migrate-manifest mysql://d6@localhost/d6 manifest.yml

You will need the latest Drush if you want support D8! A UI-based workflow is being developed here: https://drupal.org/node/2200379

Rolling Back

This does not exist yet, but if you are familiar with doing this manually in D7 the process is the same:

Use the UI to delete the affected things (in this example, delete your nodes)
Clear the database table that stores the migration info for the specific migration: mysql> delete from migrate_map_d6_node;

Other howtos

Writing D6 to D8 entity migrations - https://groups.drupal.org/node/387488
How to test Drupal 6 to Drupal 8 migrations - https://groups.drupal.org/node/398588
eliza411 - http://dspeak.com/fldc14/dothis.html

More information on Migrate in Core / IMP

You can follow the initiative at groups.drupal.org/imp.

Getting content into Drupal with Migrate : Montréal (v.0.4)

This year I'm presenting on Migrate at Montréal DrupalCamp! It will be the best one so far! I refactored the presentation yet again.

Recently I have published a new module, migrate_webform, that is a good example of the items in the talk. I also refactored commerce_migrate_ubercart to work with migrate version 2.6, which is the one with the amazing new UI config.

My company, Kafei Interactive, is a sponsor this year. We also sponsored Toronto DrupalCamp. I welcome Git tip if you want to buy me a coffee.

AttachmentSize
File migrate_montreal.odp600.7 KB
PDF icon migrate_montreal.pdf584.13 KB

Getting content into Drupal with the Migrate module (Vancouver v.0.3)

This is the third version of my migration presentation, now with much more code samples and information on how to use migrate_d2d. This time it was presented in Vancouver for the Pacific Northwest Drupal Summit.

Update: on slide 24 migration.inc file should be "kafei.migrate.inc".

Migrate all the things!

This is a presentation I did for the 2013 Toronto DrupalCamp.

Migrate allows you to bring content into your new Drupal site from a variety of data sources including past versions of Drupal. In this session we will talk about why you should consider using Migrate over other methods such as upgrading Drupal, using feeds, node export, copy and paste, etc. We will look at what data sources can be imported, how to create migration classes and where to find starter templates you can use to build your own custom migrations.

At the end of the presentation we walked through my previous blog post for code examples and more detail. http://www.verbosity.ca/working-drupals-migrate-module

EDIT: the prepareRow and prepare functions had their names inverted on the slide titles.

Working with Drupal's Migrate module

UPDATE: if you're interested in this article you should check out the slides from my 4th migrate presentation. The registration system in migrate has changed ("automatic" registration is gone) and I've explained it all in better detail in the slides. I even cover the pathway of the variables if you're new to OOP. ...and yes, I'm available for support and/or workshops.

I have had the opportunity to work on two Drupal sites that involved the Migrate module. I have in the past worked fairly extensively with the Feeds module to do this type of thing. Feeds is a great and powerful module, but at a certain point you will probably find that you want to use Migrate.

Why use Migrate rather than Feeds?

  • Rollbacks. The migrate module makes it damn easy to rollback and re-import.
  • Rules in code. Any changes you do to your input file are stored in a coded template.

The downside as you have probably already guessed is that you're going to spend a lot of time writing custom code here. On the bright side, there isn't that much complicated stuff once you know what everything is trying to do.

Example Migrate Templates

These are the ways that do not involve using much code. The reality is that you *will* be doing code with almost any Migrate project. Before you get started the best things you can do are (1) read the manual on Drupal.org, and (2) find a reference template that is close to what you want to do.

There is one additional thing... the beer and the wine templates. These are part of the migrate_example package that is part of the main migrate module. Review these, the code is commented specifically for this purpose.

Also check out the following as possible starting points:

Additionally, there is a Wordpress conversion template, and ones for Typo3 and PhpBB. The Wordpress one has a UI and works quite flawlessly (though you will need to remap your image locations somehow).

Migrate and the Future of Drupal

Some people have suggested using Migrate as the main way to upgrade between versions of Drupal. My guess is that my team is going to use it a lot for D7->D8 migrations when the time comes, and I suspect that it will be the default way for D9.

Setting Up Your Dev Environment and Migrate Basics

One thing about Migrate that you should know is that some of the queries that will run will be enormously expensive processor-wise. Especially when things aren't working correctly. I find it is best to work from a local environment rather than a dev server until you have a working template. Additionally, it helps to have a process monitor showing the activity on the machine so you can see when things go wrong.

Another tip that you will need to know is the command drush_print_r($variable); in your PHP code that will help you to analyze the objects referenced in your code and to see what structure Drupal is expecting when you put stuff into $row->something; (prepareRow) or $entity->field_name[][0]['value']; (prepare).

Familiarize yourself with the following commands, as it is recommend to use drush to do your migration:

drush ms
drush mi $migration
drush mr $migration

You will probably need some debugging tools as well:

drush_print_r($var); - this command does what you would expect - dumps a variable to the command line. You put this in your PHP code.

drush_print_r($query->__toString()); - this will dump the MySQL query that is generated by Drupal's dynamic query engine. If you have one template that won't run, this can come in handy. You can adapt the output of this to run in MySQL directly - useful if you need to confirm that the query is causing memory problems. You can prepend the keyword EXPLAIN to the query generated for additional info from MySQL.

Anatomy of a Migrate template

The migrate module needs some basic definitions so that it will show up on the Migrate page, which is a tab on the Content list page. Most of the time "registration" of the migration classes is "automatic" but I have found that it is useful to use the tab on that page to force it to look for migrations.

In the case of Commerce Ubercart Migration module, there is also a settings tab which will appear for that. You will want to configure it to connect to the existing data set and it will create a bunch of "migrations" to show on the main Migrate page.

When your migrations are all listed, you can click one and see what fields are mapped and which are not. This is where you're going to spend a lot of your migrate time.

Typically you will have a custom module with the following:

  • custom.info file
  • custom.module file, blank... unless you want to use it for something*
    • *an optional API definition if not using automatic-registration
  • something.inc, your first migration, listed in your files in the info file
  • another.inc, your second migration, also listed in files of info file

Within your .inc files you will need to know about the following components which are almost always going to be in your migration:

  • class CommerceMigrateUbercartNodeMigration extends DynamicMigration class definition setting what migration will be used as a base. Then the following sub-components:
    • public function __construct(array $arguments); // initialize the migration, connect to source and grab the settings! Contains a reference to what will be used to map the migration (keeps track of changes to IDs)
    • public function prepareRow($row); // preprocess function, DOES NOT RUN FOR XML IMPORTS
    • function prepare($entity, stdClass $row); // last chance to do stuff. Can run SQL queries here, first param is always the target node/order/entity fully expanded, so put in your languages setting and delta values!
    • function complete($entity, stdClass $row); // rarely needed if you did things right. Can be used for doing post-migration tasks but could complicate your rollbacks, so be sure to test!

Initializing the Migration

With Migration templates you're dealing with Object Oriented Programming, not a staple in our Drupal PHP diet yet. If you're familiar with OOP you'll be right at home. Let's start out with a definition:

class CommerceMigrateUbercartNodeMigration extends DynamicMigration {
  // An array mapping D6 format names to this D7 databases formats.
  public $filter_format_mapping = array();

  public function __construct(array $arguments) {
    $this->arguments = $arguments;
    parent::__construct();
    $this->description = t('Import product nodes from Ubercart.');
    $dependency_name = 'CommerceMigrateUbercartProduct' . ucfirst($this->arguments['type']);
    $this->dependencies = array($dependency_name);

    // Create a map object for tracking the relationships between source rows
    $this->map = new MigrateSQLMap($this->machineName,
      array(
        'nid' => array(
          'type' => 'int',
          'unsigned' => TRUE,
          'not null' => TRUE,
          'description' => 'Ubercart node ID',
        ),
      ),
      MigrateDestinationNode::getKeySchema()
    );

There is a lot of stuff going on here. Firstly, we're extending a base class. That means this will apply to any objects of that class. So we can just assume that if the base class Migrate is running, that this class will run as well. What follows is a constructor, which what is going to run at startup.

The last part of this code block is the most interesting: $this->map. Here is how Drupal is going to track the objects in your migration. It is your key for each item in your import - and it is going to BREAK if you are trying to use XML as your import source. Then you're going to have to dig around in your DB and change that type from int to something else.

Connecting to Source(s)

The most exciting part about this part of the migration template is that you have many options. In this case we're looking at a D6->D7 migration, so in our case D6 is our source! Migrate takes care of the actual connection to the database so that part is easy, you can point it right at production to grab the data. As noted above, XML can also be your source(s). When I say source(s) I mean possibly plural... the example code provided will let you cycle through many XML files during the import process. Potentially really handy if your exports were run in batches.

Note, that the switch statement in the start of this module is related to the UI of Migrate, where the author of this module has made a switch available. Most migrations don't provide a settings form for Migrate but this one does.

    // Create a MigrateSource object, which manages retrieving the input data.
    $connection = commerce_migrate_ubercart_get_source_connection();

    $query = $connection->select('node', 'n');

    switch (SOURCE_DATABASE_DRUPAL_VERSION) {
      case 'd7':
        $query->leftJoin('field_data_body', 'fdb', 'n.nid = fdb.entity_id AND n.vid = revision_id');
        $query->leftJoin('field_data_field_campaign_id', 'cid', 'n.nid = cid.entity_id');
        $query->fields('n', array('nid', 'type', 'title', 'created', 'changed'))
              ->fields('fdb', array('body_value', 'body_summary', 'body_format', 'language'))
              ->fields('cid', array('field_campaign_id_value'))
              ->condition('n.type', $arguments['type'])
              ->distinct();
        break;
      case 'd6':
        $query->leftJoin('node_revisions', 'nr', 'n.nid = nr.nid AND n.vid = nr.vid');
        $query->leftJoin('filter_formats', 'ff', 'nr.format = ff.format');
        // ... and so on ...
    }
    $this->source = new MigrateSourceSQL($query, array(), NULL, array('map_joinable' => FALSE, 'skip_count' => TRUE)); // connect!

You probably noted the D7 and D6 options have different mappings. Take note. If you're going to extend one, do the one you intend to use.

When your main Migrate page stops working...

Eventually if you start adding fields your query will silently break, and will force Migrate's listing page to hang. Usually you can still access the page for the specific migration but since the purpose of these pages is to help guide your clients through the process this is a severe deal-breaker. Fortunately, it is easily solved one of two different ways. Either (1) don't bother counting how many tasks are to be done or (2) specify a basic query that can be tabulated quickly.

Here is how you do that:

$this->source = new MigrateSourceSQL($query, array(), NULL, array('map_joinable' => FALSE, 'skip_count' => TRUE));

In your source you specify skip_count as a parameter and you set it to TRUE. Now you're going to have an X where the number of items to be migrated was... but you can at least get to that page. Whew...

Adding each of your fields

This is where start getting fun... or really boring. Your pick. We now have to go through each field that will be brought into the database and create a relationship with the place where we want it to go.

Each field in your migration is going to have the following components:

  • A field in your target content type to hold the incoming data
  • An extension to the $query object to perform a LEFT JOIN if your source is in a table that is not in the base query
  • An extension to the $query object's fields to SELECT by column name
  • An extension to $this to connect the old field to the new field
  $query->leftJoin('content_field_location', 'loc', 'n.nid = loc.nid');

  $query->fields('n', array('nid', 'type', 'title', 'created', 'changed'))
             ->fields('loc', array('field_location_value'));

  $this->addFieldMapping('body', 'body_value') // add semicolon if no options
         ->arguments($arguments)  // optional
         ->defaultValue(''); // optional

This wouldn't be Drupal without some "gotchas". As I was adding fields in each of the migrations I have done eventually the main Migrate page stops listing out my migrations. This was REALLY aggravating and I spent at least a full week debugging this issue across the two migrations I have done. The solution? Stop counting how many things you are going to process (see the section above about using skip_count when connecting to the source.

It probably goes without saying... but test EVERY field after you have defined it. This is where you'll start finding weird stuff in your data. Unless, like nobody before you, your previous database is perfection in every way. Highly doubtful. As you discover problems in your code you can deal with them in processRow or process functions.

Processing the Data

Now that we have all of our fields coming into Drupal and our Migrate page has no errors on the main list and additionally no errors on our specific migration it is time to process the data! Most of the time this will not be necessary, but sometimes you must change things as they import so it will fit the new data structure.

  public function prepareRow($row) {
    // Transform body format to something we can use if it's not already.
    if (!filter_format_exists($row->body_format)) {
      $row->body_format = $this->transformFormatToMachineName($row->body_format);
    }
    $row->temp_data = "somejunk"; // will be passed to prepare function then dropped
    $row->field_text = "XXX"; // sets the value of a text field to the string "XXX" (works for many simple fields)
  }

Within your migration class you will probably always have a prepareRow function. This is where you can make some changes, or alternatively, reject a certain row before it gets imported by return FALSE;. This step DOES NOT RUN for any XML import you do, not even to carry over temp data. Gotcha!

  function prepare($entity, stdClass $row) {
    $entity->field_text['fr'][0] = 'YYY'; // manually set field_text French version (if using entity_translation), for delta 0 (the first instance if allowing multiple values) to YYY
  }

The prepare function is your second chance to change things using the entity pre-rendered, fully expanded into Drupal's typical array syntax. If you're doing XML processing, this is where you need to make any transformations. You can also grab any data you setup in prepareRow... or just run SQL queries here. Be forewarned... if you do SQL queries here it is generally better to load all of your data earlier on (in $this->query) so that you're not hammering your database with requests or blowing out the memory on your server. Those things are hard to detect.

  function complete($entity, stdClass $row) {
  }

The complete function runs at the end. In some cases it is necessary to make an update after the data has been imported. For example: if you are importing line items that are part of an order in Commerce, you will have a task here to update the order total after each line item has been imported. Migrate uses $this->map to track the progress and ID changes between all of your migrations. This function takes that into consideration.

So... sweet, we're already done and we don't even need to use the complete function! That was easy right? Sort-of. It is not always easy. It is a long time-consuming process to verify all the data is correct and to get this far. If you're considering taking on a Migrate project be sure to consider all the possibilities of problems that will come up with the old data. Quantify all of your content types and entities that will be migrated and every individual field contained therein. Expect that each field will need a certain level of review and base your estimates/calculations on that.

Getting started with multilingual content in Drupal 8

Today I re-installed my copy of Drupal 8 that I use for testing and I must say... wow, there is so much good stuff in here. I resisted upgrading to Drupal 7 with my projects when it first came out and for good reason... many modules that are necessary for my job were not ready when the core module was. I understand this was an improvement over previous upgrades but with Drupal 8 I'm going to be ready to dive in the first day that Drupal 8 is released. This is awesome.

Getting Started Working with Core Dev

First, you're going to need a recent version of PHP. For this reason, I had to use my laptop as a host because my development server uses an older version and I'm not ready to stop production for a beta test. Once you have a suitable environment ready we can proceed with installation.

When the download is complete, visit that site on your webserver. You will now begin the standard Drupal install process. But wait! We have a few extra things to note...

  • I browse in French, so it asked me to install in French!
  • I had to create both files and files/translations in sites/defualt and give them both appropriate permissions (777?)
  • Copy default.settings.php in your sites/default folder to settings.php ... permission it the same way as above
  • Put in your database settings. I picked "minimal" configuration, as I always do.

As I was doing these steps Drupal fetched the French translations for me and did most of the installation in French. Drupal, you are so awesome.

Post-Install Configuration

I do a few things to get setup in my new dev/beta environment because I picked the "minimal" installer.

  • Switch to Bartik theme
  • Enable "color" module and change Bartik's blue to something nicer
  • Enable "toolbar" module so you don't have to click around all over the place
  • Create a "page" content type
  • Enable "field ui" module

Multilingual Config

  • Enable the "content translation" module (the other multilingual modules, "interface translation" and "language" were already enabled for me)
  • Go to Configuration > Regional and Language > Content language settings and enable "content" (thanks Gábor for pointing this out). You can also enable which fields get translated on this page but you may need to enable the content type translation (see next two steps) first.
  • Edit the "page" content type, enable translation on the type
  • Go to "manage fields" for the "page" content type, edit the "body" field, go to the "field settings" tab, enable translation of the field (you must have >1 fields enabled for translation or your page will just redirect to the "manage fields" page
  • Now go edit some content! You will first need to create a node, and then click the translate tab to add a translation
  • You will need a language switcher. I found this hard to figure out at first, but then I realized that most blocks don't exist by default, so if you go to Structure > Add blocks you will be presented with a list of blocks you can create, one of which is a language switcher for content

Now you should be able to create content and switch between the languages. It seems that if a field is blank in one language, it will be sourced from a different language where that content is available.

What Needs Work (Jan 18 2013)

So far I have only noticed a few issues (Jan 18 2013)...

  • The title field is not (yet) translatable
  • If fields aren't translatable on a node type, you get redirected to the "manage fields" page but no message to tell you why you are there
  • Not knowing how to add a language switcher block was a bit confusing
  • The "check for updates now" link is really low profile, I tried to add additional languages but they all have only 7.x branches of the translation files so this link did not work unfortunately
  • The new toolbar does not have icons for the translated versions of the items. Not sure why these little pictures need to be translated/translatable!

Overall these are very minor concerns that anyone who is familiar with Drupal should be able to overcome easily. Especially if you have been doing entity translation in Drupal 7. You'll probably note that the old method of having duplicate nodes of each piece of content, one node per language, is officially gone. This is not a possible way to translate in Drupal 8. Good.

How to Help

Drupal.org is the best place to look for information on how to contribute but I'll try to make a quick summary of the types of places you can contribute.

  • First, look at the issue queue for Drupal 8 issues on the Drupal.org project page for Drupal noted at the start of this article. Read over the issues and comment on them if you can summarize and/or offer feedback on the progress that is being made.
  • Once you are aware of the issues, and you have done some testing on your own, create new issues for items that should be done prior to release. I will be looking at the items I mentioned above as my next steps.
  • Create patches for issues where you are able to fix the code. Since you used git to get the latest version at the start, you can just run git to create your patch: git diff >ddo-issue-number-comment-number.patch ... then take that patch over to the relevant issue and post it to drupal.org!
  • Once you're comfortable with creating patches, you can also test some patches! They are a bit harder to manage sometimes, as you may need to look at the path contained in the patch file. In general, you can probably go to the drupal root and run patch < patchfile.patch. Cross your fingers and hope for no rejects. Then test it and provide feedback on the issue.
  • Join an IRC channel. If you are at a code sprint, this is how we share links during discussion. At other times, IRC is a good substitute when you need to ask a question but you're developing on your own.

Finally, after you've done a few of these things, you're going to come back to the site in a day or two and... throw the whole thing in the garbage. I wish I were joking. Since Drupal 8 is still in development at the time of this writing, major changes can happen that break your config. So often, you will simply want to wipe out your dev site and start over rather than face bugs that are undocumented and temporary. Best to start with the latest, each time you start... unless you're reading every commit message and doing a git diff to see what happened.

Following the Initiatives

There are many "core initiatives" which are focused areas of work on Drupal 8. I have been most involved in Drupal 8 Multilingual... or D8MI. If you search D8MI as a tag on the Drupal project page, you will see what work is to be done. This is possible because the D8MI volunteers have tagged the issues and Gábor Hojtsy, the initiative leader, does an excellent job of keeping us organized.

You can also check out the individual sites for each initiative to see what issues need the most work. Joining the initiative on groups.drupal.org is a good way to stay informed, and each group also hosts IRC meetings on a regular basis.

Drupal 8 Core Initiatives list
Drupal 8 Multilingual homepage (see "focus" page for top priorities)

The All-Purpose Drupal Entity Translation Guide

Great news Drupalists! Fresh for 2013 is the new 1.0 version of the Entity module. That means we're on stable ground to build our entity-based sites that allow us to translate anything in Drupal. To celebrate the 1.0 milestone, we're finalizing and documenting our build recipe that we use to build multilingual sites.

Modules

locale
entity and entity_token
entity_translation
i18n and i18n_menu
l10n_update
token
pathauto
transliteration

Install

First enable the locale and the l10n_update modules. I put the l10n_update in there because I like to see the translated Drupal strings in all their glory.

Go to Configuration > Languages

Enable your second, third, or more additional languages. Drupal will fetch the strings for each language for the modules that you have installed using the l10n_update module.

Edit the English language, add a prefix "en". This will help us deal with English when we setup language negotiation.

Go to Configuration > Languages > Detection and Selection.

Enable path translation, and put it at the top. Then I would enable the browser setting as a fallback. This means that if a user with a French web browser comes to our site, we will give them FR content, unless they explicitly put a different language in the path. For this reason we don't need to worry about what "default" language the site has - the site will always default to the user's default.

Ok. We have languages and a way to manage which one we're using. Now we need an interface to do that.

Go to Structure > Blocks.

Enable the "Language switcher (user interface)" block. Put it somewhere you can get at easily so we can do some testing with it. If you need to change the text name for the language that appears in this block, you can go back to the Config > Languages page and update the title there.

At this point you've configured Drupal to do multiple languages, negotiate between them, and you have an interface to do that as a user. Additionally, Drupal translated itself by fetching all the strings from drupal.org for each module you have installed. As you add more modules, l10n_update will continue to pull down new updates to the text. So to your translators, Drupal will provide service in all the languages that you support. Handy.

Translating Content with Entity Translation

Enabling Entity Translation

Go to Structure > Content Types > Edit the type you want

Publication options: Enabled, with field translation.

Then go to Structure > Content Types > Manage fields for the type you enabled above

Edit the body field, scroll to the bottom, and check the box to enable translation. Repeat this step for every field that you wish to translate. Leave it unchecked if you wish to have the content synchronize all versions.

For the title field, get the title module. Enable it, then go to Structure > Content Types > Manage fields

Then choose the "replace" option for the title field. This will create a new field_title that behaves the same way as the traditional field, but with multilingual features. Edit this field. Remove the description text. Save.

Revisioning

Entity now fully supports revisions. Each update to a translation creates a new revision to the host node.

Rollbacks of versions are not separated by language, so if you must roll back, do it before someone adds new content in a different language. Rolling back will affect all languages that have been updated between the revisions.

Making Menus Entity Translatable

Entities can now have translatable menus. You can structure your entity-based menus two different ways. The entity_translation_menus module takes care of setting up a different menu entry for each version of a page. That is awesome. How you structure the menus are therefor up to you.

You can:

  • Create one menu that has all the language versions grouped together, or...
  • Create a copy of the menu for each version of the language, and allow editors to put the entry in any of those trees when they select where it goes

With the first strategy, if you go to Structure > Menus > Main Menu > List links you will see all the pages at once, but only on this page and when the editor puts something into the site (if you have big menus this could quickly become a big jumbly multi-language mess).

The second method means that when the user selects where they want to put the content into the menu, they must first find the right language menu and put it within the same language part of the site. In other words, the drop-down menu that lets you pick where the menu entry will be, is grouped by language.

This one is a workflow/editorial question I would consider depending on the size of the site and how many languages are being supported. In any case, the decision doesn't matter because you have only one option to configure:

Structure > Menu > Edit menu > Translate and localize (even if you are using it as fixed language)

Then, when you edit... remember to check off "Menu link enabled only for the English language" when you do the original version of the post. It would be really nice if that was set to be the default somehow, but I have not found any settings for this yet.

Entity Translated Paths

By default the pathauto module will do most of what you need here. It will generate aliases for each language version of a node. You can also enable the entity_tokens module to give yourself some additional translated tokens if you need to create some more complex URLs.

Since we are dealing with a multilingual site here, there is one additional module I have to tell you about: transliteration.

This module converts non-ascii characters to ascii for the URL. So rather than "catégories" becoming "cat_gories" we get "categories" as you would more logically expect.

Enable the transliteration module, then:

Configuration > URL aliases > Transliterate prior to creating alias. Done.

Date Formats

This applies more to specific field configurations, but it is likely something you are going to have to take a look at. If you go to the following configuration page you will find your options:

Configuration > Date and Time > Localize

You can also change the first day of the week in the site to change how calendars will be displayed. You can do that at Configuration > Regional Settings.

Translating Blocks

Use the i18n_blocks module. When you create a new block, you will need to specify what language it is. Usually you will want to create blocks with a fixed language, as otherwise you will need to search for the block content as a string. So far we've managed to stay away from Drupal's built-in translation engine, and I like that, because we should only really need to use it to translate strings in Drupal's interface.

Blocks are not the greatest with translations but they do the job. In some cases, it might be easier to just create a new content type and display the translated data with a View.