1 of 42

Progress on Feeds Migrate Module

Presented by Irina Zaks

Stanford Open Source Lab

Fibonacci Web Studio

​

2 of 42

Irina Zaks

https://www.drupal.org/u/irinaz

One of favorite Authors

Charles Lutwidge Dodgson (27 January 1832 – 14 January 1898), better known by his pen name Lewis Carroll, was an English writer of world-famous children's fiction, notably Alice's Adventures in Wonderland and its sequel Through the Looking-Glass.

3 of 42

In this presentation

  • Content migration and import - use cases
  • Overview and comparison of Feeds and Migrate
  • Feeds + Migrate – updates on development
  • Demo - Power of Open Source�UC SiteFarm distribution importing xml news using Feeds_migrate module
  • Help Wanted - How to contribute
  • Q&A
  • ​

4 of 42

WARNING!

  • This demo uses dev tools
  • Things might and will break
  • Interface and features presented today can and will change

​

5 of 42

6 of 42

Website is supporting mission and operation of your organization

  • Websites are living creatures
  • Websites change, grow, and shrink as do organizations that these sites support
  • New content is generated inside and outside of your organization and needs to be added or imported into your site
  • Content is the king

7 of 42

Moving content into your website: �What

  • Importing structured data in various formats (.csv, xml, json)
  • Importing from various sources
    • Other websites (for example, news, events, people profiles, etc.)
    • Documents (for example, bulk data upload from excel)
    • Database (s)
  • One time import / Periodic import
  • Update, replace or preserve previously imported content items
  • Monitor import state (last import, # of items, etc.)

​

8 of 42

Moving content into Drupal 8:�How

  • “One-click “ upgrade via web interface from existing D6/7 to D8
    • Configuration > Development->Upgrade
  • Feeds Module – define feed type via web interface and import data from any source with defined structure
    • Structure -> Feeds
  • Migrate Suite - set up migrations via drush/yml files from any pre-defined source (db, files, url)
    • Structure -> Migrations
  • Custom scripts

​

​

​

9 of 42

10 of 42

Moving Data: Migration �Define destination based on data source

Data �Source

  • Entities
  • Actual data

Data Processing / �Tampering

  • Reformat date (to ISO)
  • Explode array (from csv list)
  • Find/replace
  • Other transformations

​

Destination container

  • Entities configuration
  • Data import with correct settings

11 of 42

Use case:�Works well for migration from old site to new site

12 of 42

Moving Data: Imports �Reformat data to fit into destination container

Data Source

  • Entities
  • Actual data

Data Processing / �Tampering

  • Reformat date (to ISO)
  • Explode array (from csv list)
  • Find/replace
  • Other transformations

​

Destination container

  • Entities configuration
  • Data formats and settings

13 of 42

Use case:�Works well for importing new data into existing site, for example, news, profiles, products

14 of 42

Two import/migration frameworks in Drupal

​

Migrate and Feeds

15 of 42

Migrate

The Migrate module is used by developers to import and migrate data for simple and complex migrations

​

15

16 of 42

Migrate in core in Drupal 8

  • One click/one time upgrade from D6/7
    • Configuration -> Development -> Upgrade
    • used by site builders
  • Migrations coded using .yaml / drush
    • used by developers

�

17 of 42

Migrate UI

18 of 42

Execute migration

19 of 42

Major Limitation

Migrations are (now) set in code and data sources or processors can not be changed via UI

​

Implication

Migration cannot be changed without updating code �= deployment from dev to prod

​

20 of 42

Modules that provide limited UI

21 of 42

Feeds

The Feeds module has an user interface and aims to support non-developers for importing their content.

​

21

22 of 42

Feeds Module UI Features

  • Define mappings of data sources to existing fields
  • Define data alterations (tamper)
  • Import from RSS, CSV, XML, JSON and other data formats;
  • Fetch from HTTP, file upload or server directory;
  • Import content for about any content entity type;
  • Define schedule to Import content regularly
  • Unpublish/delete content that is no longer provided by the source.
  • Let your editors provide source files and perform imports in UI

​

​

​

23 of 42

Feeds Module Workflow

  • Same is in D8 and D7
  • Add feed type
  • Map sources to existing structure (nodes, taxonomy, users, etc.)
  • Tamper import data if needed
  • Add Feed importer and define data source, frequency, etc.

​

24 of 42

Comparing Feeds and Migrate

1 - Configure Feed type (site builder $$)

  • Select Destination - (node, term, etc) `
  • Define processors - Parser /Source type (file/url)
  • Define naming for data sources and map to destination, add tampers and other feed properties (frequency, etc)

2 – Create Feed Import

  • Add actual data source (upload file or enter url)
  • Import data
  • Delete data

1 - Code migration (developer $$$$)

  • Define data source (file)
  • Define processors
  • Define destination
  • If data source is changed new migration has to be set up

2 - Run Migrations

  • Run import
  • Stop import
  • Rollback - delete all imported data

Migrate - coded in .yml files

Feeds - configured via Web UI

24

25 of 42

Feeds and Migrate working together in Drupal 8

Templates provided by MidCamp.org

26 of 42

New in D8 - Migrate module in Core

  • Migrate in core
    • Parsers
    • Processors
    • Missing - good UI

​

  • Feeds
    • Great UX/UI
    • Feed-specific features
    • Need more tampers and plugins

​

​

Why code and maintain more parsers and processors for feeds if we can use parsers and processors that are already in core

27 of 42

Win-win solution for everyone

- Developers would only have to maintain one import framework;

- Site builders could use the power of Migrate without having to write code;

- Content managers gain the flexibility to import their content without the need to go through another round of development effort.

​

28 of 42

Meet Feeds_migrate module

​

https://www.drupal.org/project/feeds_migrate

​

Templates provided by MidCamp.org

29 of 42

Thank you, avantgarde developers

  • DrupalCon 2018
    • first version by @pookmish
    • @jamesdixon joined the team
    • Demo to maintainers Feeds (@megachriz) and Migrate (@heddn)
    • two workshops by @heddn on Migrate APIs
  • BADCamp 2018
    • “Powerpoint release”
    • @Etroid joined the team
  • MIDCamp 2019
    • Presented working UI
    • @franksj joined the team
  • Colorado DrupalCamp 2019
    • Sprint
    • Solutions for three issues sub

30 of 42

Features

  • Integrated with migrate ecosystem (e.g. core migrate, migrate plus, migrate tools)
  • UI tuned for site builders
  • Works with existing migrations
  • Works with other Drupal distributions (e.g. UC Site Farm)

​

31 of 42

Feeds_migrate Demo

  1. Configure migration
  2. Set up field mapping
  3. Add importer

​

31

32 of 42

Feeds Migrate Module Flow - live demo

  • Same is in Feds 8 and Feeds 7
  • Add Events migration
  • Map sources to existing structure (nodes, taxonomy, users, etc.)
  • Tamper import data if needed in process plugins - work in progress
  • Add Feeds_migrate importer and define data source, frequency, etc.

​

33 of 42

Yes, it is compatible with current migrations structure

34 of 42

Current State of the Feeds Migrate

35 of 42

Features that already work

  • Adding a migration
  • Mapping UI
  • Documentation - work in progress https://www.drupal.org/docs/8/modules/feeds-migrate
  • ​

​

​

36 of 42

37 of 42

We need your help!

​

​

​

​

38 of 42

Wishlist

- Authentication support (like OAuth)

- Prepopulate source fields

- Import preview

​

​

39 of 42

Related core issues

40 of 42

THANK YOU, QUESTIONS?

41 of 42

PLEASE PROVIDE YOUR FEEDBACK!

42 of 42

Contribute to Drupal

You don't have to know code to give back!

​

For new Contributor training contact�AmyJune Hineline of Kanopi Studios

Templates provided by MidCamp.org