Learning Drupal
I’ll break this apart into a multi-part series soon. Importing data in Drupal drupal Importing data The general workflow from original record (source_record) to AI-extracted record (extracted_entity) and finally the innovation (innovation) you find on the user-facing website is: write migration.yml file upload source file (json, csv, …) to web/modules/custom/<your_module_name>/source/<your_source_name_as_in_yml>) drush migrate:import <your migration id> Rolling back This generally fails because of a malformatted migration file or typos etc in the source data, to rollback you do drush migrate:stop <your migration id> to stop any migration process going on drush migrate:reset <your migration id> to set the migration status to ‘idle’ (i.e. ready to import) drush migrate:rollback <your migration id> to remove any partially imported records Some notes on rollback: If you already have downstream content types based off of earlier imported source records, you’re prone to create duplicates when rolling back & re-importing the same sources. Your options are then: drush migrate:rollback <your migration id> --update to only update source_record that changed in the source data (because you e.g. changed the json structure) Purge all downstream content by going to (Web menu -> BACK-END -> Manage <content type>) Once there, filter for all content of interest, select the action ‘Delete’ from the Action drop-down and execute You might need to do this 3 times, for ‘Original Record’, ‘AI-extracted innovation record’ and ‘Innovation’ After purging existing data, you can run drush migrate:import <your migration id> again yaml Template for import uuid: <added after import, delete this column before importing> langcode: en status: true dependencies: enforced: module: - <import_STI_portal_data / import_IRD_jsons> id: <machine reference to this migration> class: null field_plugin_method: null cck_plugin_method: null migration_tags: STI migration_group: STI-import-group label: <description, e.g. the one found in Manage data sources> source: plugin: <csv / json> constants: SOURCE: <source name as found in Manage data sources> SOURCE_ID: <source name as found in the Manage data sources url> RECTYPE: <check what rectypes are set in Mange data source> path: <path to the json or csv file to migrate> header_offset: 0 ids: - <the data column containing unique IDs, eg. id, url, title> process: field_data_source: constants/SOURCE_ID # Needed so that imported records are assigned to the correct data source field_original_internal_id: plugin: skip_on_empty source: <give the same as set in 'ids' above> method: row message: 'Row does not contain Project Symbol: skipped' title: plugin: skip_on_empty source: <data column containing the name/title of the entry> method: row message: 'Row does not contain title: skipped' type: plugin: default_value default_value: source_record # leave unchanged, you're importing a 'source_record' field_shorter_description/value: - plugin: skip_on_empty source: <data column containing a 1-2 sentence description> method: process message: 'Row does not contain short descr.' field_shorter_description/format: plugin: default_value default_value: full_html field_long_description/value: - plugin: skip_on_empty source: <data column containing a long freetext description> method: process message: 'Row does not contain short descr.' field_long_description/format: plugin: default_value default_value: full_html time: plugin: callback callable: time unpack_source: true source: { } field_impact_sdgs: - plugin: explode delimiter: ',' source: <data column mentioning sdgs> - plugin: callback callable: trim - plugin: preg_replace pattern: '\..*' replace: '' - # Often it is necessary to map however sdgs are named in the source data to the STI portal taxonomy plugin: static_map map: '1': 'Goal 1: No poverty' '2': 'Goal 2: Zero hunger' '3': 'Goal 3: Good health and well-being' '4': 'Goal 4: Quality education' '5': 'Goal 5: Gender equality' '6': 'Goal 6: Clean water and sanitation' '7': 'Goal 7: Affordable and clean energy' '8': 'Goal 8: Decent work and economic growth' '9': 'Goal 9: Industry, innovation and infrastructure' '10': 'Goal 10: Reduced inequalities' '11': 'Goal 11: Sustainable cities and communities' '12': 'Goal 12: Responsible consumption and production' '13': 'Goal 13: Climate action' '14': 'Goal 14: Life below water' '15': 'Goal 15: Life on land' '16': 'Goal 16: Peace, justice and strong institutions' '17': 'Goal 17: Partnerships for the goals' default_value: '' - plugin: entity_lookup entity_type: taxonomy_term ignore_case: true value_key: name bundle: impact_sdgs field_region: - plugin: entity_generate entity_type: taxonomy_term ignore_case: true value_key: name source: <region column or field> bundle: countries_no_standard field_innovation_type: - plugin: explode source: <innovation column or field> delimiter: '-' - plugin: callback callable: trim - plugin: entity_lookup entity_type: taxonomy_term ignore_case: true value_key: name bundle: type field_use_cases: - plugin: explode source: <use cases column or field> delimiter: '-' - plugin: callback callable: trim - plugin: entity_lookup entity_type: taxonomy_term ignore_case: true value_key: name bundle: use_cases field_adoption_countries_ns: # this field is needed in case country names do not follow the UN/FAO - Standards - plugin: skip_on_empty method: process source: <use cases column or field> - plugin: explode delimiter: '-' - plugin: entity_generate entity_type: taxonomy_term ignore_case: true value_key: name bundle: countries_no_standard destination: plugin: 'entity:node' default_bundle: source_record overwrite_properties: - field_data_source - field_original_internal_id - field_shorter_description/value - field_long_description/value migration_dependencies: required: { } The migration has 4 top-level parts: ...