I’ll break this apart into a multi-part series soon.
Importing data in Drupal drupal
Importing data
- The general workflow from original record (
source_record) to AI-extracted record (extracted_entity) and finally the innovation (innovation) you find on the user-facing website is:- write migration.yml file
- upload source file (json, csv, …) to
web/modules/custom/<your_module_name>/source/<your_source_name_as_in_yml>) drush migrate:import <your migration id>
Rolling back
- This generally fails because of a malformatted migration file or typos etc in the source data, to rollback you do
drush migrate:stop <your migration id>to stop any migration process going ondrush migrate:reset <your migration id>to set the migration status to ‘idle’ (i.e. ready to import)drush migrate:rollback <your migration id>to remove any partially imported records
- Some notes on rollback: If you already have downstream content types based off of earlier imported source records, you’re prone to create duplicates when rolling back & re-importing the same sources. Your options are then:
drush migrate:rollback <your migration id> --updateto only updatesource_recordthat changed in the source data (because you e.g. changed the json structure)- Purge all downstream content by going to (Web menu -> BACK-END -> Manage <content type>)
- Once there, filter for all content of interest, select the action ‘Delete’ from the Action drop-down and execute
- You might need to do this 3 times, for ‘Original Record’, ‘AI-extracted innovation record’ and ‘Innovation’
- After purging existing data, you can run
drush migrate:import <your migration id>again
yaml Template for import
uuid: <added after import, delete this column before importing>
langcode: en
status: true
dependencies:
enforced:
module:
- <import_STI_portal_data / import_IRD_jsons>
id: <machine reference to this migration>
class: null
field_plugin_method: null
cck_plugin_method: null
migration_tags: STI
migration_group: STI-import-group
label: <description, e.g. the one found in Manage data sources>
source:
plugin: <csv / json>
constants:
SOURCE: <source name as found in Manage data sources>
SOURCE_ID: <source name as found in the Manage data sources url>
RECTYPE: <check what rectypes are set in Mange data source>
path: <path to the json or csv file to migrate>
header_offset: 0
ids:
- <the data column containing unique IDs, eg. id, url, title>
process:
field_data_source: constants/SOURCE_ID # Needed so that imported records are assigned to the correct data source
field_original_internal_id:
plugin: skip_on_empty
source: <give the same as set in 'ids' above>
method: row
message: 'Row does not contain Project Symbol: skipped'
title:
plugin: skip_on_empty
source: <data column containing the name/title of the entry>
method: row
message: 'Row does not contain title: skipped'
type:
plugin: default_value
default_value: source_record # leave unchanged, you're importing a 'source_record'
field_shorter_description/value:
-
plugin: skip_on_empty
source: <data column containing a 1-2 sentence description>
method: process
message: 'Row does not contain short descr.'
field_shorter_description/format:
plugin: default_value
default_value: full_html
field_long_description/value:
-
plugin: skip_on_empty
source: <data column containing a long freetext description>
method: process
message: 'Row does not contain short descr.'
field_long_description/format:
plugin: default_value
default_value: full_html
time:
plugin: callback
callable: time
unpack_source: true
source: { }
field_impact_sdgs:
- plugin: explode
delimiter: ','
source: <data column mentioning sdgs>
- plugin: callback
callable: trim
- plugin: preg_replace
pattern: '\..*'
replace: ''
-
# Often it is necessary to map however sdgs are named in the source data to the STI portal taxonomy
plugin: static_map
map:
'1': 'Goal 1: No poverty'
'2': 'Goal 2: Zero hunger'
'3': 'Goal 3: Good health and well-being'
'4': 'Goal 4: Quality education'
'5': 'Goal 5: Gender equality'
'6': 'Goal 6: Clean water and sanitation'
'7': 'Goal 7: Affordable and clean energy'
'8': 'Goal 8: Decent work and economic growth'
'9': 'Goal 9: Industry, innovation and infrastructure'
'10': 'Goal 10: Reduced inequalities'
'11': 'Goal 11: Sustainable cities and communities'
'12': 'Goal 12: Responsible consumption and production'
'13': 'Goal 13: Climate action'
'14': 'Goal 14: Life below water'
'15': 'Goal 15: Life on land'
'16': 'Goal 16: Peace, justice and strong institutions'
'17': 'Goal 17: Partnerships for the goals'
default_value: ''
-
plugin: entity_lookup
entity_type: taxonomy_term
ignore_case: true
value_key: name
bundle: impact_sdgs
field_region:
-
plugin: entity_generate
entity_type: taxonomy_term
ignore_case: true
value_key: name
source: <region column or field>
bundle: countries_no_standard
field_innovation_type:
-
plugin: explode
source: <innovation column or field>
delimiter: '-'
- plugin: callback
callable: trim
-
plugin: entity_lookup
entity_type: taxonomy_term
ignore_case: true
value_key: name
bundle: type
field_use_cases:
-
plugin: explode
source: <use cases column or field>
delimiter: '-'
- plugin: callback
callable: trim
-
plugin: entity_lookup
entity_type: taxonomy_term
ignore_case: true
value_key: name
bundle: use_cases
field_adoption_countries_ns: # this field is needed in case country names do not follow the UN/FAO - Standards
-
plugin: skip_on_empty
method: process
source: <use cases column or field>
-
plugin: explode
delimiter: '-'
-
plugin: entity_generate
entity_type: taxonomy_term
ignore_case: true
value_key: name
bundle: countries_no_standard
destination:
plugin: 'entity:node'
default_bundle: source_record
overwrite_properties:
- field_data_source
- field_original_internal_id
- field_shorter_description/value
- field_long_description/value
migration_dependencies:
required: { }
The migration has 4 top-level parts:
- The header sections
id,label, etc sourcedescribing the source data and its structureprocessdescribing how to read, process and pass on the source data fieldsdestinationdescribing where the data is supposed to end up. Just put:
plugin: 'entity:node' default_bundle: source_record- The header sections
Minimum fields for migration
What should you parse out from the data? What should be there? I think the list below is good for a start:
titlefield_original_internal_idfield_shorter_descriptionfield_long_description
Additional: Should be sourced from the data
field_linkfield_ownerfield_impact_sdgsfield_country_originfield_countries_adoption
Drupal plugins and their use to fill different fields in the migration
entity_lookup,skip_on_emptyandexplode- Example
field_adoption_countries_ns: # this field is needed in case country names do not follow the UN/FAO - Standards - plugin: skip_on_empty method: process source: <use cases column or field> - plugin: explode delimiter: '-' - plugin: entity_generate entity_type: taxonomy_term ignore_case: true value_key: name bundle: countries_no_standard- Explanation:
- Multiple plugins can be chained like above. The execution order is top-to-bottom. The first plugin must receive the
sourcedata column or field. skip_on_emptyis a sanity check. If the field/column is not filled for this row or entry, it will simply not be filled (and skipped). This avoids errors when migrating data with empty fields. You need to give it amethod:(process or row)explodeis used if a field/column contains multiple values. In a csv file it might be that ‘|’ or ‘-’ are used as a separator for a ‘unclean’ field containing more than one valueentity_generategenerates a new taxonomy term if no exactly matching one can be found (e.g. data uses ‘FAO’ while taxonomy expects ‘Food and Agriculture Organization of the United Nations’. That can either be cleaned up later by manually replacing or one uses ai-mapping logic like is done for thefield_adoption_countries_nsentry. It finds the closest real taxonomy term and automatically replaces them
- Multiple plugins can be chained like above. The execution order is top-to-bottom. The first plugin must receive the
static_map- Example:
- # Often it is necessary to map however sdgs are named in the source data to the STI portal taxonomy plugin: static_map map: '1': 'Goal 1: No poverty' '2': 'Goal 2: Zero hunger' '3': 'Goal 3: Good health and well-being' '4': 'Goal 4: Quality education' '5': 'Goal 5: Gender equality' '6': 'Goal 6: Clean water and sanitation' '7': 'Goal 7: Affordable and clean energy' '8': 'Goal 8: Decent work and economic growth' '9': 'Goal 9: Industry, innovation and infrastructure' '10': 'Goal 10: Reduced inequalities' '11': 'Goal 11: Sustainable cities and communities' '12': 'Goal 12: Responsible consumption and production' '13': 'Goal 13: Climate action' '14': 'Goal 14: Life below water' '15': 'Goal 15: Life on land' '16': 'Goal 16: Peace, justice and strong institutions' '17': 'Goal 17: Partnerships for the goals' default_value: ''- Explanation: This plugin works if you can guarantee a one-to-one mapping of how data entries are written in the data source to how they are represented in the taxonomy. On the left-hand side of the colon put the way the entry is written in the data source, on the right side the way its written in the taxonomy. Right-hand side seems to ignore case, but make sure that the entries are written exactly matching to the taxonomy terms.
entity_lookup- Example
field_type_of_information_manage: plugin: entity_lookup entity_type: taxonomy_term ignore_case: true value_key: name source: constants/RECTYPE bundle: source_entity_types- Explanation: Fill in the corresponding term fromt he taxonomy found in the corresponding bundle. This needs to be an exact match, i.e. this step is done after
static_mapor using a pre-defined constant like in the example. - to find the correct machine name for the bundle in question go to Web menu -> About -> Taxonomies -> <click on taxonomy> -> <click on the taxonomy name/title again>. You should find the Machine name next to the title
Content workflow; Imports
- This commands lists all content types’ machine names
drush eval "print_r(array_keys(\Drupal::entityTypeManager()->getStorage('node_type')->loadMultiple()));"
Array
(
[0] => asti_data
[1] => definitions
[2] => digital_asset
[3] => extracted_entity
[4] => initiative
[5] => innovation
[6] => innovation_core
[7] => innovation_extracted_from_origin
[8] => internal_content
[9] => learning_resource
[10] => organization
[11] => source_record
[12] => taxonomy_description
[13] => web_page
[14] => website_section
)
- To find out which fields are available for each content type, run
drush field:info node <e.g. source_record>
- This will give you an idea which information should ideally already be present when importing the data
- To understand what those fields are supposed to contain, you can consult the taxonomy page (Web menu -> About -> Taxonomies -> <e.g. Use Cases>
- This will help you understand if the data source of choice has a matching taxonomy
From Original record to innovation
- Web menu -> BACK-END -> manage data sources
- Search: <your data source>
- Edit tag: STI portal data source (also ATIO, should the data be imported into ATIO)
- Publish (I encountered problems with the AI-enhancement if I didn’t do this)
- select if
- can be overwritten by AI
- allow overwriting by original record
- allow overwriting by extracted innovation record
- Web menu -> BACK-END -> Workflows for original records
- Source: <your data source> (should auto-complete at this point)
- filter
- select all
- generate/update extracted innovations from original records
- Web menu -> BACK-END -> Workflows for ai-extracted innovations
- Enrich derived innovation record with AI if empty (settings allow a bit less cautious)
- max 250, better 50 at a time (otherwise you risk a timeout error)
- if not loading, change the ‘start’ in the url for ‘stop’:
https://sti-portal-prototype.net/stiportal_dev/web/batch?id=3342&op=start->https://sti-portal-prototype.net/stiportal_dev/web/batch?id=3342&op=stop
- Generate/update innovation records from AI extracted records
- to see what can go wrong & how to fix it: Error log: AI enhancement
- Check that the innovations are displayed correctly in the website if you open your data source’s collection page
- Enrich derived innovation record with AI if empty (settings allow a bit less cautious)
Notes
- Make sure that there are no duplicates in the source data. This means whatever field is used as ‘id’ is truly unique. Good candidates are project numbers, urls or, if no alternative, the full title
- ‘Titles’ have a character limit
Error log: AI enhancement
Data source not found in exception list
Error:
Check action successor current_prov_id (Activity_12o35fp) from ECA VBO - Generate / update AI derived innovation records from original records (process_yoqnpd7) for event eca_vbo.execute. - session_user (Entity user/46/wiessalla) - entity (Entity node/source_record/35689/ Corte (Poda) das folhas do coqueiro na região de Bicol) - node (Entity node/source_record/35689/ Corte (Poda) das folhas do coqueiro na região de Bicol) - except_providers_view (DTO) - 0 (Entity node/digital_asset/28182/Country Annual Report (CAR)) - 1 (Entity node/digital_asset/28184/Digital Agriculture Programme Priority Area (BP5) ) - 2 (Entity node/digital_asset/25260/Seeding The Future Global Food System Innovation Database and Network) - 3 (Entity node/digital_asset/19987/Technologies for African Agricultural Transformation (TAAT)) - 4 (Entity node/digital_asset/20007/World Overview of Conservation Approaches and Technologies (WOCAT)) - exception_provider (DTO "0") - exceptions_count (DTO "0") - provider_id_read (DTO "28182") - provider_id (DTO "28182") - exception_providers_list (DTO) - 0 (DTO "20007") - 1 (DTO "19987") - 2 (DTO "25260") - 3 (DTO "28184") - 4 (DTO "28182") - orig_rec (Entity node/source_record/35689/ Corte (Poda) das folhas do coqueiro na região de Bicol) - rec_sources (DTO) - 0 (Entity node/digital_asset/4/FAO Technologies and Practices for Small Agricultural Producers (TECA)) - counter (DTO "-17243") - current_prov_id (NULL) - user (Entity user/1/admin) - event (DTO) - view (DTO) - id (string "backend2") - display_id (string "page_7") - action (DTO) - plugin (string "eca_vbo_execute:generate_update_extracted_innovations_from_original_records") - config (DTO) - operation_name (string "Generate / update extracted innovations from original records") - message_override (string "") - skip_confirm (integer "0") - entity (DTO) - id (string "35689") - label (string " Corte (Poda) das folhas do coqueiro na região de Bicol") - type (string "node") - bundle (string "source_record") - langcode (string "en") - machine_name (string "eca_vbo.execute")- Explanation: In the example I was trying to add new AI-extracted innovations to the provider ‘TECA’ with the
provider_id4. This id was not found and not added to theexception_providers_list. The import enters an infinite loop and fails with HTTP Error 500. - Solution: For me, setting the data source’s status to ‘published’ worked
- Explanation: In the example I was trying to add new AI-extracted innovations to the provider ‘TECA’ with the
Cannot access offset of type string on string
- Error:
ResponseText: The website encountered an unexpected error. Try again later. TypeError: Cannot access offset of type string on string in Drupal\ai_automators\PluginBaseClasses\Boolean->verifyValue() (line 94 of modules/contrib/ai/modules/ai_automators/src/PluginBaseClasses/Boolean.php). - the same error is thrown on line 110
- Explanation: Some boolean elements in the ECA (such as whether or not the field ‘overwrite existing entries by AI’ is checked in the data source settings) are apparently passed as strings through the ECA. The Boolean.php of the
ai_automatorsplugin (web/modules/contrib/ai/modules/ai_automators/src/PluginBaseClasses/Boolean.php)
- Explanation: Some boolean elements in the ECA (such as whether or not the field ‘overwrite existing entries by AI’ is checked in the data source settings) are apparently passed as strings through the ECA. The Boolean.php of the
takes only arrays in line 94 and 110.
- Solution: As a hotfix I forced casting every value that is not an array to an array. That seems to work for now
public function verifyValue(ContentEntityInterface $entity, $value, FieldDefinitionInterface $fieldDefinit ion, array $automatorConfig) { // Has to be string boolean. if (!is_array($value)) { #! changed $value = ['value' => $value]; } if (!in_array($value['value'], ['TRUE', 'FALSE', '0', '1', 0, 1])) { return FALSE; } // Otherwise it is ok. return TRUE; } /** * {@inheritDoc} */ public function storeValues(ContentEntityInterface $entity, array $values, FieldDefinitionInterface $field Definition, array $automatorConfig) { // Transform string to boolean. foreach ($values as $key => $value) { if (!is_array($value)) { #! changed $value = ['value' => $value]; } $values[$key] = in_array($value['value'], ['TRUE', '1', 1]) ? TRUE : FALSE; } // Then set the value. $entity->set($fieldDefinition->getName(), $values); return TRUE; }
OpenAI API doesn’t handle strings
- Error:
TypeError: OpenAI\Responses\Chat\CreateResponse::from(): Argument #1 ($attributes) must be of type array, string given, called in /home/stiprototype/public_html/stiportal_dev/vendor/openai-php/client/src/Resources/Chat.php on line 35 in OpenAI\Responses\Chat\CreateResponse::from() (line 46 of /home/stiprototype/public_html/stiportal_dev/vendor/openai-php/client/src/Responses/Chat/CreateResponse.php). - Explanation: Like in this error the ECA that does the AI-enrichment seems to pass a string where an array is expected
- Solution:
- Changing the php code and forcing strings to array could work
- Fundamentally, the issue should be addressed in the ECA
- Changing the API from OpenAI to Anthropic avoids the issue so I did this
Examples of formatting errors when running a migration
Whitespaces and different languages
- Error:
#+begin_src 4536 1 teca:field_information_resource_date:format_date: Format date plugin could not transform 'Augst 2006' using the format 'F Y'. Error: The date cannot be created from a format. 8363 1 teca:field_information_resource_date:format_date: Format date plugin could not transform ' 2015' using the format 'F Y'. Error: The date cannot be created from a format. 8653 1 teca:field_information_resource_date:format_date: Format date plugin could not transform 'February 2016 ' using the format 'F Y'. Error: The date cannot be created from a format. 8707 1 teca:field_information_resource_date:format_date: Format date plugin could not transform ' April 2016 ' using the format 'F Y'. Error: The date cannot be created from a format. 2471 1 teca:field_information_resource_date:format_date: Format date plugin could not transform 'August 2015 ' using the format 'F Y'. Error: The date cannot be created from a format. 2699 1 teca:field_information_resource_date:format_date: Format date plugin could not transform 'May 2013 ' using the format 'F Y'. Error: The date cannot be created from a format. 2019 1 teca:field_information_resource_date:format_date: Format date plugin could not transform 'May 2011 ' using the format 'F Y'. Error: The date cannot be created from a format. 2466 1 teca:field_information_resource_date:format_date: Format date plugin could not transform 'March 2005 ' using the format 'F Y'. Error: The date cannot be created from a format. 2555 1 teca:field_information_resource_date:format_date: Format date plugin could not transform 'March 2018 ' using the format 'F Y'. Error: The date cannot be created from a format. 10038 1 teca:field_information_resource_date:format_date: Format date plugin could not transform 'May 2015 ' using the format 'F Y'. Error: The date cannot be created from a format. 10126 1 teca:field_information_resource_date:format_date: Format date plugin could not transform ' April 2021' using the format 'F Y'. Error: The date cannot be created from a format. 10105 1 teca:field_information_resource_date:format_date: Format date plugin could not transform 'Février 2015' using the format 'F Y'. Error: The date cannot be created from a format. 10106 1 teca:field_information_resource_date:format_date: Format date plugin could not transform 'Février 2015' using the format 'F Y'. Error: The date cannot be created from a format. - Explanation:
- Some entries don’t follow the general formatting of ‘F Y’ (written month in English and Year)
- Some entries have trailing or leading whitespaces. In this particular case the
trimfunction of Drupal migrate didn’t remove them, because they are non-standard whitespaces - Some Month names are written in French
- Solution: In this case it was only a handful of entries and I fixed them manually. In general this should be flagged to whoever was/is curating the original data
Gemini API changed
- this seems to be the case quite often
Drupal stuck looking for a broken twig files
- Error:This manifested itself in completely blocking all DB operations (
2285210 05/Nov 12:55 php Error Twig\Error\LoaderError: Template ".html.twig" is not defined. in Twig\Loader\ChainLoader->getCacheKey() (line 38 of /var/www/html/web/themes/custom/fao/templates/block/block--views.h tml.twigcim,cex,cr) and resulting in Drupal not being able to serve the site anymore
- Explanation: Whenever Drupal renders anything — even an admin page, a Drush command, or a cron job — it loads the theme registry: A huge PHP array mapping every template name to the file that implements it. That registry is stored in the
cache_discoverytable and rebuilt whenever:- You clear caches (drush cr)
- You export/import config (drush cex / cim)
- You install or update modules/themes
So even Drush CLI commands (which don’t render HTML) still invoke the theme system during bootstrapping.
- Solution:
Remove the offending file and rebuild the cache
rm /var/www/html/web/themes/custom/fao/templates/block/block--views.html.twig
drush cr
- In the future
drush theme:debugcould point out those errors before loading the theme into the cache
How to change taxonomy terms drupal
- Web menu -> Structure -> Taxonomy -> <AFS innovation use cases>
Custom Taxonomy
- Structure -> Taxonomy -> Create vocabulary
- Add terms manually one by one
- Faster alternative (deactivated): Extend -> Taxonomy Manager
- Install
- Structure -> Taxonomy Manager -> <new category> -> paste \n - separated list
- Structure -> Content types -> original record -> create new fields
- Structure -> Content types -> ai-extracted record -> create new fields
- Structure -> Content types -> innovation -> create new fields
- Change ECA; add the new
- Faster alternative (deactivated): Extend -> Taxonomy Manager
Installing custom modules drupal
- The default location for custom modules is
web/modules/custom/<your module> - The minimum set of files is
<your module>.info.yml: Metadata and description of the module dependencies. What is the module?<your module>.libraries.yml: Describes location of css/js and other library files. Lists other plugin dependencies.<your module>.module(php syntax): Optional. Adds a php hook (function) controlling when and how the library will be activated
- Additionally, you might have a
cssandjsfolder in the module structure providing custom code and styling - The steps to activate your module are
- Install all necessary dependencies using
composer requireanddrush en - Create/write your module file structure and put it in
modules/custom drush en <your module> -y
- Install all necessary dependencies using
- If your module is changing the behavior of other custom modules you should add it in the
dependencies:field there. E.g. if you change theming, just add your module to your custom themes.libraries.ymlfile
Project structure
- A minimal working example project structure could be:
web/modules/custom/sti_leaflet/
├── js/
│ └── custom_cluster_icons.js
├── sti_leaflet.info.yml
├── sti_leaflet.libraries.yml
└── sti_leaflet.module
- Additional you might need
- /css → Custom styling
- /js → JavaScript behaviors or logic
- /templates → Twig overrides
- /src → PHP classes (services, plugins, controllers, etc.)
Setting up a new server instance
Now, all user-changed drupal code (mostly js, twg, yml and php/module) is now on FAO’s github repository. ‘Main’ is currently tracking our ‘review’ server.
Server roles
| Server Name | Role | Hosted |
|---|---|---|
| test | playground; obsolete | private |
| dev | staging ground for changes made trough gui | private |
| review | mirrors production, test for changes pulled from dev | gcloud |
| production | user-facing, pulls only from review | gcloud |
Steps to get new server instance
- Precondition is that the necessary tech stack is installed (php, drupal, etc). Here a link to a [[][shell.nix development environment]]
# Get upstream config
git clone git@github.com:valeriapesce/ATIO-KB-harvesting-IRD.git
# Get the non-tracked files (images, etc)
./sync.sh <user>@review:<path to drupal root> fao-sti-portal
# Install dependencies speciefied in composer.lock
composer install --no-interaction --prefer-dist
# Run the server
vendor/bin/drush serve 127.0.0.1:5050
sync.sh file
Under development, will be added to the github repo
#!/usr/bin/env bash
#
# Safely sync environment-specific Drupal config and file directories
# from one local Drupal root to another.
#
# Usage:
# ./sync.sh /path/to/source/drupal/root /path/to/target/drupal/root
#
set -euo pipefail
if [[ $# -ne 2 ]]; then
echo "Usage: $0 /path/to/source/drupal/root /path/to/target/drupal/root"
exit 1
fi
SRC=$(realpath "$1")
DST=$(realpath "$2")
if [[ ! -d "$SRC/web/sites" ]]; then
echo "Error: Source Drupal root '$SRC' does not appear to contain web/sites/"
exit 1
fi
mkdir -p "$DST"
echo ">>> Syncing from:"
echo " $SRC"
echo ">>> To:"
echo " $DST"
echo
# Explicit file list (relative to Drupal root)
FILES=(
"web/sites/config_UayHOzMAbX_oN-5FQkTXdB6Ot6tWDOFLY6ljN5FFOQ0bKajw9MX6UFRsu0IsRTqQ-C9vQ4_kng/sync/key.key.anthropic_atiokb.yml"
"web/sites/config_UayHOzMAbX_oN-5FQkTXdB6Ot6tWDOFLY6ljN5FFOQ0bKajw9MX6UFRsu0IsRTqQ-C9vQ4_kng/sync/key.key.google_ai.yml"
"web/sites/config_UayHOzMAbX_oN-5FQkTXdB6Ot6tWDOFLY6ljN5FFOQ0bKajw9MX6UFRsu0IsRTqQ-C9vQ4_kng/sync/key.key.milvus_zilliz.yml"
"web/sites/config_UayHOzMAbX_oN-5FQkTXdB6Ot6tWDOFLY6ljN5FFOQ0bKajw9MX6UFRsu0IsRTqQ-C9vQ4_kng/sync/key.key.openai_api_key.yml"
)
# Directory patterns
DIRS=(
"web/sites/*/*settings*.php"
"web/sites/*/*services*.yml"
"web/sites/*/files"
"web/sites/*/public"
"web/sites/*/private"
"web/sites/*/files-public"
"web/sites/*/files-private"
)
echo ">>> Starting sync..."
echo
# Copy individual files (if missing)
for relpath in "${FILES[@]}"; do
src="$SRC/$relpath"
dst="$DST/$relpath"
if [[ -f "$src" ]]; then
mkdir -p "$(dirname "$dst")"
if [[ ! -f "$dst" ]]; then
echo "Copying file: $relpath"
rsync -a "$src" "$dst"
else
echo "Skipping (already exists): $relpath"
fi
else
echo "Missing source file: $relpath"
fi
done
# Copy directories and pattern-matched files
for pattern in "${DIRS[@]}"; do
for path in "$SRC"/$pattern; do
if [[ -e "$path" ]]; then
relpath="${path#$SRC/}"
src="$path"
dst="$DST/$relpath"
mkdir -p "$(dirname "$dst")"
echo "Syncing: $relpath"
rsync -a --ignore-existing "$src" "$dst"
fi
done
done
echo
echo ">>> Sync complete!"
- I had to add this, since there still is a bug in the script
mv sites/default/files/files/* sites/default/files
mv sites/default/files/files/.* sites/default/files
VS Code configuration to connect to gcloud
There are a couple of ways to edit your ~/.ssh/config file so that the Remote Explorer in VSCode connects to your google cloud instance.
Direct editing using the google cloud sdk
gcloud compute config-ssh
This command writes host entries to your local ~/.ssh/config like:
Host my-instance.us-central1-a.my-project
HostName 34.72.x.x
IdentityFile ~/.ssh/google_compute_engine #this file exists after connecting to the vm the first time
User my-username
CheckHostIP no
StrictHostKeyChecking no
So you can just `ssh my-instance.us-central1-a.my-project` directly without remembering zones or IPs. See: https://cloud.google.com/sdk/gcloud/reference/compute/config-ssh
Using a Proxy Command in the ssh config
TUNNEL_PORT=2222
VM_NAME=drupal-vm
PROJECT=fao-sti-review
ZONE=europe-west1-b
Host $VM_NAME
User <your remote user name>
IdentityFile ~/.ssh/google_compute_engine
ProxyCommand gcloud compute start-iap-tunnel %h 22 --listen-on-stdin --project=$PROJECT --zone=$ZONE
Opening a tunnel and specifying the tunnel’s port on localhost
If you specify your ssh connection in the config like:
Host drupal-fao
User <your remote user name>
HostName localhost
IdentityFile /home/<your local user name>/.ssh/google_compute_engine
Port 2222
ForwardAgent yes
Then you can open the tunnel like this:
TUNNEL_PORT=2222
VM_NAME=drupal-vm
PROJECT=fao-sti-review
ZONE=europe-west1-b
gcloud compute start-iap-tunnel "$VM_NAME" 22 \
--local-host-port=localhost:$TUNNEL_PORT \
--project="$PROJECT" \
--zone="$ZONE" \
--quiet
Afterwards you can connect through vscode and ssh drupal-fao works automatically
Set up new instance
- Prepare mysql
CREATE USER 'drupal_user'@'localhost' IDENTIFIED BY <password as in settings.php>;
GRANT ALL PRIVILEGES ON drupal.* TO 'drupal_user'@'localhost';
FLUSH PRIVILEGES;
CREATE DATABASE drupal;
- Get the code
git clone git@github.com:un-fao/fao-sti-portal.git
- Get the files
sync.sh fao-review:/var/www/html/ ./sti-portal-fao #replace fao-review with whatever is in your .ssh/config
- Get the DB
# Get the latest DB dump here: https://console.cloud.google.com/storage/browser/fao-sti-review-backups;tab=objects?q=search&referrer=search&project=fao-sti-review&prefix=&forceOnObjectsSortingFiltering=false
mysql -u drupal_user -p drupal < ~/Downloads/<your recent backup>.sql
- Finish
vendor/bin/drush cr
vendor/bin/drush serve 127.0.0.1:8080
Styling
New view display
- Task: Change the layout of the popup cards in the leaflet map view. (Remove the image).
- Steps to solve:
- Create a
View Modelangcode: en status: true dependencies: module: - node id: node.map_teaser label: 'Map teaser' description: 'Used for Leaflet popups.' targetEntityType: node cache: true - Create a
Entity View Displaylangcode: en status: true dependencies: config: - core.entity_view_mode.node.map_teaser - field.field.node.innovation.ai_automator_status - field.field.node.innovation.body - field.field.node.innovation.field_actual_users - field.field.node.innovation.field_adoptability_dimensions - field.field.node.innovation.field_adoption_description - field.field.node.innovation.field_adoption_level - field.field.node.innovation.field_afs_stf_areas - field.field.node.innovation.field_ai_stamp - field.field.node.innovation.field_allow_ai_update - field.field.node.innovation.field_allow_ai_update_if_empty - field.field.node.innovation.field_allow_overwr_if_empty - field.field.node.innovation.field_allow_overwriting_by_origi - field.field.node.innovation.field_areas_of_expertise - field.field.node.innovation.field_areas_of_need - field.field.node.innovation.field_atio_innovation_category - field.field.node.innovation.field_atio_kb_source - field.field.node.innovation.field_award_year - field.field.node.innovation.field_bib_citation_of_pub - field.field.node.innovation.field_challenges_addressed - field.field.node.innovation.field_city - field.field.node.innovation.field_collection - field.field.node.innovation.field_countries_adoption - field.field.node.innovation.field_country_origin - field.field.node.innovation.field_curation_notes - field.field.node.innovation.field_data_source - field.field.node.innovation.field_description_markup - field.field.node.innovation.field_display_mode - field.field.node.innovation.field_entity_creation_mode - field.field.node.innovation.field_environmental_impact - field.field.node.innovation.field_ffs_type - field.field.node.innovation.field_grassroot_partner - field.field.node.innovation.field_grassroots_innovation_narr - field.field.node.innovation.field_human_impact - field.field.node.innovation.field_identifying_feature - field.field.node.innovation.field_if_grassroots - field.field.node.innovation.field_impact_sdgs - field.field.node.innovation.field_inclusion_and_representati - field.field.node.innovation.field_info_on_partial_editing - field.field.node.innovation.field_information_resource_date - field.field.node.innovation.field_innovation_domain - field.field.node.innovation.field_innovation_stage - field.field.node.innovation.field_innovation_type - field.field.node.innovation.field_innovative_elements_narrat - field.field.node.innovation.field_innovative_features - field.field.node.innovation.field_investment_type - field.field.node.innovation.field_languages - field.field.node.innovation.field_long_term_beneficiaries - field.field.node.innovation.field_original_descr_markup - field.field.node.innovation.field_original_extracted_innovat - field.field.node.innovation.field_owner - field.field.node.innovation.field_partners - field.field.node.innovation.field_prospective_users - field.field.node.innovation.field_readiness_description - field.field.node.innovation.field_readiness_level - field.field.node.innovation.field_record_long_metadata - field.field.node.innovation.field_record_metadata - field.field.node.innovation.field_record_status - field.field.node.innovation.field_region - field.field.node.innovation.field_related_innovation_core - field.field.node.innovation.field_scaling_level_idia - field.field.node.innovation.field_shorter_description - field.field.node.innovation.field_sponsors - field.field.node.innovation.field_start_date - field.field.node.innovation.field_start_year - field.field.node.innovation.field_state - field.field.node.innovation.field_stf_development_status - field.field.node.innovation.field_stf_level_of_recognition - field.field.node.innovation.field_stf_organization_size - field.field.node.innovation.field_stf_organization_type - field.field.node.innovation.field_thematic_area - field.field.node.innovation.field_title_instructions - field.field.node.innovation.field_url_s - field.field.node.innovation.field_use_case_image - field.field.node.innovation.field_use_cases - field.field.node.innovation.field_use_cases_description - node.type.innovation module: - text id: node.innovation.map_teaser targetEntityType: node bundle: innovation mode: map_teaser content: field_ai_disclaimer: type: markup label: hidden settings: { } third_party_settings: { } weight: 2 region: content field_country_origin: type: entity_reference_label label: above settings: link: true third_party_settings: { } weight: 2 region: content field_data_source: type: entity_reference_label label: above settings: link: true third_party_settings: { } weight: 3 region: content field_shorter_description: type: text_default label: above settings: { } third_party_settings: { } weight: 1 region: content title: type: string label: above settings: link_to_entity: false third_party_settings: { } weight: 0 region: content hidden: ai_automator_status: true body: true content_moderation_control: true created: true field_actual_users: true field_adoptability_dimensions: true field_adoption_description: true field_adoption_level: true field_afs_stf_areas: true field_ai_stamp: true field_allow_ai_update: true field_allow_ai_update_if_empty: true field_allow_overwr_if_empty: true field_allow_overwriting_by_origi: true field_areas_of_expertise: true field_areas_of_need: true field_atio_innovation_category: true field_atio_kb_source: true field_award_year: true field_bib_citation_of_pub: true field_challenges_addressed: true field_city: true field_collection: true field_countries_adoption: true field_curation_notes: true field_description_markup: true field_display_mode: true field_entity_creation_mode: true field_environmental_impact: true field_ffs_type: true field_grassroot_partner: true field_grassroots_innovation_narr: true field_human_impact: true field_identifying_feature: true field_if_grassroots: true field_impact_sdgs: true field_inclusion_and_representati: true field_info_on_partial_editing: true field_information_resource_date: true field_innovation_domain: true field_innovation_stage: true field_innovation_type: true field_innovative_elements_narrat: true field_innovative_features: true field_investment_type: true field_languages: true field_long_term_beneficiaries: true field_original_descr_markup: true field_original_extracted_innovat: true field_owner: true field_partners: true field_prospective_users: true field_readiness_description: true field_readiness_level: true field_record_long_metadata: true field_record_metadata: true field_record_status: true field_region: true field_related_innovation_core: true field_scaling_level_idia: true field_sponsors: true field_start_date: true field_start_year: true field_state: true field_stf_development_status: true field_stf_level_of_recognition: true field_stf_organization_size: true field_stf_organization_type: true field_thematic_area: true field_title_instructions: true field_url_s: true field_use_case_image: true field_use_cases: true field_use_cases_description: true field_variant: true langcode: true links: true search_api_excerpt: true synonyms: true uid: true - Manage the content display (
admin/structure/types/manage/innovation/display/)- If you imported working configs like above not needed
- Check that under tab
default, your new display is checked underCustom display settings(all the way at the bottom of the page)
- Apply a corresponding twig template, in this case:
/var/www/html/web/themes/custom/fao/templates/content/node--innovation--map-teaser.html.twig{# /** * @file * Default theme implementation to display a node. * * Available variables: * - node: The node entity with limited access to object properties and methods. * Only method names starting with "get", "has", or "is" and a few common * methods such as "id", "label", and "bundle" are available. For example: * - node.getCreatedTime() will return the node creation timestamp. * - node.hasField('field_example') returns TRUE if the node bundle includes * field_example. (This does not indicate the presence of a value in this * field.) * - node.isPublished() will return whether the node is published or not. * Calling other methods, such as node.delete(), will result in an exception. * See \Drupal\node\Entity\Node for a full list of public properties and * methods for the node object. * - label: (optional) The title of the node. * - content: All node items. Use {{ content }} to print them all, * or print a subset such as {{ content.field_example }}. Use * {{ content|without('field_example') }} to temporarily suppress the printing * of a given child element. * - author_picture: The node author user entity, rendered using the "compact" * view mode. * - metadata: Metadata for this node. * - date: (optional) Themed creation date field. * - author_name: (optional) Themed author name field. * - url: Direct URL of the current node. * - display_submitted: Whether submission information should be displayed. * - attributes: HTML attributes for the containing element. * The attributes.class element may contain one or more of the following * classes: * - node: The current template type (also known as a "theming hook"). * - node--type-[type]: The current node type. For example, if the node is an * "Article" it would result in "node--type-article". Note that the machine * name will often be in a short form of the human readable label. * - node--view-mode-[view_mode]: The View Mode of the node; for example, a * teaser would result in: "node--view-mode-teaser", and * full: "node--view-mode-full". * The following are controlled through the node publishing options. * - node--promoted: Appears on nodes promoted to the front page. * - node--sticky: Appears on nodes ordered above other non-sticky nodes in * teaser listings. * - node--unpublished: Appears on unpublished nodes visible only to site * admins. * - title_attributes: Same as attributes, except applied to the main title * tag that appears in the template. * - content_attributes: Same as attributes, except applied to the main * content tag that appears in the template. * - author_attributes: Same as attributes, except applied to the author of * the node tag that appears in the template. * - title_prefix: Additional output populated by modules, intended to be * displayed in front of the main title tag that appears in the template. * - title_suffix: Additional output populated by modules, intended to be * displayed after the main title tag that appears in the template. * - view_mode: View mode; for example, "teaser" or "full". * - teaser: Flag for the teaser state. Will be true if view_mode is 'teaser'. * - page: Flag for the full page state. Will be true if view_mode is 'full'. * * @see template_preprocess_node() * * @ingroup themeable */ #} {% set classes = [ 'card', 'card-events' ] %} {{ attach_library('fao/maps') }} <article{{ attributes.addClass(classes) }}> {{ title_prefix }} {{ title_suffix }} {# <div class="card-image ratio ratio-3x2"> #} {# {{ content.field_use_case_image }} #} {# </div> #} <div class="card-body"> <h6 class="title-category "> {{ node.field_innovation_type.0.entity.name.value }} </h6> <h5 class="card-title"><a href="{{ url }}" class="title-link">{{ node.title.value }}</a></h5> <h6 class="date-location card-date-location"> <span class="location"> {% if content.field_country_origin|render|trim %} <i class="bi bi-geo-alt-fill"></i> {{ content.field_country_origin }} {% endif %} <span class="date">{{ node.getCreatedTime()|date('d/m/Y') }}</span> </span> </h6> <div class="card-text"> {{ content.field_shorter_description }} </div> <div class="classifications card-classifications"> {{ content.field_innovation_type }} {{ content.field_use_cases }} {% if sdgs %} <div class="sdg-list"> {% for sdg in sdgs %} <a href="{{ sdg.url }}" class="sdg-small text-white text-decoration-none {{ sdg.class }}"></a> {% endfor %} </div> {% endif %} </div> {{ content.field_data_source }} </div> </article> {# Custom template for Innovation nodes in Map teaser view mode. {% set classes = ['card-events', 'card-map-popup'] %} <article{{ attributes.addClass(classes) }}> <h5 class="popup-title"> {{ label }} </h5> <div class="popup-body"> {{ content.field_shorter_description }} {{ content.field_innovation_type }} {{ content.field_use_cases }} </div> </article> #} - Fine-tune what elements are shown on the card, by moving items to the ‘Field’ section at
admin/structure/types/manage/innovation/display/
- Create a
- Explanation:
Drupal styling goes through a couple of steps: The content display determines which elements are displayed, and twig then styles those elements
it seems twig can also override this behavior and automatically display certain elements even if they’re not selected in the content display
Make search bar scrollable
- Can e.g. be achieved by this:
.view-filters { max-height: 100vh; overflow-y: auto; position: sticky; top: 0; padding-right: 10px; overflow-x: hidden; }
Apache config
Maintaining two drupal installs and resulting sites requires to set up some wiring in apache, in order to have two meaningful addresses.
Path resolution
- configure
/etc/apache2/sites-enabled/000-default.conf - every drupal root needs to have its own Directory
<VirtualHost *:80>
# The ServerName directive sets the request scheme, hostname and port that
# the server uses to identify itself. This is used when creating
# redirection URLs. In the context of virtual hosts, the ServerName
# specifies what hostname must appear in the request's Host: header to
# match this virtual host. For the default virtual host (this file) this
# value is not decisive as it is used as a last resort host regardless.
# However, you must set it for any further virtual host explicitly.
#ServerName www.example.com
ServerAdmin webmaster@localhost
DocumentRoot /var/www/html/sti-portal/web
<Directory /var/www/html>
Options FollowSymLinks MultiViews
AllowOverride None
Require all granted
</Directory>
<Directory /var/www/html/sti-portal/html>
Options FollowSymLinks MultiViews
AllowOverride All
Require all granted
</Directory>
<Directory /var/www/html/sti-social/html>
Options FollowSymLinks MultiViews
AllowOverride All
Require all granted
</Directory>
# The path after alias is the path relative to root that will be accessible
Alias /network /var/www/html/sti-social/html
<Directory /var/www/html/sti-social/html>
Options FollowSymLinks MultiViews
AllowOverride All
Require all granted
</Directory>
# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
#LogLevel info ssl:warn
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# For most configuration files from conf-available/, which are
# enabled or disabled at a global level, it is possible to
# include a line for only one particular virtual host. For example the
# following line enables the CGI configuration for this host only
# after it has been globally disabled with "a2disconf".
#Include conf-available/serve-cgi-bin.conf
</VirtualHost>
Inststall Drupal Page
sudo -u www-data vendor/bin/drush --root=html --uri="http://localhost/network" cr
Add a custom block
- Create the block under
/admin/structure/block-content - Add a block instance under
/block/add/hero_submit_idea - Find the block to set under
/admin/structure/block
Install open social
For version 13 you need php3.8-fhm
curl -fsSL https://packages.sury.org/php/apt.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/sury-php.gpg echo "deb [signed-by=/etc/apt/keyrings/sury-php.gpg] https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/sury-php.list sudo apt update sudo apt install -y php8.3-fpm libapache2-mod-fcgid php8.3-xml php8.3-mysql php8.3-gd php8.3-mbstring php8.3-intl php8.3-curl sudo a2enmod proxy_fcgi setenvif php8.3-fpm sudo a2disconf php8.2-fpm 2>/dev/null || true sudo systemctl enable --now php8.3-fpm sudo systemctl restart php8.3-fpm sudo systemctl restart apache2Set up the site & DB
sudo -u www-data vendor/bin/drush --root=html --sites-subdir=default -y site:install social --db-url='mysql://social:social@127.0.0.1:3306/social'Load in old db
zcat /home/tristan/db.sql.gz | vendor/bin/drush --root=html sqlcConfigure apache
cat /etc/apache2/sites-enabled/000-default.conf <VirtualHost *:80> # The ServerName directive sets the request scheme, hostname and port that # the server uses to identify itself. This is used when creating # redirection URLs. In the context of virtual hosts, the ServerName # specifies what hostname must appear in the request's Host: header to # match this virtual host. For the default virtual host (this file) this # value is not decisive as it is used as a last resort host regardless. # However, you must set it for any further virtual host explicitly. #ServerName www.example.com ServerAdmin webmaster@localhost DocumentRoot /var/www/html/sti-portal/web <Directory /var/www/html/sti-portal/web> Options FollowSymLinks MultiViews AllowOverride All Require all granted </Directory> Alias /network /var/www/html/sti-social/html <Directory /var/www/html/sti-social/html> Options FollowSymLinks MultiViews AllowOverride All Require all granted </Directory> # Available loglevels: trace8, ..., trace1, debug, info, notice, warn, # error, crit, alert, emerg. # It is also possible to configure the loglevel for particular # modules, e.g. #LogLevel info ssl:warn ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined # For most configuration files from conf-available/, which are # enabled or disabled at a global level, it is possible to # include a line for only one particular virtual host. For example the # following line enables the CGI configuration for this host only # after it has been globally disabled with "a2disconf". #Include conf-available/serve-cgi-bin.conf # Make Drupal think the original request was HTTPS externally. <Location "/network"> RequestHeader set X-Forwarded-Proto "https" RequestHeader set X-Forwarded-Port "443" RequestHeader set X-Forwarded-Host "sti-portal.edge.faofao.org" </Location> </VirtualHostcheck config with
apachectl configtestenable with
sudo systemctl restart apache2
Kiautschou str 6
Solution
- adding
kdePackages.akonadi-calendarto my config and re-loading plasma (systemctl --user restart plasma-plasmashell.service) solved the issue
Resources
- commented on github: https://github.com/NixOS/nixpkgs/issues/344025