Failed syncing nested fields with Strapi plugin

[oluademola]

Discussed in meilisearch/meilisearch#2388

^{Originally posted by pedrogaudencio May 11, 2022}
Hi there!

I’m having an issue when I update a collection-type entry that is nested inside a component/single type/collection. Not sure if this case is already referenced but I couldn't find it anywhere.

Example:

1. Create an Author collection type entry -> Meilisearch plugin creates an Author document;
2. Create an Article collection type entry, add an Author collection type field inside the repeatable component AuthorsList in the Article -> Meilisearch plugin creates an Article document with its nested fields;
3. Update the Author entry -> Meilisearch plugin updates the Author document;
4. Meilisearch plugin does not update the Author entry inside the nested fields in the Article document.

As a workaround, (in this example) I thought of creating an afterUpdate() lifecycle hook in the Author triggering the Article document update by calling the Meilisearch plugin controller. Is there a better way to do this? If not, is there an example to call the document update from the Meilisearch plugin?

Using:

• Meilisearch v0.27.0 docker image
• Strapi v4.1.1
• Meilisearch Strapi Plugin 0.6.2

Thank you! :)

[pedrogaudencio]

@oluademola I'm guessing along with the collection type update there could be a way to specify rules for populate including various component or relation fields by passing an array of attribute names as described in the documentation:

const entries = await strapi.entityService.findMany('api::article.article', {
  populate: ['componentA', 'relationA', 'componentB.relationC']
});

Maybe it could be extended from #410 latest additions. Would pass in custom parameters from the settings - as proposed in #423 - while having '*' as a default fallback solve this?

[bidoubiwa]

@oluademola

Meilisearch plugin does not update the Author entry inside the nested fields in the Article document.

Unfortunately we are bound to the triggers of Strapi to update documents.

What is happening when you link a collection to meilisearch

On Article:

• Adds all Article to an index called article in Meilisearch
• Subscribe the Article collection to the Strapi's hook

Now every time an action is done on an Article, it will enter the associated hook (afterUpdate or afterCreate, ...).

On Author:

• Adds all Author to an index called author in Meilisearch
• Subscribe the Author collection to the Strapi's hook

Now every time an action is done on an Author, it will enter the associated hook (afterUpdate or afterCreate, ...).

These triggers are decorrelated to each other. Relationships do not trigger the associated documents. If authors update it will not trigger the associated Articles and vice versa.

I thought of creating an afterUpdate() lifecycle hook in the Author triggering the Article document update by calling the Meilisearch plugin controller

I'm not sure this is possible.

A possible solution would be to create a setting childOf or relationshipWith which on a trigger (afterUpdate, afterCreate, ...) will make a call to the database to find every related member and update them as well.

For example, the settings of Author, we add a field childOf: "Article". Now, image we update an Author with id 1, in the hook we make a call to the database to find all Articles that have a relation with the Author with id 1. We proceed to update them all in Meilisearch.

What do you think?

@pedrogaudencio

Unfortunately this does not solve the issue. If an author is updated, even with the custom populate it will only update the author not the associated article.

[oluademola]

Hi @pedrogaudencio

A possible solution would be to create a setting childOf or relationshipWith which on a trigger (afterUpdate, afterCreate, ...) will make a call to the database to find every related member and update them as well.

For example, the settings of Author, we add a field childOf: "Article". Now, image we update an Author with id 1, in the hook we make a call to the database to find all Articles that have a relation with the Author with id 1. We proceed to update them all in Meilisearch.

Have you considered trying out this workaround?

[pedrogaudencio]

Thanks for the follow up.

A possible solution would be to create a setting childOf or relationshipWith which on a trigger (afterUpdate, afterCreate, ...) will make a call to the database to find every related member and update them as well.

For example, the settings of Author, we add a field childOf: "Article". Now, image we update an Author with id 1, in the hook we make a call to the database to find all Articles that have a relation with the Author with id 1. We proceed to update them all in Meilisearch.

@bidoubiwa @oluademola you're suggesting implementing childOf or relationshipWith as a setting in the plugin, correct? Can this be done the same way as #427 by introducing a new task within actionInBatches?

[bidoubiwa]

Another possibility which may open more possibilities is to add the hooks as a setting. We could add the following as settings:

• afterCreate
• afterUpdate
• afterDelete

Which would trigger at the end of the current hook. To take back the example of Author and Article.
I update an Article, first it triggers the current afterUpdate hook.

afterCreate(result) {
    // 1. adds the Article to Meilisearch
    // ...
    // 2. Runs the hook defined in the plugins settings 
   pluginsConfig.afterCreate(result)
}

// In the settings:

module.exports = {
  meilisearch: {
    config: {
      article: {
        async afterCreate(result) { 
             // fetches the authors related to the updated article
             const authors = await contentTypeService.getEntries(...) 
             
             // Updates the authors 
             await meilisearch
              .updateEntriesInMeilisearch({
                contentType: "author",
                entries: [authors],
              })
            }
        },
      }
    }
  },
}

With this solution we are not limited by the deepness of a relation or the type of the relation. A user can decide how the update of one content-type affects another one in Meilisearch.

Btw this is already possible using the hooks provided by Strapi. You add a custom hook in your application in which you update whatever related entry/content-type in Meilisearch.

[pedrogaudencio]

Those suggestions all seem very interesting. Do you know if there's already something in place?

await meilisearch
  .updateEntriesInMeilisearch({
    contentType: "author",
    entries: [authors],
  })

This was kinda what I was aiming towards with my suggestion of triggering the parent document update by calling the Meilisearch plugin controller. As you say, this is possible using the hooks provided by Strapi and probably the cleanest solution at this point. Although it's likely that the entry data won't pass through transformEntry() when we bypass the plugin and call meilisearch.updateEntriesInMeilisearch()...

[bidoubiwa]

I think you can have access to transformEntry by using the strapi API like this:

strapi.config.get('plugin.meilisearch')

[Pablo-Aldana]

I'm actually having this issue right now and look pretty complicate it to working around it ourselves. In my case I have events with tickets.

Events {
...
tickets:[many tickets]
...
}

if a customer buy a ticket I need to update the quantity of the specific ticket that have been bought. Right now the only workaround I can think of is a for for all the tickets of the event until I find the one I need to update change it and call the Meilisearch API updating the event document with all the array of tickets as I cannot just update one of the ticket myself.

have you consider kind of a re-index function by ID? i.e meilisearchPlugin.reindexById('/indexes/events', eventID) and with that trigger a whole refresh as it does when you save the event on the strapi dashboard.

In case it is useful to someone I managed to worked it around by replicating what the plugin does on the update lifecycle. Hardcoding the type I need to update and passing the id of the event. (Remember I'm updating here the ticket as a customer is performing a buy of a specific ticket for that event).

it could be useful to expose some kind of function where as a user you pass the contentType + the ID of the entry and it will trigger the entire reindex of this document.

However I faced a challenge here if the relationship with the parent is multiple, i.e a ticket can belong to multiple events only the event you are passing the ID is the one reindexing and the other would have the old version of the ticket.

 const meilisearch = strapi.plugin('meilisearch').service('meilisearch');
            // Fetch complete entry instead of using result that is possibly
            // partial.
            const contentTypeService = strapi.plugin('meilisearch').service('contentType');
      
            var contentType = 'api::event.event';
            const entry = await contentTypeService.getEntry({
              contentType: 'api::event.event',
              id: ticket.event.id,
              entriesQuery: meilisearch.entriesQuery({ contentType }),
            });

            meilisearch
              .updateEntriesInMeilisearch({
                contentType: contentType,
                entries: [entry],
              })
              .catch((e) => {
                strapi.log.error(`Meilisearch could not update entry with id: ${result.id}: ${e.message}`);
              });

[bidoubiwa]

have you consider kind of a re-index function by ID? i.e meilisearchPlugin.reindexById('/indexes/events', eventID) and with that trigger a whole refresh as it does when you save the event on the strapi dashboard.

Do you mean that if someone buys a ticket, you'd like the whole event index to be re-indexed or only the events that have a relationship with the ticket that has been bought?

A possibility for you instead of changing the code of the plugin is to create your own trigger and subscribe your ticket collection to it. See doc.

Which would look roughly like this:

// The file goes here:
// ./src/api/[api-name]/content-types/ticket/lifecycles.js

module.exports = {
  afterCreate(event) {
    const { result, params } = event;

     // fetch all the events that have a relationship with the `ticket` that has been added
     // see this doc: https://docs.strapi.io/dev-docs/api/entity-service/crud#findmany
    const events = await strapi.entityService.findMany(
      'events',
      { 
      // add the tickets to the events returned
      // I suppose something like this 
       populate: ["tickets"],
        // add the filter to only include the entries containing the ticket
       // I suppose something like this
        filters: { "tickets.id" : { $eq : result.id } 
      }
    )
    const eventsId = events.map(event => event.id)

     const meilisearch = strapi.plugin('meilisearch').service('meilisearch');
    meilisearch
        .updateEntriesInMeilisearch({
           contentType: contentType,
           entries: [eventsId],
        })
    // do something to the result;
  },
};

[Pablo-Aldana]

have you consider kind of a re-index function by ID? i.e meilisearchPlugin.reindexById('/indexes/events', eventID) and with that trigger a whole refresh as it does when you save the event on the strapi dashboard.

Do you mean that if someone buys a ticket, you'd like the whole event index to be re-indexed or only the events that have a relationship with the ticket that has been bought?

I meant just the documents that have a relationship with this ticket not the entire index.

Your solution seams plausible for this scenario, but not scalable for the long term as a document (i.e: event) can have multiple relationship and it would imply a lot of manual work to modify every lifecycle per each relationship, and we are just talking about one contentType / index on small apps you might have dozens indexes with tons of relationships, not even considering large applications, that what I think some kind of automation on the plugin would be interesting so that if any of the nested field of a document get updated get the document updated to.

Thanks a lot for your answer and suggestion!!

[20x-dz]

Based on this discussion I started working on a lifecycle hook for my specific use case that I then refactored into being slightly more generic:

async function updateRelationsInMeilisearch({ action, model, params }) {
  const meilisearchPlugin = strapi.plugin('meilisearch');
  const store = meilisearchPlugin.service('store');
  const meilisearch = meilisearchPlugin.service('meilisearch');
  const indexedContentTypes = await store.getIndexedContentTypes();

  Object.entries(model.attributes)
    .reduce((acc, [ key, value ]) => {
      // Extract contentType relations of this entity indexed in meilisearch
      if (value.type === 'relation' && indexedContentTypes.includes(value.target)) {
        const contentType = value.target;
        acc.push({ key, contentType });
      }
      return acc;
    }, [])
    .forEach(async ({ key, contentType }) => {
      // Relations are linked in connect/disconnect properties as part of the
      // attribute, e.g. params.data.foo: { disconnect: [], connect: [] }
      // and always contain an object with the id of the object that should be
      // connected/disconnected (longhand syntax).
      // This reducer extracts all ids of both connected/disconnected models to
      // be able to update them accordingly.
      // The only exception to this is for the afterDelete hook, where the ids
      // are extracted from the populated relations directly.
      //
      // @see https://docs.strapi.io/dev-docs/api/rest/relations
      const ids = Object.values(params.data[key]).reduce((acc, entry) => {
        const ids = action === 'afterDelete' ? [entry.id] : entry.map(item => item.id);

        return acc.concat(ids);
      }, []);

      if (ids.length === 0) {
        return;
      }

      const entries = await strapi.entityService.findMany(
        contentType,
        {
          populate: '*',
          filters: {
            id: {
              $in: ids,
            },
          },
        }
      );

      meilisearch.updateEntriesInMeilisearch({
        contentType,
        entries,
      });
    });
}

module.exports = {
  afterCreate(event) {
    updateRelationsInMeilisearch(event);
  },
  afterUpdate(event) {
    updateRelationsInMeilisearch(event);
  },
  async beforeDelete({ model, params, state }) {
    // store the full entry to be deleted for the afterDelete hook
    const entry = await strapi.entityService.findOne(
      model.uid,
      params.where.id,
      {
        populate: '*',
      }
    );

    state.entry = entry;
  },
  afterDelete(event) {
    const { state } = event;

    // mimick after{Create,Update} event params
    event.params = {
      data: {
        ...state.entry
      }
    };
    updateRelationsInMeilisearch(event);
  },
};

In a nutshell:
It first checks the attribute definitions of the underlying model and finds attributes that are in indexed by meilisearch.
For every indexed relationship, the changed ids (stored in connect/disconnect attributes) are being extracted.
For these, the complete models are then fetched from strapi and then pushed to meilisearch.

[edit] expanded to be able to handle afterDelete as well.

This could be used to register programmatically global in lifecycle hooks, and then perform the necessary updates of associated models when needed.

@bidoubiwa What do you think?

[brunoocasali]

Hi @20x-dz I'm not an expert on this plugin as @bidoubiwa, but I think you have something very promising, I will be glad to review a PR if you publish one. This seems to be a common issue here.

Also, I didn't check your code entirely but what will happen when you have circular dependencies?

Like a user has many users "friends" (not sure if that's possible in strapi, but in any case, I'm curious 😅)