-
Notifications
You must be signed in to change notification settings - Fork 1k
Redocly Migration Scripting and Pre‐Migration Moves
For the migration to the Redocly toolchain, there are several cases where Redocly needs things to be in a different format or syntax than they currently are.
There are three strategies we can use to make these changes, on a case by case basis. In order from most to least preferable, they are:
-
Adapt the current site to a format that's compatible with both tools.
- "Remove" Variant: Remove any Dactyl-specific syntax and switch to something plainer that both can handle.
- "Shim" Variant: Implement a quick Dactyl plugin site so that it's compatible or mostly-compatible.
-
Script a tool that can quickly apply sweeping changes from the Dactyl format to the Redocly format starting from basically any commit. Instead of committing the changes to the Redocly branch and copying work over, we commit the script to the main branch.
- "Extend" Variant: We implement something in Redocly that is a close analogue of what Dactyl supports, and the script does a find-and-replace from old→new syntax.
- Manual changes on a Redocly migration branch, which would have to be kept up-to-date and in-sync over a potentially extended period of time. In many cases we can assume that scripted / shimmed changes above may cover 90% of the work and manual changes can be applied as a last step during a code freeze just before the final migration.
Rather than forking once and having to keep the branches in-sync for a long time, it should be less friction to make a series of incremental changes to make the master branch more "Redocly-ready" while still maintaining compatibility with the current Dactyl site. Some of these changes may involve a large changeset, but by keeping each one constrained to one category of change and having the PR open only a couple days at most should reduce the impact and migration pain of any individual step.
However, we need to have a working branch for Redocly work in the meantime. We can use the redocly-migration branch for this purpose. During development, we can do some widespread conversions, especially by script, and commit them to the redocly-migration branch with a commit message that starts with [DROP]. Then, we can periodically rebase redocly-migration onto the latest master branch, and when we do we can remove and re-create the [DROP] commits by applying the same steps, which should automatically adapt to the latest changes from the master branch. To reduce the risk of merge conflicts, we should avoid doing manual touchups to files extensively modified by [DROP] commits. (This mostly means not editing the content md files by hand until we're ready for a code freeze.)
When it's time for the code freeze, one of the first things we'll do is run the conversion script(s) with a version of the commits that isn't marked [DROP]. We can then add any manual touchups that would otherwise have risked causing merge conflicts.
| Approach: | Adapt (filenames), Script (links) |
|---|---|
| Status | ✅ Done (including snippets✅) |
| Related PRs/issues | #2206, #2259 |
The Dactyl-based site used to use a convention where translated pages are {file}.ja.md in the same dir as the English versions. Redocly needs these to be under @i18n/ja/ in a parallel folder structure and named {file}.md.
The key is that Dactyl doesn't actually need the md files to follow any specific filename/folder structure. We did one update to put them under an @i18n/ja folder without any breaking changes to how Dactyl handles them; this would only be a small, one-time disturbance to translation efforts and then they could resume and the files would already be in the right place for when we're ready to flip the switch to Redocly.
The exception which wasn't as straightforward was snippets. For those, a couple lines in the migration script move and rename the files.
| Approach: | Adapt, Script |
|---|---|
| Status | ✅ Scripted (including snippets & Japanese) |
| Related commits | a5640c6, #2259 |
The Dactyl site uses relative links in the form {file}.html in most places. Redocly links can either be a relative path to the source file or an absolute path to the output path. Absolute links are convenient if you know what the final URL for a page will be and they're often shorter and don't need to change when you move the file you're linking from. Relative links work when viewing a page on GitHub, and they sometimes don't need to change, for example if files that link each other are moved as a group such that their relative hierarchy doesn't change.
The migration script (migrate.sh) uses the Dactyl build to convert the normal links into relative links, and reusable links (see below) into absolute links. This stops working after the conversion breaks the Dactyl build, so it needs to be re-run as part of rebasing on the latest master, and any fixes to the migration script need to be moved/squashed into the earlier part of the history.
Old (Dactyl)
See [Stablecoin Issuer](stablecoin-issuer.html).New (Redocly)
See [Stablecoin Issuer](../../use-cases/tokenization/stablecoin-issuer.md).| Approach: | Script |
|---|---|
| Status | Working, but slow ✅ |
The Dactyl site currently defines a three files with reusable common links and uses {% include '...' %} to pull those definitions into a variety of files, so you can write [server_info method][], [AccountSet transaction][], or [drops of XRP][] and have those automatically link to the right place.
The reusable links definitions include a bunch of Jinja syntax, so the migration script removes those and replaces them with a single _snippets/common-links.md file with the sorted results of exporting the links. This fixes several of the broken links with the Redocly build and allows us to continue to use commonly-defined links on any page that includes the common link definitions with {% raw-partial ... %}. Unfortunately, since this file is >1000 lines long and gets included in ~600 markdown files, it dramatically increases the time to start up the local dev server.
Redocly is looking into the code to see if there are ways they can improve the efficiency of how this is handled. We can also improve things by reducing the size of the common links file, for example by removing definitions for things like "New in:" badges older than ~2 years (see also, #1978).
Old (Dactyl):
<!--{# common link defs #}-->
{% include '_snippets/rippled-api-links.md' %}
{% include '_snippets/tx-type-links.md' %}
{% include '_snippets/rippled_versions.md' %}
New (Redocly):
{% raw-partial file="/_snippets/common-links.md" /%}
| Approach: | Adapt & Script |
|---|---|
| Status | ✅ Scripted |
| Related PRs / Issues | #2215, #2238 |
The Dactyl site has a bunch of auto-generated landing pages that contain just a list of children with blurbs for each; these are defined in the config file but don't generally have any representation in the filesystem. Redocly recently added some tooling so you can do something similar with a plugin, but it still needs the files to have an index.md file (or something like it) in the filesystem where the landing goes.
We've most moved from having index pages that only exist as stanzas in the Dactyl config to having index.md pages with frontmatter, but there are a couple straggler pages that aren't in the right place. The conversion script to add the necessary syntax to use a Redocly plugin ({% child-pages %} component) works now. The Redocly plugin needed a couple adjustments to not error out.
In some cases, the auto-generated indexes are showing empty bullet points as a side-effect of those pages lacking proper frontmatter (#2330). That should resolve itself when the frontmatter is added.
| Approach: | Adapt & Script |
|---|---|
| Status | ✅ Scripted |
Our _snippets/ folder contains various reusable bits of content. Redocly supports similar functionality, with {% partial file="..." /%} and {% raw-partial ... /%} Markdoc tags, although some of the details are different. Between the migration script and some manual commits in the migration branch, these are mostly handled now, including differences in how relative links in snippets are relative to the snippet location, the required location for translated snippets, the syntax to reference them, snippets not being allowed inside tables, and so on.
In the process, we found several Redocly bugs which are now fixed, including referencing partials from within partials using absolute paths, and query parameters.
Old (Dactyl):
{% include '_snippets/conf-file-location.md' %}
New (Redocly):
{% partial file="/_snippets/conf-file-location.md" /%}
| Approach: | Script |
|---|---|
| Status | ✅ Scripted |
This is one of a few cases where we use {{ variable }} syntax in the current docs. Redocly has {% $variable %} syntax but which variables it provides in which contexts doesn't quite line up with Dactyl's.
Now that Redocly has added the auto-detected page title to the $frontmatter.seo.title variable and fixed the bug with using variables in a code block, we can do a direct conversion between syntax in many contexts. One major exception is when the variable is used in inline code syntax, like `{{currentpage.name}}`, which we use extensively in reference pages; Redocly's Markdoc parser does not support variables in that context. For these, I've created a custom component, {% code-page-name %/} which produces the desired output, and configured the conversion script to use it in the appropriate places.
We don't have to change the existing pages, although many instances of {{currentpage.name}} in the existing docs are extraneous and we could just remove them. (For example, the header "AccountSet Flags" could just be "Flags" because it's already on the AccountSet page.) This would change/break some links, so this might be best saved for another time.
Old (Dactyl):
The `{{ currentpage.name }}` method is...
## {{ currentpage.name }} Flags
New (Redocly):
The {% code-page-name /%} method is...
## {% $frontmatter.seo.title %} Flags
| Approach: | Script & Adapt |
|---|---|
| Status | ✅ Scripted |
| Related PRs/issues | #2242 |
Another case where we use {{ variable }} syntax is the {{ include_svg(...) }} macro, which is helpful for theme-aware diagrams. Redocly has now implemented a component, {% inline-svg ... %} that does something similar, but we encountered some bugs with the first version of it. The bugs have largely been fixed in Redocly 0.65.6.
There are also some things {{include_svg(...)}} does that {% inline-svg ... /%} doesn't:
- link to the original image
- add a caption.
The script handles this by using plain Markdown syntax, so in the end, the result is like this:
Old (Dactyl):
{{ include_svg("img/ticket-creation.svg", "Diagram: Creating three Tickets") }}
New (Redocly):
[{% inline-svg file="/img/ticket-creation.svg" /%}](/img/ticket-creation.svg "Diagram: Creating three Tickets")
| Approach: | Script & Extend |
|---|---|
| Status | ✅ Scripted |
| Related issues / PRs | #2259 |
Redocly's code-snippet Markdoc component now does what we need (as of Redocly version 0.62.0). The syntax is slightly different, but we have a script that does a mass find-and-replace of the syntax.
Old (Dactyl):
{{ include_code("_code-samples/send-xrp/py/send-xrp.py", end_before="# Connect", language="py") }}
New (Redocly):
{% code-snippet file="/_code-samples/send-xrp/py/send-xrp.py" before="# Connect" language="py" /%}
| Approach: | Adapt & Remove |
|---|---|
| Status | ✅ Removed |
| Related issues/PRs | #2267 |
There were a few cases where our snippets were supposed to display something slightly different depending on where they're being included, using syntax like the following example:
{% if currentpage.md == "tutorials/manage-the-rippled-server/installation/install-rippled-on-ubuntu.md" or
currentpage.md == "tutorials/manage-the-rippled-server/installation/install-rippled-on-centos-rhel-with-yum" %}
sudo systemctl restart rippled.service
{% elif currentpage.md == "tutorials/manage-the-rippled-server/installation/build-run-rippled-ubuntu.md" or
currentpage.md == "tutorials/manage-the-rippled-server/installation/build-run-rippled-macos.md" %}
* Use Ctrl-C to stop `rippled`, then start it again:
./rippled
{% endif %}In a couple cases these weren't maintained when pages were moved/renamed so they silently stopped working as intended, and frequently weren't that useful anyway, so PR #2259 removed all instances of this syntax from the master branch. These changes should be fixed on the redocly-migration branch next time we rebase it.
Moving forward, if we need snippets to adapt based on the context they're used in, Redocly supports variables passed in as attributes of the {% partial /%} element's syntax.
| Approach: | Adapt & Script |
|---|---|
| Status | ✅ Partially scripted; Waiting for updates from Redocly for advanced features |
| Related issues/PRs | #2259 |
Two variables we use commonly are github_forkurl and github_branch which are set to https://github.com/XRPLF/xrpl-dev-portal and master respectively, but can be overwritten with commandline vars, which lets preview builds refer to in-repo files that come from the fork & branch of the pull request, but automatically switch those links to the master branch when merged.
Two problems came up with converting these vars to Redocly syntax:
- Redocly's Markdoc engine doesn't support using variables in the href part of a link; they simply don't get parsed. To work around this, I created a custom component which can be used like this:
{% repo-link path="content/some/file.md" /%}and added a rule to the conversion script to convert instances of the old syntax to the new one. This is more concise than the old way but less flexible (for example, it wouldn't support the "edit" link that the Client Libraries page used to have). - Redocly does not yet support automatically detecting the current GitHub fork & branch from a PR and passing it into the vars, but Roman has suggested that they would be open to adding that feature eventually. For now, the component uses variables from a
.envfile that's committed to the repo.
Old (Dactyl):
For an example of such a file, see the provided [`ledger-file.json`]({{target.github_forkurl}}/blob/{{target.github_branch}}/content/_api-examples/rippled-cli/ledger-file.json).New (Redocly):
For an example of such a file, see the provided {% repo-link path="content/_api-examples/rippled-cli/ledger-file.json" %}`ledger-file.json`{% /repo-link %}.
| Approach: | Script |
|---|---|
| Status | ✅ Scripted |
| Related issues/PRs | #2259 |
These variables are used in a couple places. The idea was any place where we want to tell people what the current reserve values are, we can use these variables and then if/when the reserves later change we can update them all in one place.
The conversion script converts instances of these to Redocly environment vars syntax, using a .env/ file. For example, {% $env.PUBLIC_GITHUB_FORK %}.
| Approach: | Adapt (with some script help) |
|---|---|
| Status | Mostly done, a couple straggling pages in progress |
| Related issues/PRs | #2265, #2254, #2243, #2270, #2276, etc. |
Marketing pages and Dev Tools have custom templates in Jinja format (template/*.html.jinja files). Redocly needs these to be React components (*.page.tsx files) instead. Roman has created a script to do the conversions, which is not perfect but provides a good start on converting things to React syntax. We've done manual edits to convert most of the templates and tools, with the only remaining ones being:
- Full Docs Index
- References
- Tutorials
| Approach: | Manual |
|---|---|
| Status | Mostly done, blocked by Redocly bug |
| Related PRs/issues | #2235 |
The Interactive tutorials have a whole lot of tricky syntax, custom JavaScript, embedded HTML, and so on. We've converted most of the things at this point, but one of the blockers is the timing of loading the scripts. When you load an interactive tutorial page directly, it works, but if you navigate to it from another page, the buttons don't do anything.
This is because of Redocly's window.onRouteChange(callback) event, which replaces $(document).ready(callback) for things that need to happen on page load: there are bugs with this callback which the Redocly team is working on fixing now. When it's fixed, the interactive tutorials should "just" work with no further effort on our part.
There is also at least one open issue with the Issue a Fungible Token tutorial, and there might be other sneaky bugs hiding in some of the tutorials.
| Approach: | Adapt, TBD |
|---|---|
| Status | Waiting for re-levelization |
The current site has a static folder for assets referenced by the templates (assets/ from repo top).
Redocly needs these to be located in a folder named static/ from the top of the build folder (content/ under the current preview).
In the migration branch, we have temporarily moved the folder to content/static/ instead of assets/ for the time being, but as part of "re-levelization" we would like to move the folder back up to the repo top. It is unclear if it will be easier/cleaner to drop the commit that moved it in the first place, or to move it back up in a separate commit.
| Approach: | Adapt, TBD |
|---|---|
| Status | TBD |
The folder for diagrams referenced by the Markdown contents (img/ from repo top) has been moved into the content/ folder so that it can be accessed by Redocly when running it with -d content. When we do "re-levelization", which will mean we stop using -d content, this move may be unnecessary or we may need to change the paths that the docs use to refer to the diagrams.
In the future, we could consider sprinkling the images throughout the docs folders near the pages that reference them and then use relative links, but that's not something that can be scripted.
Every time we rebase, if we changed anything about the img/ folder in master since the last rebase, we need to re-create the [DROP] commit to move it to content/img/. After re-levelization, we probably won't need to move the folder at all.
It still needs to be tested how this works with translated diagrams.
| Approach: | Manual |
|---|---|
| Status | ✅ Done |
We have some .json files that intentionally contain invalid syntax such as ... (trimmed) ... where long, irrelevant portions have been cut out. Redocly tries to parse these with the YAML parser and reports errors.
After various bug fixes from Redocly, we can now add an "ignore" in the redocly.yaml config file for files like this that throw errors.
| Approach: | Script |
|---|---|
| Status | ✅ Scripted |
Markdoc (used by Redocly) disables indented code blocks for some reason, which means that many of our code blocks inside lists end up displaying as plain text instead. It looks like this cannot be easily reconfigured on Redocly's end. However, Python-Markdown (used by Dactyl) does not support indented code fences, so we can't make the conversion in a way that supports both.
Roman wrote a script that converts indented code blocks into code fences, which has been wrapped up into the general-purpose migration script.
Old (Dactyl):
- This list item contains a nested code block
This is an indented code block inside the list item
New (Redocly):
- This list item contains a nested code block
```
This is a fenced code block inside the list item
```
| Approach: | Script & Adapt |
|---|---|
| Status | ✅ Scripted (mostly) |
| Related issues / PRs | #2259 |
Dactyl uses a plugin to offer multiple code blocks as tabs. Redocly has a built-in {% tabs %} component that can be used with code blocks or additional content. The conversion script converts from one type to the other (including in occasional cases where the code tabs are indented inside a list).
Not tested/not done: the current website has a script that changes all instances of code tabs on a page when you change one tab, so for example if you tab to see a Python code sample, the later samples on the same page will already be showing Python by the time you scroll to them. I don't think this works with Redocly's tabs out of the box, but it can probably be adapted for similar use.
Old (Dactyl):
<!-- MULTICODE_BLOCK_START -->
_JavaScript_
{{ include_code("_code-samples/issue-a-token/js/issue-a-token.js", start_with="// Configure issuer", end_before="// Configure hot", language="js") }}
_Python_
{{ include_code("_code-samples/issue-a-token/py/issue-a-token.py", start_with="# Configure issuer", end_before="# Configure hot", language="py") }}
_Java_
{{ include_code("_code-samples/issue-a-token/java/IssueToken.java", start_with="// Configure issuer", end_before="// Configure hot", language="java") }}
<!-- MULTICODE_BLOCK_END -->New (Redocly):
{% tabs %}
{% tab label="JavaScript" %}
{% code-snippet file="/_code-samples/issue-a-token/js/issue-a-token.js" from="// Configure issuer" before="// Configure hot" language="js" /%}
{% /tab %}
{% tab label="Python" %}
{% code-snippet file="/_code-samples/issue-a-token/py/issue-a-token.py" from="# Configure issuer" before="# Configure hot" language="py" /%}
{% /tab %}
{% tab label="Java" %}
{% code-snippet file="/_code-samples/issue-a-token/java/IssueToken.java" from="// Configure issuer" before="// Configure hot" language="java" /%}
{% /tab %}
{% /tabs %}
| Approach: | Script & Adapt |
|---|---|
| Status | ✅ Scripted |
| Related issues/PRs | #2259 |
Redocly's syntax for "admonitions" is a little different than Dactyl's for callouts, so we need to convert it over as part of the migration script.
| Approach: | Script |
|---|---|
| Status | ✅ Done |
The :not_enabled: macro has to be replaced with a similar React component. The migration script handles this automatically.
Old (Dactyl):
_(Requires the [DID amendment][] :not_enabled:)_
New (Redocly:
_(Requires the [DID amendment][] {% not-enabled /%})_
| Approach: | Manual |
|---|---|
| Status | ✅ Done |
Some of our existing code uses a syntax like <!-- {# comment here #} --> (with or without spaces) which puts a Jinja comment inside an HTML comment, so it doesn't show up when viewing the MD file in a normal parser nor using Dactyl, and the comments are empty/omitted when viewed on the published site. Some of these were throwing errors in Redocly, so we removed them. Others are seemingly OK with no change (possibly the problems were resolved with a Redocly update).