Making Ansible playbook file identification deterministic

[ssbarnea]

The challange

Once recurring problem we had since Ansible was created was that tools, and sometimes people, might struggle to distinguish between a random YAML file and a playbook.

@cidrblock documented this problem few months ago, with examples of most common layouts and the their challenges. None of those layouts could be defined as optimal as all of them either did not work in all cases or required questionable alterations in configuration to make them work. To quote his conclusion:

In no scenario, does everything work "out of the box".

As a directory might contain both types of files, it is impossible to reliably determine which ones are supposed to be playbooks and which ones are not.

Some people mentioned that a tool could look inside a YAML file and try to figure-out if that might be a playbook or not. That requires relatively complex and slow heuristics that can easily fail, especially if that playbook might contain typos or not being a valid YAML file just because of some mistakes introduces by the user. Effectively with an invalid file is not possible to make identification deterministic. Also some tools, editors or IDEs do not even have the ability to change file-type based on its content and they allow only the filename to be used for for determining it.

Finding a solution

We already considered the option of using a totally new file extension like site.play, but we decided to avoid this because it would be intrusive and will prevent editors from recognizing these files as YAML files.

So the next possible solution was to use a double extension, like site.playbook.yaml in order to keep it still YAML while being able to determine that the file is a playbook and not something else.

The best thing about using a double extension is that we can start using it right away without breaking any tool.

Deciding which double extension to use

We only need to decide which double extension we want to agree on setting as standard, so we can define the right file association patterns for it in tools that need to identify playbooks.

Take a moment and pick your favourite option from the list below, consider implications of having a little bit longer playbooks filenames.

[briantist]

Will both forms of the YAML extension, .yaml and .yml be supported? I prefer the latter and almost never use the former, but I think both should be supported.

[ssbarnea]

The yaml/yml is another discussion but we will likely add support for both, don't worry. I also almost exclusively used .yml for ansible because that was, more or less, advertised in official documentation. Still, that might change in the long term, mainly because YAML specification is clear about the fact that .yaml is the recommended extension.

[cognifloyd]

I exclusively use .yaml except for the few Ansible bits that require .yml

[bcoca]

The basic distinction is simpler, playbooks/task files are always lists, all other YAML files are mappings.

[sivel]

I'm 100% against file naming as the convention to detect a playbook. Ultimately this would cause yet another confusion, that will have blowback affects on core documentation that we are likely not willing to adopt. I can describe the minimal content to define a populated playbook.

- hosts: whatever

or

- import_playbook: whatever.yml

With the above, the file is a YAML list, that has at least a hosts key or an import_playbook key.

Another alternative is an explicit pragma or modeline in the file.

[watsonb]

I tend to agree with @sivel on this one. Moreover, I've become accustomed to maintaining a .ansible-lint config file at the root of my projects and explicitly state which files/folders I want to exclude.

[cognifloyd]

I have been using .playbook.yaml for 6 or 7 years because that facilitated configuring my editor to recognize not just that it was yaml, but also add some Ansible goodies (in pycharm).

[ganeshrn]

From extension point of view we rely on file.associations to enable either yaml extension or ansible extension and these associations doesn't guarantee that a document opened in VSCode is correctly identified with ansible language resulting is issue like ansible/vscode-ansible#553

With double extension we can deterministically identify all the ansible documents and enable the ansible extension by adding the extension here. However I suggest we can add two more options to the poll which can represent all types of ansible files.

.ansible.yaml / .ansible.yml
.an.yaml / .an.yml

[ganeshrn]

@sivel Unfortunately VSCode doesn't have the capability to introspect the content of file and associate a language type (eg. ansible/YAML etc) to the opened document and that's why we have to rely on naming convention and file paths. Because of this issue most of the times ansible extension doesn’t work out of box and user have to add additional configurations to make it work. Double extension for ansible file will solve this issue easily.

[sivel]

Based on https://github.com/ctlajoie/vscode-modelines it would seem possible.

And while it solves your technical issue easily, it's going to create issues with documentation just the same way as yes/no vs true/false did. Effectively you are asking to create a new naming requirement for playbooks when that is not the real issue.

[ganeshrn]

When a document is identified with a particular language VScode extension spawns the associated language-server refer and the VSCode decision to activate extension for an open document is based on entries in package.json so I am not sure how the extension you linked solves the problem of vscode-ansible extension not working out of the box.

. Effectively you are asking to create a new naming requirement for playbooks when that is not the real issue.

It might not be an issue for ansible-core but it is certainly a major issue for vscode-ansible extension users.

[ganeshrn]

Below is the summary of 3 approaches that are discussed so far to solve the extension issue.

1. VSCode file associations:
   Example:

"files.associations": {
	"**/*ansible*/**/*.yml": "ansible",
},

Pros:

• No additional changes are required for in file path and vscode-ansible extension

Cons:

• Doesn’t work out of the box for ansible files. That is after the extension is installed additional steps are required from the user side to make the extension work.
• The file associations can result in YAML files that are not related to ansible to be associated with ansible language type.
• The file associations are not deterministic and the default user experience with vscode-ansible extension will vary for a given ansible project based on the VSCode settings.

2. Adding modelines feature to vscode-ansible extension.
   Pros:

• No additional changes are required for in file path and vscode-ansible extension
• The file associations will be deterministic and the default user experience with vscode-ansible extension will be the same for a given ansible project.

Cons:

• Doesn’t work out of the box for ansible files. That is after the extension is installed additional steps are required from the user side to make the extension work. After installing the extension users will have to add
  - modelines for each ansible file
  - Run Apply modeline command from the command palette.

IMO approach 2 is slightly better than approach 1, also it can be extended to introspect file path and file content to identify language type however it won't always correctly work at runtime when the user is typing the content. This is where approach 3 can help.

3. Adding double extensions
   Pros:

• With the double extension to identify ansible files the file associations will be deterministic and the default user experience with vscode-ansible extension will be the same for a given ansible project.
• The extension works out of the box for all ansible files without any additional settings. It solves one of the major issues with vscode-ansible extension.

Cons:

• All ansible files will be required to be renamed with a double extension

I think we can use a combination of approach 2 and approach 3 and document it for vscode-ansible extension.

[cognifloyd]

That analysis applies to pycharm as well, only modelines are not a thing in pycharm afaik. It's either extension-based, or I can't use the ansible extension (s) in pycharm.

So, the double extension is great for IDEs and for ansible-lint. @sivel does not want to document that in ansible's docs. That's fine, it can be a convention that is supported by devtools projects. Ansible itself does not need it or care about the double extension. So, it makes sense for devtools projects to document it as an optional way to simplify file type identification.

[ganeshrn]

@ssbarnea Is it possible to move this discussion to devtools repository as it affects multiple devtool projects

[bcoca]

An issue with the multiple extensions is a possible conflict with collection playbooks, so if you go this route, please avoid 'playbook' and 'playbooks' as the 2nd extension, as that might end up using the wrong playbook on execution.

[ssbarnea]

@bcoca We are sure we do not want to touch the last extension, as that must always be YAML or YML, in order to allow other editors to recognize the files. If we do anything, we would add something before it.

We also identified a way to address this issue with vscode that might make the need for double extension almost obsolete, something that can be implemented in both ansible-lint and vscode. We created a task for it on ansible/vscode-ansible#582 and hopefully we should have a fix sooner. Take a look at that ticket and comment there if the proposed logic for language detection is flawed.

[bcoca]

my point is with the 'before', the collection loading does not care if 'no extension' or yml or yaml are used, but the python importer (what the collection loading uses) DOES care that the .something. matches existing directories/files in the collection