Infrastructure as Code to manage DNS configurations

[btlogy]

Scope

AsIs: The DNS configurations of tahoe-lafs.org are manually managed by Meejah and/or Brian via the ​admin WebUI provided by the DNS registrar and hosting 3rd party ​Gandi.

The current DNS configurations lack of visibility, reproducibility and agility, making it difficult, error-prone and slow to be audited, reviewed, changed or improved.

ToBe: The DNS configuration would be declaratively defined in a version-controlled repository and deployed using automated workflows, based on the principle of Infrastructure as Code (IaC).

Value

• Contributors would be able to see the current configurations and propose changes using a well known workflow (pull request).
• Maintainers would be able to approve and deploy changes w/o direct interact with the DNS provider.
• The configurations and the workflows would be consistent, repeatable, and easily auditable.

Requirements

• A fresh export of the DNS tahoe-lafs.org zone hosted by Gandi (optional)
• A valid Personal Access Token (PAT) to read/write this zone via ​API of Gandi
• Permissions to create/manage secrets in ​infrastructure repository
• ​OpenToFu plan defining the current state in the existing ​infrastructure repository (WiP ​here)
• Automated workflow (e.g.: using GHA) to continuously integrate and deploy the plan (WiP ​here)

Additional information

This enhancement is a very nice to have requirement for the execution of the MoveOffTrac project (in which we are already planning to re-use and expand the existing ​Infrastructure as Code repository:

• #4161

And has already been discussed here:

• #3742

In addtion, it could help making progress/improvement on those issues:

• #2717
• #2718
• #2719
• #2772
• #4142
• #4160

[btlogy]

The Gandi API allows to create Personal Access Token with some granularity in terms of:

• validity: for 7, 30, 60, 90 days or 1 year
• resources: the whole org. or only some specific domains (e.g.: tahoe-lafs.org)
• permissions, 16 levels of which the following 2 should suffice in our use case:
  • "See and renew domain names" (required by the next one):
    • See the organization's domain names, domain contact information, history and technical configuration.
    • Renew and restore domain names
    • See incoming transfers status, resend FOAs and update authinfo codes for incoming transfers.
  • "Manage domain name technical configurations":
    • Edit domain technical configuration (nameservers, DNS records, glue records, DNSSEC, and web forwarding).

[btlogy]

If this is not already the case (and if it still is possible), it would preferable to have the "tahoe-lafs.org" domain assigned to what Gandi defines as an organization (which does not have to be a legal entity).

Alternatively, the domain could stay assigned to the individual who's has first registered the name, because Gandi seems to treat each user account as an "user organization" anyway. Though, the steps below will then give access to all other domains (and products) own by that individual. Which may not be a problem as long as all those resources are solely related to Tahoe-LAFS...

Here are the proposed steps based on test made in the sandbox of Gandi:

1. Create a team under the selected organization (e.g.: "DomainOps")

2. Give this team only 2 additional permissions (on top of the default "View Organization") in the "Domains" scope: "See and renew domain names" and "Manage domain name technical configurations"

3. Add members to this team by inviting them via their email address.

This should allow new members to create and rotate Personal Access Token which will be used as secrets by (GitHub) runners to manage the DNS records (and nothing else).

Unfortunately, the sandbox does not allow to fully test the steps above. So it might be needed to add the permission to "Manage personal access tokens" from the "Organization" scope to the team...

[btlogy]

This issue is blocking the deployment of the new services which should hosts the tickets, wiki and landing page after their migration out of Trac:

• ​Deploy and configure Forgejo and its requirements on webforge - tahoe-lafs/infrastructure#44
• ​Configure webforge to host the new landing page and allow its automatic deployment - tahoe-lafs/infrastructure#45

[meejah]

I have explained many months ago that I'm fine with adding an A or AAAA record or two (via "click ops" in my Gandi account), but I don't believe I can create a token for just the delegated access I have for tahoe-lafs.org and will of course not be giving a token for my Gandi account, which controls many other domains.

Nobody has asked me to add such records. I was told you were talking to Brian about this.

(My understanding of the wider "migration" piece was that the next step was to weigh pros / cons of another self-hosted situation vs. cloud-hosted services. I'm not privy to any discussion that may have happened with Brian or others outside N&B and this tracker though)

[blaisep]

In today's Nuts&Bolts, we suggested preparing the new content of the A, CNAME, AAAA, etc records and sending them to Meejah for a one-off change.

The more robust automation can be deployed separately. This decouples the architecture from the implementation details.

[btlogy]

I would not have proposed the idea of an API key to delegate the access if it was not possible. But only the owner of the domain can act upon this. Hence the discussion with Brian.

Also, it's not going to be a single "one-off" change: there will be a few steps and possibly some urgent rollback(s) if things are not going as planned. Having someone else doing the "click ops" synchronously (in an other timezone) will be increase downtime, be error-prone and make it uselessly harder for the person who will be migrating.

[btlogy]

If Brian is not available for this delegation and since Meejah can not do it, maybe an alternative is to have the zone hosted elsewhere and change the NS records (separating the registrar from the hosting service).

If the automation is not deemed valuable on the long term, this option could be reverted after the migration (with only 2 one-off changes then).

[meejah]

Can you describe what all these complex / real-time DNS changes are for?

I was imaging that any transition plan would look a lot more like: 1. set up new thing; 2. point DNS at new thing.

What's the best spot right now for an overview: pros, cons, alternatives etc? (Both so I know the right thing to look at, and for anyone reading this)

[blaisep]

• Tasks:

Setup New Thing:

• @meejah extracts list of hosts record in *.tahoe-lafs.org @ gandi
• @meejah reduces TTL to 60 (optional)
• @b3n prepares Hetzner with that list of hosts

Point DNS to New Thing

• @meejah updates gandi with the new NS server

Change gandi.net to point tahoe-lafs to the Hetzner server

Now: ` tahoe-lafs.org. 10800 IN NS a.dns.gandi.net. tahoe-lafs.org. 10800 IN NS c.dns.gandi.net. tahoe-lafs.org. 10800 IN NS b.dns.gandi.net. `

Later: ` tahoe-lafs.org. 60 IN NS hydrogen.ns.hetzner.com tahoe-lafs.org. 60 IN NS helium.ns.hetzner.com tahoe-lafs.org. 60 IN NS oxygen.ns.hetzner.com `

[btlogy]

To clarify a bit:

• This issue is about defining the DNS configuration as code, not about migrating off Trac (though it would help doing this)
• The initial proposal described all the way above was to delegate the management of the DNS records using features provided by Gandi which is currently both the DNS registrar and the DNS hosting party. The new proposal described by Blaise in the comment above is an alternative way of (hopefully) achieving the same delegation, but by splitting the registrar from the hosting.
• The pros are the same as the ones listed in the initial description (see value), in addition to those new ones:
  • separating the role of the registrar from the hosting one would likely reduce possible disruption due to losing (access to) one of those party (e.g. the DNS zone could be easily migrated elsewhere).
  • the steps to achieve the delegation seem slightly simpler than the one describe in comment:7: ​https://docs.gandi.net/en/domain_names/common_operations/changing_nameservers.html
• As for the cons and more alternatives, I would invite anyone wiling to participate to describe those here in some new comments.

[btlogy]

I was imaging that any transition plan would look a lot more like: 1. set up new thing; 2. point DNS at new thing.

Not if we want to smoothly integrate the services that will be replaced with the ones that will still be hosted on the Linode server (e.g.: valid certificates and working outgoing mail traffic).

Though, I suppose a couple of hours of blackout between step 1 and 2 and all legacy services left unreachable could work too.