Last major update time: Jul 29, 2016 - 2100 UTC

Comments welcome: here

This feature is now in GoCD. Its documentation is here. The rest of this document is no longer needed and maintained. It’s only kept around for historical reasons.

What does “Config in a repository” mean?

It is a feature (almost all credit to Tomasz and his work over the last year or so, and Jyoti for working on some important areas) which allows you to move pipeline configurations out of GoCD and its cruise-config.xml file into one or more source-control repositories (think “git repository”), so that you can modify them externally. Such modifications will be seen by a periodic poller in the GoCD server and it will merge those pipeline configurations into the pipelines it finds in the main configuration XML file. You can see a quick preview video of this feature here.

It is important to note that not all pipelines need to be external to the config (any existing config, in any existing GoCD server will be valid). It is also important to note that this feature includes the ability for GoCD to monitor and merge external pipelines defined in multiple “config repositories”. It is also possible for a pipeline defined in a config repository to be dependent on a pipeline defined in the main config XML of the GoCD server.

   

This ability is exposed as a plugin endpoint and so, anyone can write a plugin for a config repository, to store the configuration in any manner you choose.

Here’s an image which shows the relationship between the different pieces of a setup such as this:

A quick note about “Infrastructure as code”: Many people seem to associate only being able to check in configuration to a repository as a part of “Infrastructure as code”. However, the ability to configure the GoCD server through code has existed in various forms. For instance, gomatic, using GoCD APIs, yagocd, gocd-cli, etc. I like to think of this as another way of doing the same. In this case, it’s possible to make it more declarative, depending on the plugin and possibly give more control to others.

What does it currently allow you to do? - or - “Current state”

Currently, 2 plugins exist - for a JSON or YAML config repository store. They are not bundled with the GoCD server at this time (as of GoCD version 16.7.0), but there is a strong possibility that it will soon be. If you’re willing to install one of the plugins yourself, you can definitely try it out on an instance of  the GoCD server.

What does not work yet? - or - “Known issues and unknown concerns

This is because the GoCD server does not keep a copy of the externally configured pipelines. The external config repositories are themselves the source of truth for those pipelines. If the source of truth itself is invalid, upon the restart of the GoCD server, then the server cannot load those pipelines. It will show an appropriate error.

However, the history of those pipelines is not lost. As soon as the error is fixed in the config repository, and when the server polls for changes and sees the commit that fixes it, the pipelines will all come back, along with their history.

If you lose network connection to an external config repository, and you restart the server, the same behavior will happen. Again, this is because the repository itself is the source of truth for those pipelines.

Can I try this in production?

Yes. The feature is ready to be used with release 16.7.0.

How do I try it out?

JSON plugin:

1. Pick one of the latest experimental builds or GoCD 17.3.0 from https://www.gocd.io/download/.

2. Install the JSON config repository plugin

3. After starting the server, open the config XML (“Admin -> Config XML”) and add a config repository for the server to poll. This tag should be added just after the "<server>" tag, at the top level, as a child of "<cruise>":

<config-repos>

 <config-repo plugin="json.config.plugin">

   <git url="https://github.com/arvindsv/gocd-demo-config-repo-json.git" />

 </config-repo>

</config-repos>

You can fork the repository mentioned above to your own GitHub account or locally, so that you can make some changes to see what happens. Any file ending in “.gopipeline.json” is picked up by the plugin. Documentation of what is possible in the JSON is here.

4. Give it a minute or so for the polling to happen. Once that happens, you should see three new pipelines on your dashboard, as a part of the “demo” pipeline group. You can make some changes to the JSON (change a group, add a stage, etc), and upon the next poll, the server will see those changes and apply them to the pipeline config. Remember that, if you make any changes and make a mistake, you’ll see an error at the bottom right corner (a red box).

You can even have multiple repositories. Just repeat the <config-repo> tag and make sure that there are no duplicate pipelines.

YAML plugin:

Recently, Tomasz announced a Yaml plugin for this endpoint. To try it:

1. Pick one of the latest experimental builds or GoCD 17.3.0 from https://www.gocd.io/download/

2. Install the YAML config repository plugin

3. After starting the server, open the config XML (“Admin -> Config XML”) and add a config repository for the server to poll. This tag should be added just after the "<server>" tag, at the top level, as a child of "<cruise>":

<config-repos>

 <config-repo plugin="yaml.config.plugin">

   <git url="https://github.com/tomzo/gocd-yaml-config-example.git" />

 </config-repo>

</config-repos>

You can fork the repository mentioned above to your own GitHub account or locally, so that you can make some changes to see what happens. Any file ending in “.gocd.yaml” is picked up by the plugin. Documentation of what is possible in the YAML file is here.

4. Give it a minute or so for the polling to happen. Once that happens, you should see three new pipelines on your dashboard, as a part of the “demo” pipeline group. You can make some changes to the YAML file (change a group, add a stage, etc), and upon the next poll, the server will see those changes and apply them to the pipeline config. Remember that, if you make any changes and make a mistake, you’ll see an error at the bottom right corner (a red box).

You can even have multiple repositories. Just repeat the <config-repo> tag and make sure that there are no duplicate pipelines.

What next? - or - “Future”

References

Comments welcome for the contents of this page: here

Changelog:

Old parts of this document


Moved here from “What does not work yet?” section on July 05, 2016 - 2200 UTC:

Open PR: #2298 Without this pull request merged, there is a known issue where a change to the configuration made through an API request or by changing the XML directly on the file system, can cause the config XML to become invalid, potentially causing the server to stop. This can happen when a pipeline or a stage is renamed, causing an externally-defined pipeline, dependent on the one that was renamed, to become invalid now. It looks like the recommendation will be to not have pipelines dependent across repositories or to ones in the main config XML.


Moved here on June 29, 2016 - 2100 UTC: This part about the XML plugin existed earlier, but is removed because it is not going to be part of the final release, since it is not a true plugin and a lot of the validations, etc. will not work properly. If, later, someone wants an XML way of representing pipelines, a proper plugin can be written. For now, please use the JSON plugin instead.

Trying the XML plugin:

1. Pick one of the latest experimental builds or GoCD 16.6.0 from https://go.cd/downloads.

2. After starting the server, open the config XML and add a config repository for the server to poll. This tag should be added just after the "<server>" tag, at the top level, as a child of "<cruise>":

<config-repos>

 <config-repo>

   <git url="https://github.com/arvindsv/gocd-demo-config-repo-xml.git" />

 </config-repo>

</config-repos>

You can fork the repository mentioned above to your own GitHub account or locally, so that you can make some changes to see what happens. Any file ending in “.gocd.xml” is picked up by the plugin.

3. Give it a minute or so for the polling to happen. Once that happens, you should see three new pipelines on your dashboard, as a part of the “demo” pipeline group. You can make some changes to the XML (change a group, add a stage, etc), and upon the next poll, the server will see those changes and apply them to the pipeline config. Remember that, if you make any changes and make a mistake, you’ll see an error at the bottom right corner (a red box).

The XML plugin is actually bundled as a part of the GoCD server and so, it can be used without installing anything. The “happy-path” of it works. However, there is a known issue around providing a pipeline XML with information about authorization, etc. which can cause the pipeline to not show up properly. Authorization is usually not a part of the external pipeline configuration declaration, but the XML plugin has a flaw in which it allows it.

Example config repo for XML plugin


Changed on July 29, 2016 to say that it is ready for production:

Can I try this in production?

No. Not yet. Due to some of the concerns mentioned above, it is not recommended to try this in production, unless you’re happy taking a risk of something happening. We don’t think anything very catastrophic will happen, but it would be prudent to allow a little more time. We encourage you to try it out on test or non-critical instances of GoCD, though - and raise any issues you find.

We expect to make another release of GoCD soon (early July) and more releases if we need to (quicker than monthly) to get this feature and others to everyone out there.


July 29, 2016: Moved here from the “concerns” section. These PRs are merged.


Changed on July 29, 2016: Changed to say that it can be tried on an any instance of GoCD (and not just non-critical)

What does it currently allow you to do? - or - “Current state”

If you’re willing to install one of the plugins yourself, you can definitely try it out on a non-critical  instance of  the GoCD server.


Moved here from “Known issues and unknown concerns” section.