coreos-cloudinit: bump to 0.9.3

configdrive: Remove broken support for ec2 metadata

Documentation/cloud-config-oem.md

@@ -13,7 +13,7 @@ If no **id** field is provided, coreos-cloudinit will ignore this section.
 
 For example, the following cloud-config document...
 
-```
+```yaml
 #cloud-config
 coreos:
   oem:
@@ -26,7 +26,7 @@ coreos:
 
 ...would be rendered to the following `/etc/oem-release`:
 
-```
+```yaml
 ID=rackspace
 NAME="Rackspace Cloud Servers"
 VERSION_ID=168.0.0

Documentation/cloud-config.md

@@ -1,6 +1,6 @@
 # Using Cloud-Config
 
-CoreOS allows you to declaratively customize various OS-level items, such as network configuration, user accounts, and systemd units. This document describes the full list of items we can configure. The `coreos-cloudinit` program uses these files as it configures the OS after startup or during runtime. 
+CoreOS allows you to declaratively customize various OS-level items, such as network configuration, user accounts, and systemd units. This document describes the full list of items we can configure. The `coreos-cloudinit` program uses these files as it configures the OS after startup or during runtime. Your cloud-config is processed during each boot.
 
 ## Configuration File
 
@@ -16,7 +16,7 @@ We've designed our implementation to allow the same cloud-config file to work ac
 
 The cloud-config file uses the [YAML][yaml] file format, which uses whitespace and new-lines to delimit lists, associative arrays, and values.
 
-A cloud-config file should contain an associative array which has zero or more of the following keys:
+A cloud-config file should contain `#cloud-config`, followed by an associative array which has zero or more of the following keys:
 
 - `coreos`
 - `ssh_authorized_keys`
@@ -40,9 +40,9 @@ CoreOS tries to conform to each platform's native method to provide user data. E
 #### etcd
 
 The `coreos.etcd.*` parameters will be translated to a partial systemd unit acting as an etcd configuration file.
-We can use the templating feature of coreos-cloudinit to automate etcd configuration with the `$private_ipv4` and `$public_ipv4` fields. For example, the following cloud-config document...
+If the platform environment supports the templating feature of coreos-cloudinit it is possible to automate etcd configuration with the `$private_ipv4` and `$public_ipv4` fields. For example, the following cloud-config document...
 
-```
+```yaml
 #cloud-config
 
 coreos:
@@ -57,7 +57,7 @@ coreos:
 
 ...will generate a systemd unit drop-in like this:
 
-```
+```yaml
 [Service]
 Environment="ETCD_NAME=node001"
 Environment="ETCD_DISCOVERY=https://discovery.etcd.io/<token>"
@@ -68,13 +68,15 @@ Environment="ETCD_PEER_ADDR=192.0.2.13:7001"
 For more information about the available configuration parameters, see the [etcd documentation][etcd-config].
 Note that hyphens in the coreos.etcd.* keys are mapped to underscores.
 
+_Note: The `$private_ipv4` and `$public_ipv4` substitution variables referenced in other documents are only supported on Amazon EC2, Google Compute Engine, OpenStack, Rackspace, and Vagrant._
+
 [etcd-config]: https://github.com/coreos/etcd/blob/master/Documentation/configuration.md
 
 #### fleet
 
 The `coreos.fleet.*` parameters work very similarly to `coreos.etcd.*`, and allow for the configuration of fleet through environment variables. For example, the following cloud-config document...
 
-```
+```yaml
 #cloud-config
 
 coreos:
@@ -85,7 +87,7 @@ coreos:
 
 ...will generate a systemd unit drop-in like this:
 
-```
+```yaml
 [Service]
 Environment="FLEET_PUBLIC_IP=203.0.113.29"
 Environment="FLEET_METADATA=region=us-west"
@@ -114,7 +116,7 @@ The `reboot-strategy` parameter also affects the behaviour of [locksmith](https:
 
 ##### Example
 
-```
+```yaml
 #cloud-config
 coreos:
   update:
@@ -123,7 +125,9 @@ coreos:
 
 #### units
 
-The `coreos.units.*` parameters define a list of arbitrary systemd units to start. Each item is an object with the following fields:
+The `coreos.units.*` parameters define a list of arbitrary systemd units to start after booting. This feature is intended to help you start essential services required to mount storage and configure networking in order to join the CoreOS cluster. It is not intended to be a Chef/Puppet replacement.
+
+Each item is an object with the following fields:
 
 - **name**: String representing unit's name. Required.
 - **runtime**: Boolean indicating whether or not to persist the unit across reboots. This is analogous to the `--runtime` argument to `systemctl enable`. Default value is false.
@@ -138,7 +142,7 @@ The `coreos.units.*` parameters define a list of arbitrary systemd units to star
 
 Write a unit to disk, automatically starting it.
 
-```
+```yaml
 #cloud-config
 
 coreos:
@@ -159,7 +163,7 @@ coreos:
 
 Start the built-in `etcd` and `fleet` services:
 
-```
+```yaml
 #cloud-config
 
 coreos:
@@ -177,7 +181,7 @@ The `ssh_authorized_keys` parameter adds public SSH keys which will be authorize
 The keys will be named "coreos-cloudinit" by default.
 Override this by using the `--ssh-key-name` flag when calling `coreos-cloudinit`.
 
-```
+```yaml
 #cloud-config
 
 ssh_authorized_keys:
@@ -189,7 +193,7 @@ ssh_authorized_keys:
 The `hostname` parameter defines the system's hostname.
 This is the local part of a fully-qualified domain name (i.e. `foo` in `foo.example.com`).
 
-```
+```yaml
 #cloud-config
 
 hostname: coreos1
@@ -203,7 +207,7 @@ All but the `passwd` and `ssh-authorized-keys` fields will be ignored if the use
 - **name**: Required. Login name of user
 - **gecos**: GECOS comment of user
 - **passwd**: Hash of the password to use for this user
-- **homedir**: User's home directory. Defaults to /home/<name>
+- **homedir**: User's home directory. Defaults to /home/\<name\>
 - **no-create-home**: Boolean. Skip home directory creation.
 - **primary-group**: Default group for the user. Defaults to a new group created named after the user.
 - **groups**: Add user to these additional groups
@@ -222,7 +226,7 @@ The following fields are not yet implemented:
 - **selinux-user**: Corresponding SELinux user
 - **ssh-import-id**: Import SSH keys by ID from Launchpad.
 
-```
+```yaml
 #cloud-config
 
 users:
@@ -261,7 +265,7 @@ Using a higher number of rounds will help create more secure passwords, but give
 
 Using the `coreos-ssh-import-github` field, we can import public SSH keys from a GitHub user to use as authorized keys to a server.
 
-```
+```yaml
 #cloud-config
 
 users:
@@ -274,7 +278,7 @@ users:
 We can also pull public SSH keys from any HTTP endpoint which matches [GitHub's API response format](https://developer.github.com/v3/users/keys/#list-public-keys-for-a-user).
 For example, if you have an installation of GitHub Enterprise, you can provide a complete URL with an authentication token:
 
-```
+```yaml
 #cloud-config
 
 users:
@@ -284,7 +288,7 @@ users:
 
 You can also specify any URL whose response matches the JSON format for public keys:
 
-```
+```yaml
 #cloud-config
 
 users:
@@ -304,7 +308,7 @@ The `write-file` parameter defines a list of files to create on the local filesy
 Explicitly not implemented is the **encoding** attribute.
 The **content** field must represent exactly what should be written to disk.
 
-```
+```yaml
 #cloud-config
 write_files:
   - path: /etc/fleet/fleet.conf
@@ -321,7 +325,7 @@ Currently, the only supported value is "localhost" which will cause your system'
 to resolve to "127.0.0.1".  This is helpful when the host does not have DNS
 infrastructure in place to resolve its own hostname, for example, when using Vagrant.
 
-```
+```yaml
 #cloud-config
 
 manage_etc_hosts: localhost

Documentation/config-drive.md

@@ -14,17 +14,21 @@ The image should be a single FAT or ISO9660 file system with the label
 
 For example, to wrap up a config named `user_data` in a config drive image:
 
-    mkdir -p /tmp/new-drive/openstack/latest
+```sh
-    cp user_data /tmp/new-drive/openstack/latest/user_data
+mkdir -p /tmp/new-drive/openstack/latest
-    mkisofs -R -V config-2 -o configdrive.iso /tmp/new-drive
+cp user_data /tmp/new-drive/openstack/latest/user_data
-    rm -r /tmp/new-drive
+mkisofs -R -V config-2 -o configdrive.iso /tmp/new-drive
+rm -r /tmp/new-drive
+```
 
 ## QEMU virtfs
 
 One exception to the above, when using QEMU it is possible to skip creating an
 image and use a plain directory containing the same contents:
 
-    qemu-system-x86_64 \
+```sh
-        -fsdev local,id=conf,security_model=none,readonly,path=/tmp/new-drive \
+qemu-system-x86_64 \
-        -device virtio-9p-pci,fsdev=conf,mount_tag=config-2 \
+    -fsdev local,id=conf,security_model=none,readonly,path=/tmp/new-drive \
-        [usual qemu options here...]
+    -device virtio-9p-pci,fsdev=conf,mount_tag=config-2 \
+    [usual qemu options here...]
+```

coreos-cloudinit.go

@@ -8,13 +8,18 @@ import (
 	"time"
 
 	"github.com/coreos/coreos-cloudinit/datasource"
+	"github.com/coreos/coreos-cloudinit/datasource/configdrive"
+	"github.com/coreos/coreos-cloudinit/datasource/file"
+	"github.com/coreos/coreos-cloudinit/datasource/metadata/ec2"
+	"github.com/coreos/coreos-cloudinit/datasource/proc_cmdline"
+	"github.com/coreos/coreos-cloudinit/datasource/url"
 	"github.com/coreos/coreos-cloudinit/initialize"
 	"github.com/coreos/coreos-cloudinit/pkg"
 	"github.com/coreos/coreos-cloudinit/system"
 )
 
 const (
-	version               = "0.8.1"
+	version               = "0.9.3"
 	datasourceInterval    = 100 * time.Millisecond
 	datasourceMaxInterval = 30 * time.Second
 	datasourceTimeout     = 5 * time.Minute
@@ -24,11 +29,12 @@ var (
 	printVersion  bool
 	ignoreFailure bool
 	sources       struct {
-		file            string
+		file               string
-		configDrive     string
+		configDrive        string
-		metadataService bool
+		metadataService    bool
-		url             string
+		ec2MetadataService string
-		procCmdLine     bool
+		url                string
+		procCmdLine        bool
 	}
 	convertNetconf string
 	workspace      string
@@ -40,9 +46,10 @@ func init() {
 	flag.BoolVar(&ignoreFailure, "ignore-failure", false, "Exits with 0 status in the event of malformed input from user-data")
 	flag.StringVar(&sources.file, "from-file", "", "Read user-data from provided file")
 	flag.StringVar(&sources.configDrive, "from-configdrive", "", "Read data from provided cloud-drive directory")
-	flag.BoolVar(&sources.metadataService, "from-metadata-service", false, "Download data from metadata service")
+	flag.BoolVar(&sources.metadataService, "from-metadata-service", false, "[DEPRECATED - Use -from-ec2-metadata] Download data from metadata service")
+	flag.StringVar(&sources.ec2MetadataService, "from-ec2-metadata", "", "Download data from the provided metadata service")
 	flag.StringVar(&sources.url, "from-url", "", "Download user-data from provided url")
-	flag.BoolVar(&sources.procCmdLine, "from-proc-cmdline", false, fmt.Sprintf("Parse %s for '%s=<url>', using the cloud-config served by an HTTP GET to <url>", datasource.ProcCmdlineLocation, datasource.ProcCmdlineCloudConfigFlag))
+	flag.BoolVar(&sources.procCmdLine, "from-proc-cmdline", false, fmt.Sprintf("Parse %s for '%s=<url>', using the cloud-config served by an HTTP GET to <url>", proc_cmdline.ProcCmdlineLocation, proc_cmdline.ProcCmdlineCloudConfigFlag))
 	flag.StringVar(&convertNetconf, "convert-netconf", "", "Read the network config provided in cloud-drive and translate it from the specified format into networkd unit files (requires the -from-configdrive flag)")
 	flag.StringVar(&workspace, "workspace", "/var/lib/coreos-cloudinit", "Base directory coreos-cloudinit should use to store data")
 	flag.StringVar(&sshKeyName, "ssh-key-name", initialize.DefaultSSHKeyName, "Add SSH keys to the system with the given name")
@@ -78,7 +85,7 @@ func main() {
 
 	dss := getDatasources()
 	if len(dss) == 0 {
-		fmt.Println("Provide at least one of --from-file, --from-configdrive, --from-metadata-service, --from-url or --from-proc-cmdline")
+		fmt.Println("Provide at least one of --from-file, --from-configdrive, --from-ec2-metadata, --from-url or --from-proc-cmdline")
 		os.Exit(1)
 	}
 
@@ -102,6 +109,7 @@ func main() {
 		die()
 	}
 
+	// Extract IPv4 addresses from metadata if possible
 	var subs map[string]string
 	if len(metadataBytes) > 0 {
 		subs, err = initialize.ExtractIPsFromMetadata(metadataBytes)
@@ -110,49 +118,116 @@ func main() {
 			die()
 		}
 	}
-	env := initialize.NewEnvironment("/", ds.ConfigRoot(), workspace, convertNetconf, sshKeyName, subs)
 
-	if len(metadataBytes) > 0 {
+	// Apply environment to user-data
-		if err := processMetadata(string(metadataBytes), env); err != nil {
+	env := initialize.NewEnvironment("/", ds.ConfigRoot(), workspace, convertNetconf, sshKeyName, subs)
-			fmt.Printf("Failed to process meta-data: %v\n", err)
+	userdata := env.Apply(string(userdataBytes))
-			die()
+
-		}
+	var ccm, ccu *initialize.CloudConfig
+	var script *system.Script
+	if ccm, err = initialize.ParseMetaData(string(metadataBytes)); err != nil {
+		fmt.Printf("Failed to parse meta-data: %v\n", err)
+		die()
+	}
+	if ud, err := initialize.ParseUserData(userdata); err != nil {
+		fmt.Printf("Failed to parse user-data: %v\n", err)
+		die()
 	} else {
-		fmt.Println("No meta-data to handle.")
+		switch t := ud.(type) {
+		case *initialize.CloudConfig:
+			ccu = t
+		case system.Script:
+			script = &t
+		}
 	}
 
-	if len(userdataBytes) > 0 {
+	var cc *initialize.CloudConfig
-		if err := processUserdata(string(userdataBytes), env); err != nil {
+	if ccm != nil && ccu != nil {
-			fmt.Printf("Failed to process user-data: %v\n", err)
+		fmt.Println("Merging cloud-config from meta-data and user-data")
-			if !ignoreFailure {
+		merged := mergeCloudConfig(*ccm, *ccu)
-				die()
+		cc = &merged
-			}
+	} else if ccm != nil && ccu == nil {
-		}
+		fmt.Println("Processing cloud-config from meta-data")
+		cc = ccm
+	} else if ccm == nil && ccu != nil {
+		fmt.Println("Processing cloud-config from user-data")
+		cc = ccu
 	} else {
-		fmt.Println("No user-data to handle.")
+		fmt.Println("No cloud-config data to handle.")
+	}
+
+	if cc != nil {
+		if err = initialize.Apply(*cc, env); err != nil {
+			fmt.Printf("Failed to apply cloud-config: %v\n", err)
+			die()
+		}
+	}
+
+	if script != nil {
+		if err = runScript(*script, env); err != nil {
+			fmt.Printf("Failed to run script: %v\n", err)
+			die()
+		}
 	}
 }
 
+// mergeCloudConfig merges certain options from mdcc (a CloudConfig derived from
+// meta-data) onto udcc (a CloudConfig derived from user-data), if they are
+// not already set on udcc (i.e. user-data always takes precedence)
+// NB: This needs to be kept in sync with ParseMetadata so that it tracks all
+// elements of a CloudConfig which that function can populate.
+func mergeCloudConfig(mdcc, udcc initialize.CloudConfig) (cc initialize.CloudConfig) {
+	if mdcc.Hostname != "" {
+		if udcc.Hostname != "" {
+			fmt.Printf("Warning: user-data hostname (%s) overrides metadata hostname (%s)\n", udcc.Hostname, mdcc.Hostname)
+		} else {
+			udcc.Hostname = mdcc.Hostname
+		}
+
+	}
+	for _, key := range mdcc.SSHAuthorizedKeys {
+		udcc.SSHAuthorizedKeys = append(udcc.SSHAuthorizedKeys, key)
+	}
+	if mdcc.NetworkConfigPath != "" {
+		if udcc.NetworkConfigPath != "" {
+			fmt.Printf("Warning: user-data NetworkConfigPath %s overrides metadata NetworkConfigPath %s\n", udcc.NetworkConfigPath, mdcc.NetworkConfigPath)
+		} else {
+			udcc.NetworkConfigPath = mdcc.NetworkConfigPath
+		}
+	}
+	return udcc
+}
+
+// getDatasources creates a slice of possible Datasources for cloudinit based
+// on the different source command-line flags.
 func getDatasources() []datasource.Datasource {
 	dss := make([]datasource.Datasource, 0, 5)
 	if sources.file != "" {
-		dss = append(dss, datasource.NewLocalFile(sources.file))
+		dss = append(dss, file.NewDatasource(sources.file))
 	}
 	if sources.url != "" {
-		dss = append(dss, datasource.NewRemoteFile(sources.url))
+		dss = append(dss, url.NewDatasource(sources.url))
 	}
 	if sources.configDrive != "" {
-		dss = append(dss, datasource.NewConfigDrive(sources.configDrive))
+		dss = append(dss, configdrive.NewDatasource(sources.configDrive))
 	}
 	if sources.metadataService {
-		dss = append(dss, datasource.NewMetadataService())
+		dss = append(dss, ec2.NewDatasource(ec2.DefaultAddress))
+	}
+	if sources.ec2MetadataService != "" {
+		dss = append(dss, ec2.NewDatasource(sources.ec2MetadataService))
 	}
 	if sources.procCmdLine {
-		dss = append(dss, datasource.NewProcCmdline())
+		dss = append(dss, proc_cmdline.NewDatasource())
 	}
 	return dss
 }
 
+// selectDatasource attempts to choose a valid Datasource to use based on its
+// current availability. The first Datasource to report to be available is
+// returned. Datasources will be retried if possible if they are not
+// immediately available. If all Datasources are permanently unavailable or
+// datasourceTimeout is reached before one becomes available, nil is returned.
 func selectDatasource(sources []datasource.Datasource) datasource.Datasource {
 	ds := make(chan datasource.Datasource)
 	stop := make(chan struct{})
@@ -175,7 +250,7 @@ func selectDatasource(sources []datasource.Datasource) datasource.Datasource {
 				select {
 				case <-stop:
 					return
-				case <-time.Tick(duration):
+				case <-time.After(duration):
 					duration = pkg.ExpBackoff(duration, datasourceMaxInterval)
 				}
 			}
@@ -192,55 +267,25 @@ func selectDatasource(sources []datasource.Datasource) datasource.Datasource {
 	select {
 	case s = <-ds:
 	case <-done:
-	case <-time.Tick(datasourceTimeout):
+	case <-time.After(datasourceTimeout):
 	}
 
 	close(stop)
 	return s
 }
 
-func processUserdata(userdata string, env *initialize.Environment) error {
+// TODO(jonboulle): this should probably be refactored and moved into a different module
-	userdata = env.Apply(userdata)
+func runScript(script system.Script, env *initialize.Environment) error {
-
+	err := initialize.PrepWorkspace(env.Workspace())
-	parsed, err := initialize.ParseUserData(userdata)
-	if err != nil {
-		fmt.Printf("Failed parsing user-data: %v\n", err)
-		return err
-	}
-
-	err = initialize.PrepWorkspace(env.Workspace())
 	if err != nil {
 		fmt.Printf("Failed preparing workspace: %v\n", err)
 		return err
 	}
-
+	path, err := initialize.PersistScriptInWorkspace(script, env.Workspace())
-	switch t := parsed.(type) {
+	if err == nil {
-	case initialize.CloudConfig:
+		var name string
-		err = initialize.Apply(t, env)
+		name, err = system.ExecuteScript(path)
-	case system.Script:
+		initialize.PersistUnitNameInWorkspace(name, env.Workspace())
-		var path string
-		path, err = initialize.PersistScriptInWorkspace(t, env.Workspace())
-		if err == nil {
-			var name string
-			name, err = system.ExecuteScript(path)
-			initialize.PersistUnitNameInWorkspace(name, env.Workspace())
-		}
 	}
-
 	return err
 }
-
-func processMetadata(metadata string, env *initialize.Environment) error {
-	parsed, err := initialize.ParseMetaData(metadata)
-	if err != nil {
-		fmt.Printf("Failed parsing meta-data: %v\n", err)
-		return err
-	}
-	err = initialize.PrepWorkspace(env.Workspace())
-	if err != nil {
-		fmt.Printf("Failed preparing workspace: %v\n", err)
-		return err
-	}
-
-	return initialize.Apply(parsed, env)
-}

coreos-cloudinit_test.go

@@ -0,0 +1,110 @@
+package main
+
+import (
+	"reflect"
+	"testing"
+
+	"github.com/coreos/coreos-cloudinit/initialize"
+)
+
+func TestMergeCloudConfig(t *testing.T) {
+	simplecc := initialize.CloudConfig{
+		SSHAuthorizedKeys: []string{"abc", "def"},
+		Hostname:          "foobar",
+		NetworkConfigPath: "/path/somewhere",
+	}
+	for i, tt := range []struct {
+		udcc initialize.CloudConfig
+		mdcc initialize.CloudConfig
+		want initialize.CloudConfig
+	}{
+		{
+			// If mdcc is empty, udcc should be returned unchanged
+			simplecc,
+			initialize.CloudConfig{},
+			simplecc,
+		},
+		{
+			// If udcc is empty, mdcc should be returned unchanged(overridden)
+			initialize.CloudConfig{},
+			simplecc,
+			simplecc,
+		},
+		{
+			// user-data should override completely in the case of conflicts
+			simplecc,
+			initialize.CloudConfig{
+				Hostname:          "meta-hostname",
+				NetworkConfigPath: "/path/meta",
+			},
+			simplecc,
+		},
+		{
+			// Mixed merge should succeed
+			initialize.CloudConfig{
+				SSHAuthorizedKeys: []string{"abc", "def"},
+				Hostname:          "user-hostname",
+				NetworkConfigPath: "/path/somewhere",
+			},
+			initialize.CloudConfig{
+				SSHAuthorizedKeys: []string{"woof", "qux"},
+				Hostname:          "meta-hostname",
+			},
+			initialize.CloudConfig{
+				SSHAuthorizedKeys: []string{"abc", "def", "woof", "qux"},
+				Hostname:          "user-hostname",
+				NetworkConfigPath: "/path/somewhere",
+			},
+		},
+		{
+			// Completely non-conflicting merge should be fine
+			initialize.CloudConfig{
+				Hostname: "supercool",
+			},
+			initialize.CloudConfig{
+				SSHAuthorizedKeys: []string{"zaphod", "beeblebrox"},
+				NetworkConfigPath: "/dev/fun",
+			},
+			initialize.CloudConfig{
+				Hostname:          "supercool",
+				SSHAuthorizedKeys: []string{"zaphod", "beeblebrox"},
+				NetworkConfigPath: "/dev/fun",
+			},
+		},
+		{
+			// Non-mergeable settings in user-data should not be affected
+			initialize.CloudConfig{
+				Hostname:       "mememe",
+				ManageEtcHosts: initialize.EtcHosts("lolz"),
+			},
+			initialize.CloudConfig{
+				Hostname:          "youyouyou",
+				NetworkConfigPath: "meta-meta-yo",
+			},
+			initialize.CloudConfig{
+				Hostname:          "mememe",
+				ManageEtcHosts:    initialize.EtcHosts("lolz"),
+				NetworkConfigPath: "meta-meta-yo",
+			},
+		},
+		{
+			// Non-mergeable (unexpected) settings in meta-data are ignored
+			initialize.CloudConfig{
+				Hostname: "mememe",
+			},
+			initialize.CloudConfig{
+				ManageEtcHosts:    initialize.EtcHosts("lolz"),
+				NetworkConfigPath: "meta-meta-yo",
+			},
+			initialize.CloudConfig{
+				Hostname:          "mememe",
+				NetworkConfigPath: "meta-meta-yo",
+			},
+		},
+	} {
+		got := mergeCloudConfig(tt.mdcc, tt.udcc)
+		if !reflect.DeepEqual(got, tt.want) {
+			t.Errorf("case #%d: mergeCloudConfig mutated CloudConfig unexpectedly:\ngot:\n%s\nwant:\n%s", i, got, tt.want)
+		}
+	}
+}

datasource/configdrive.go

@@ -1,48 +0,0 @@
-package datasource
-
-import (
-	"io/ioutil"
-	"os"
-	"path"
-)
-
-type configDrive struct {
-	root string
-}
-
-func NewConfigDrive(root string) *configDrive {
-	return &configDrive{path.Join(root, "openstack")}
-}
-
-func (cd *configDrive) IsAvailable() bool {
-	_, err := os.Stat(cd.root)
-	return !os.IsNotExist(err)
-}
-
-func (cd *configDrive) AvailabilityChanges() bool {
-	return true
-}
-
-func (cd *configDrive) ConfigRoot() string {
-	return cd.root
-}
-
-func (cd *configDrive) FetchMetadata() ([]byte, error) {
-	return cd.readFile("meta_data.json")
-}
-
-func (cd *configDrive) FetchUserdata() ([]byte, error) {
-	return cd.readFile("user_data")
-}
-
-func (cd *configDrive) Type() string {
-	return "cloud-drive"
-}
-
-func (cd *configDrive) readFile(filename string) ([]byte, error) {
-	data, err := ioutil.ReadFile(path.Join(cd.root, "latest", filename))
-	if os.IsNotExist(err) {
-		err = nil
-	}
-	return data, err
-}

datasource/configdrive/configdrive.go

@@ -0,0 +1,63 @@
+package configdrive
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path"
+)
+
+const (
+	openstackApiVersion = "latest"
+)
+
+type configDrive struct {
+	root     string
+	readFile func(filename string) ([]byte, error)
+}
+
+func NewDatasource(root string) *configDrive {
+	return &configDrive{root, ioutil.ReadFile}
+}
+
+func (cd *configDrive) IsAvailable() bool {
+	_, err := os.Stat(cd.root)
+	return !os.IsNotExist(err)
+}
+
+func (cd *configDrive) AvailabilityChanges() bool {
+	return true
+}
+
+func (cd *configDrive) ConfigRoot() string {
+	return cd.openstackRoot()
+}
+
+func (cd *configDrive) FetchMetadata() ([]byte, error) {
+	return cd.tryReadFile(path.Join(cd.openstackVersionRoot(), "meta_data.json"))
+}
+
+func (cd *configDrive) FetchUserdata() ([]byte, error) {
+	return cd.tryReadFile(path.Join(cd.openstackVersionRoot(), "user_data"))
+}
+
+func (cd *configDrive) Type() string {
+	return "cloud-drive"
+}
+
+func (cd *configDrive) openstackRoot() string {
+	return path.Join(cd.root, "openstack")
+}
+
+func (cd *configDrive) openstackVersionRoot() string {
+	return path.Join(cd.openstackRoot(), openstackApiVersion)
+}
+
+func (cd *configDrive) tryReadFile(filename string) ([]byte, error) {
+	fmt.Printf("Attempting to read from %q\n", filename)
+	data, err := cd.readFile(filename)
+	if os.IsNotExist(err) {
+		err = nil
+	}
+	return data, err
+}

datasource/configdrive/configdrive_test.go

@@ -0,0 +1,125 @@
+package configdrive
+
+import (
+	"os"
+	"testing"
+)
+
+type mockFilesystem []string
+
+func (m mockFilesystem) readFile(filename string) ([]byte, error) {
+	for _, file := range m {
+		if file == filename {
+			return []byte(filename), nil
+		}
+	}
+	return nil, os.ErrNotExist
+}
+
+func TestFetchMetadata(t *testing.T) {
+	for _, tt := range []struct {
+		root     string
+		filename string
+		files    mockFilesystem
+	}{
+		{
+			"/",
+			"",
+			mockFilesystem{},
+		},
+		{
+			"/",
+			"/openstack/latest/meta_data.json",
+			mockFilesystem([]string{"/openstack/latest/meta_data.json"}),
+		},
+		{
+			"/media/configdrive",
+			"/media/configdrive/openstack/latest/meta_data.json",
+			mockFilesystem([]string{"/media/configdrive/openstack/latest/meta_data.json"}),
+		},
+	} {
+		cd := configDrive{tt.root, tt.files.readFile}
+		filename, err := cd.FetchMetadata()
+		if err != nil {
+			t.Fatalf("bad error for %q: want %q, got %q", tt, nil, err)
+		}
+		if string(filename) != tt.filename {
+			t.Fatalf("bad path for %q: want %q, got %q", tt, tt.filename, filename)
+		}
+	}
+}
+
+func TestFetchUserdata(t *testing.T) {
+	for _, tt := range []struct {
+		root     string
+		filename string
+		files    mockFilesystem
+	}{
+		{
+			"/",
+			"",
+			mockFilesystem{},
+		},
+		{
+			"/",
+			"/openstack/latest/user_data",
+			mockFilesystem([]string{"/openstack/latest/user_data"}),
+		},
+		{
+			"/media/configdrive",
+			"/media/configdrive/openstack/latest/user_data",
+			mockFilesystem([]string{"/media/configdrive/openstack/latest/user_data"}),
+		},
+	} {
+		cd := configDrive{tt.root, tt.files.readFile}
+		filename, err := cd.FetchUserdata()
+		if err != nil {
+			t.Fatalf("bad error for %q: want %q, got %q", tt, nil, err)
+		}
+		if string(filename) != tt.filename {
+			t.Fatalf("bad path for %q: want %q, got %q", tt, tt.filename, filename)
+		}
+	}
+}
+
+func TestConfigRoot(t *testing.T) {
+	for _, tt := range []struct {
+		root       string
+		configRoot string
+	}{
+		{
+			"/",
+			"/openstack",
+		},
+		{
+			"/media/configdrive",
+			"/media/configdrive/openstack",
+		},
+	} {
+		cd := configDrive{tt.root, nil}
+		if configRoot := cd.ConfigRoot(); configRoot != tt.configRoot {
+			t.Fatalf("bad config root for %q: want %q, got %q", tt, tt.configRoot, configRoot)
+		}
+	}
+}
+
+func TestNewDatasource(t *testing.T) {
+	for _, tt := range []struct {
+		root       string
+		expectRoot string
+	}{
+		{
+			root:       "",
+			expectRoot: "",
+		},
+		{
+			root:       "/media/configdrive",
+			expectRoot: "/media/configdrive",
+		},
+	} {
+		service := NewDatasource(tt.root)
+		if service.root != tt.expectRoot {
+			t.Fatalf("bad root (%q): want %q, got %q", tt.root, tt.expectRoot, service.root)
+		}
+	}
+}

datasource/datasource.go

@@ -1,5 +1,10 @@
 package datasource
 
+const (
+	Ec2ApiVersion       = "2009-04-04"
+	OpenstackApiVersion = "2012-08-10"
+)
+
 type Datasource interface {
 	IsAvailable() bool
 	AvailabilityChanges() bool

datasource/file/file.go

@@ -1,4 +1,4 @@
-package datasource
+package file
 
 import (
 	"io/ioutil"
@@ -9,7 +9,7 @@ type localFile struct {
 	path string
 }
 
-func NewLocalFile(path string) *localFile {
+func NewDatasource(path string) *localFile {
 	return &localFile{path}
 }
 

datasource/metadata/ec2/metadata.go

@@ -0,0 +1,141 @@
+package ec2
+
+import (
+	"bufio"
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"strings"
+
+	"github.com/coreos/coreos-cloudinit/pkg"
+)
+
+const (
+	DefaultAddress = "http://169.254.169.254/"
+	apiVersion     = "2009-04-04"
+	userdataUrl    = apiVersion + "/user-data"
+	metadataUrl    = apiVersion + "/meta-data"
+)
+
+type metadataService struct {
+	root   string
+	client pkg.Getter
+}
+
+func NewDatasource(root string) *metadataService {
+	if !strings.HasSuffix(root, "/") {
+		root += "/"
+	}
+	return &metadataService{root, pkg.NewHttpClient()}
+}
+
+func (ms metadataService) IsAvailable() bool {
+	_, err := ms.client.Get(ms.root + apiVersion)
+	return (err == nil)
+}
+
+func (ms metadataService) AvailabilityChanges() bool {
+	return true
+}
+
+func (ms metadataService) ConfigRoot() string {
+	return ms.root
+}
+
+func (ms metadataService) FetchMetadata() ([]byte, error) {
+	attrs := make(map[string]interface{})
+	if keynames, err := fetchAttributes(ms.client, fmt.Sprintf("%s/public-keys", ms.metadataUrl())); err == nil {
+		keyIDs := make(map[string]string)
+		for _, keyname := range keynames {
+			tokens := strings.SplitN(keyname, "=", 2)
+			if len(tokens) != 2 {
+				return nil, fmt.Errorf("malformed public key: %q", keyname)
+			}
+			keyIDs[tokens[1]] = tokens[0]
+		}
+
+		keys := make(map[string]string)
+		for name, id := range keyIDs {
+			sshkey, err := fetchAttribute(ms.client, fmt.Sprintf("%s/public-keys/%s/openssh-key", ms.metadataUrl(), id))
+			if err != nil {
+				return nil, err
+			}
+			keys[name] = sshkey
+			fmt.Printf("Found SSH key for %q\n", name)
+		}
+		attrs["public_keys"] = keys
+	} else if _, ok := err.(pkg.ErrNotFound); !ok {
+		return nil, err
+	}
+
+	if hostname, err := fetchAttribute(ms.client, fmt.Sprintf("%s/hostname", ms.metadataUrl())); err == nil {
+		attrs["hostname"] = hostname
+	} else if _, ok := err.(pkg.ErrNotFound); !ok {
+		return nil, err
+	}
+
+	if localAddr, err := fetchAttribute(ms.client, fmt.Sprintf("%s/local-ipv4", ms.metadataUrl())); err == nil {
+		attrs["local-ipv4"] = localAddr
+	} else if _, ok := err.(pkg.ErrNotFound); !ok {
+		return nil, err
+	}
+
+	if publicAddr, err := fetchAttribute(ms.client, fmt.Sprintf("%s/public-ipv4", ms.metadataUrl())); err == nil {
+		attrs["public-ipv4"] = publicAddr
+	} else if _, ok := err.(pkg.ErrNotFound); !ok {
+		return nil, err
+	}
+
+	if content_path, err := fetchAttribute(ms.client, fmt.Sprintf("%s/network_config/content_path", ms.metadataUrl())); err == nil {
+		attrs["network_config"] = map[string]string{
+			"content_path": content_path,
+		}
+	} else if _, ok := err.(pkg.ErrNotFound); !ok {
+		return nil, err
+	}
+
+	return json.Marshal(attrs)
+}
+
+func (ms metadataService) FetchUserdata() ([]byte, error) {
+	if data, err := ms.client.GetRetry(ms.userdataUrl()); err == nil {
+		return data, err
+	} else if _, ok := err.(pkg.ErrNotFound); ok {
+		return []byte{}, nil
+	} else {
+		return data, err
+	}
+}
+
+func (ms metadataService) Type() string {
+	return "ec2-metadata-service"
+}
+
+func (ms metadataService) metadataUrl() string {
+	return (ms.root + metadataUrl)
+}
+
+func (ms metadataService) userdataUrl() string {
+	return (ms.root + userdataUrl)
+}
+
+func fetchAttributes(client pkg.Getter, url string) ([]string, error) {
+	resp, err := client.GetRetry(url)
+	if err != nil {
+		return nil, err
+	}
+	scanner := bufio.NewScanner(bytes.NewBuffer(resp))
+	data := make([]string, 0)
+	for scanner.Scan() {
+		data = append(data, scanner.Text())
+	}
+	return data, scanner.Err()
+}
+
+func fetchAttribute(client pkg.Getter, url string) (string, error) {
+	if attrs, err := fetchAttributes(client, url); err == nil && len(attrs) > 0 {
+		return attrs[0], nil
+	} else {
+		return "", err
+	}
+}

datasource/metadata/ec2/metadata_test.go

@@ -0,0 +1,324 @@
+package ec2
+
+import (
+	"bytes"
+	"fmt"
+	"reflect"
+	"testing"
+
+	"github.com/coreos/coreos-cloudinit/pkg"
+)
+
+type testHttpClient struct {
+	resources map[string]string
+	err       error
+}
+
+func (t *testHttpClient) GetRetry(url string) ([]byte, error) {
+	if t.err != nil {
+		return nil, t.err
+	}
+	if val, ok := t.resources[url]; ok {
+		return []byte(val), nil
+	} else {
+		return nil, pkg.ErrNotFound{fmt.Errorf("not found: %q", url)}
+	}
+}
+
+func (t *testHttpClient) Get(url string) ([]byte, error) {
+	return t.GetRetry(url)
+}
+
+func TestAvailabilityChanges(t *testing.T) {
+	want := true
+	if ac := (metadataService{}).AvailabilityChanges(); ac != want {
+		t.Fatalf("bad AvailabilityChanges: want %q, got %q", want, ac)
+	}
+}
+
+func TestType(t *testing.T) {
+	want := "ec2-metadata-service"
+	if kind := (metadataService{}).Type(); kind != want {
+		t.Fatalf("bad type: want %q, got %q", want, kind)
+	}
+}
+
+func TestIsAvailable(t *testing.T) {
+	for _, tt := range []struct {
+		root      string
+		resources map[string]string
+		expect    bool
+	}{
+		{
+			root: "/",
+			resources: map[string]string{
+				"/2009-04-04": "",
+			},
+			expect: true,
+		},
+		{
+			root:      "/",
+			resources: map[string]string{},
+			expect:    false,
+		},
+	} {
+		service := &metadataService{tt.root, &testHttpClient{tt.resources, nil}}
+		if a := service.IsAvailable(); a != tt.expect {
+			t.Fatalf("bad isAvailable (%q): want %q, got %q", tt.resources, tt.expect, a)
+		}
+	}
+}
+
+func TestFetchUserdata(t *testing.T) {
+	for _, tt := range []struct {
+		root      string
+		resources map[string]string
+		userdata  []byte
+		clientErr error
+		expectErr error
+	}{
+		{
+			root: "/",
+			resources: map[string]string{
+				"/2009-04-04/user-data": "hello",
+			},
+			userdata: []byte("hello"),
+		},
+		{
+			root:      "/",
+			clientErr: pkg.ErrNotFound{fmt.Errorf("test not found error")},
+			userdata:  []byte{},
+		},
+		{
+			root:      "/",
+			clientErr: pkg.ErrTimeout{fmt.Errorf("test timeout error")},
+			expectErr: pkg.ErrTimeout{fmt.Errorf("test timeout error")},
+		},
+	} {
+		service := &metadataService{tt.root, &testHttpClient{tt.resources, tt.clientErr}}
+		data, err := service.FetchUserdata()
+		if Error(err) != Error(tt.expectErr) {
+			t.Fatalf("bad error (%q): want %q, got %q", tt.resources, tt.expectErr, err)
+		}
+		if !bytes.Equal(data, tt.userdata) {
+			t.Fatalf("bad userdata (%q): want %q, got %q", tt.resources, tt.userdata, data)
+		}
+	}
+}
+
+func TestUrls(t *testing.T) {
+	for _, tt := range []struct {
+		root       string
+		expectRoot string
+		userdata   string
+		metadata   string
+	}{
+		{
+			root:       "/",
+			expectRoot: "/",
+			userdata:   "/2009-04-04/user-data",
+			metadata:   "/2009-04-04/meta-data",
+		},
+		{
+			root:       "http://169.254.169.254/",
+			expectRoot: "http://169.254.169.254/",
+			userdata:   "http://169.254.169.254/2009-04-04/user-data",
+			metadata:   "http://169.254.169.254/2009-04-04/meta-data",
+		},
+	} {
+		service := &metadataService{tt.root, nil}
+		if url := service.userdataUrl(); url != tt.userdata {
+			t.Fatalf("bad url (%q): want %q, got %q", tt.root, tt.userdata, url)
+		}
+		if url := service.metadataUrl(); url != tt.metadata {
+			t.Fatalf("bad url (%q): want %q, got %q", tt.root, tt.metadata, url)
+		}
+		if url := service.ConfigRoot(); url != tt.expectRoot {
+			t.Fatalf("bad url (%q): want %q, got %q", tt.root, tt.expectRoot, url)
+		}
+	}
+}
+
+func TestFetchAttributes(t *testing.T) {
+	for _, s := range []struct {
+		resources map[string]string
+		err       error
+		tests     []struct {
+			path string
+			val  []string
+		}
+	}{
+		{
+			resources: map[string]string{
+				"/":      "a\nb\nc/",
+				"/c/":    "d\ne/",
+				"/c/e/":  "f",
+				"/a":     "1",
+				"/b":     "2",
+				"/c/d":   "3",
+				"/c/e/f": "4",
+			},
+			tests: []struct {
+				path string
+				val  []string
+			}{
+				{"/", []string{"a", "b", "c/"}},
+				{"/b", []string{"2"}},
+				{"/c/d", []string{"3"}},
+				{"/c/e/", []string{"f"}},
+			},
+		},
+		{
+			err: pkg.ErrNotFound{fmt.Errorf("test error")},
+			tests: []struct {
+				path string
+				val  []string
+			}{
+				{"", nil},
+			},
+		},
+	} {
+		client := &testHttpClient{s.resources, s.err}
+		for _, tt := range s.tests {
+			attrs, err := fetchAttributes(client, tt.path)
+			if err != s.err {
+				t.Fatalf("bad error for %q (%q): want %q, got %q", tt.path, s.resources, s.err, err)
+			}
+			if !reflect.DeepEqual(attrs, tt.val) {
+				t.Fatalf("bad fetch for %q (%q): want %q, got %q", tt.path, s.resources, tt.val, attrs)
+			}
+		}
+	}
+}
+
+func TestFetchAttribute(t *testing.T) {
+	for _, s := range []struct {
+		resources map[string]string
+		err       error
+		tests     []struct {
+			path string
+			val  string
+		}
+	}{
+		{
+			resources: map[string]string{
+				"/":      "a\nb\nc/",
+				"/c/":    "d\ne/",
+				"/c/e/":  "f",
+				"/a":     "1",
+				"/b":     "2",
+				"/c/d":   "3",
+				"/c/e/f": "4",
+			},
+			tests: []struct {
+				path string
+				val  string
+			}{
+				{"/a", "1"},
+				{"/b", "2"},
+				{"/c/d", "3"},
+				{"/c/e/f", "4"},
+			},
+		},
+		{
+			err: pkg.ErrNotFound{fmt.Errorf("test error")},
+			tests: []struct {
+				path string
+				val  string
+			}{
+				{"", ""},
+			},
+		},
+	} {
+		client := &testHttpClient{s.resources, s.err}
+		for _, tt := range s.tests {
+			attr, err := fetchAttribute(client, tt.path)
+			if err != s.err {
+				t.Fatalf("bad error for %q (%q): want %q, got %q", tt.path, s.resources, s.err, err)
+			}
+			if attr != tt.val {
+				t.Fatalf("bad fetch for %q (%q): want %q, got %q", tt.path, s.resources, tt.val, attr)
+			}
+		}
+	}
+}
+
+func TestFetchMetadata(t *testing.T) {
+	for _, tt := range []struct {
+		root      string
+		resources map[string]string
+		expect    []byte
+		clientErr error
+		expectErr error
+	}{
+		{
+			root: "/",
+			resources: map[string]string{
+				"/2009-04-04/meta-data/public-keys": "bad\n",
+			},
+			expectErr: fmt.Errorf("malformed public key: \"bad\""),
+		},
+		{
+			root: "/",
+			resources: map[string]string{
+				"/2009-04-04/meta-data/hostname":                    "host",
+				"/2009-04-04/meta-data/local-ipv4":                  "1.2.3.4",
+				"/2009-04-04/meta-data/public-ipv4":                 "5.6.7.8",
+				"/2009-04-04/meta-data/public-keys":                 "0=test1\n",
+				"/2009-04-04/meta-data/public-keys/0":               "openssh-key",
+				"/2009-04-04/meta-data/public-keys/0/openssh-key":   "key",
+				"/2009-04-04/meta-data/network_config/content_path": "path",
+			},
+			expect: []byte(`{"hostname":"host","local-ipv4":"1.2.3.4","network_config":{"content_path":"path"},"public-ipv4":"5.6.7.8","public_keys":{"test1":"key"}}`),
+		},
+		{
+			clientErr: pkg.ErrTimeout{fmt.Errorf("test error")},
+			expectErr: pkg.ErrTimeout{fmt.Errorf("test error")},
+		},
+	} {
+		service := &metadataService{tt.root, &testHttpClient{tt.resources, tt.clientErr}}
+		metadata, err := service.FetchMetadata()
+		if Error(err) != Error(tt.expectErr) {
+			t.Fatalf("bad error (%q): want %q, got %q", tt.resources, tt.expectErr, err)
+		}
+		if !bytes.Equal(metadata, tt.expect) {
+			t.Fatalf("bad fetch (%q): want %q, got %q", tt.resources, tt.expect, metadata)
+		}
+	}
+}
+
+func TestNewDatasource(t *testing.T) {
+	for _, tt := range []struct {
+		root       string
+		expectRoot string
+	}{
+		{
+			root:       "",
+			expectRoot: "/",
+		},
+		{
+			root:       "/",
+			expectRoot: "/",
+		},
+		{
+			root:       "http://169.254.169.254",
+			expectRoot: "http://169.254.169.254/",
+		},
+		{
+			root:       "http://169.254.169.254/",
+			expectRoot: "http://169.254.169.254/",
+		},
+	} {
+		service := NewDatasource(tt.root)
+		if service.root != tt.expectRoot {
+			t.Fatalf("bad root (%q): want %q, got %q", tt.root, tt.expectRoot, service.root)
+		}
+	}
+}
+
+func Error(err error) string {
+	if err != nil {
+		return err.Error()
+	}
+	return ""
+}

datasource/metadata_service.go

@@ -1,129 +0,0 @@
-package datasource
-
-import (
-	"bufio"
-	"bytes"
-	"encoding/json"
-	"strings"
-
-	"github.com/coreos/coreos-cloudinit/pkg"
-)
-
-// metadataService retrieves metadata from either an OpenStack[1] (2012-08-10)
-// or EC2[2] (2009-04-04) compatible endpoint. It will first attempt to
-// directly retrieve a JSON blob from the OpenStack endpoint. If that fails
-// with a 404, it then attempts to retrieve metadata bit-by-bit from the EC2
-// endpoint, and populates that into an equivalent JSON blob. metadataService
-// also checks for userdata from EC2 and, if that fails with a 404, OpenStack.
-//
-// [1] http://docs.openstack.org/grizzly/openstack-compute/admin/content/metadata-service.html
-// [2] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html#instancedata-data-categories
-
-const (
-	BaseUrl              = "http://169.254.169.254/"
-	Ec2ApiVersion        = "2009-04-04"
-	Ec2UserdataUrl       = BaseUrl + Ec2ApiVersion + "/user-data"
-	Ec2MetadataUrl       = BaseUrl + Ec2ApiVersion + "/meta-data/"
-	OpenstackApiVersion  = "openstack/2012-08-10"
-	OpenstackUserdataUrl = BaseUrl + OpenstackApiVersion + "/user_data"
-	OpenstackMetadataUrl = BaseUrl + OpenstackApiVersion + "/meta_data.json"
-)
-
-type metadataService struct{}
-
-type getter interface {
-	GetRetry(string) ([]byte, error)
-}
-
-func NewMetadataService() *metadataService {
-	return &metadataService{}
-}
-
-func (ms *metadataService) IsAvailable() bool {
-	client := pkg.NewHttpClient()
-	_, err := client.Get(BaseUrl)
-	return (err == nil)
-}
-
-func (ms *metadataService) AvailabilityChanges() bool {
-	return true
-}
-
-func (ms *metadataService) ConfigRoot() string {
-	return ""
-}
-
-func (ms *metadataService) FetchMetadata() ([]byte, error) {
-	return fetchMetadata(pkg.NewHttpClient())
-}
-
-func (ms *metadataService) FetchUserdata() ([]byte, error) {
-	client := pkg.NewHttpClient()
-	if data, err := client.GetRetry(Ec2UserdataUrl); err == nil {
-		return data, err
-	} else if _, ok := err.(pkg.ErrTimeout); ok {
-		return data, err
-	}
-	return client.GetRetry(OpenstackUserdataUrl)
-}
-
-func (ms *metadataService) Type() string {
-	return "metadata-service"
-}
-
-func fetchMetadata(client getter) ([]byte, error) {
-	if metadata, err := client.GetRetry(OpenstackMetadataUrl); err == nil {
-		return metadata, nil
-	} else if _, ok := err.(pkg.ErrTimeout); ok {
-		return nil, err
-	}
-
-	attrs, err := fetchChildAttributes(client, Ec2MetadataUrl)
-	if err != nil {
-		return nil, err
-	}
-	return json.Marshal(attrs)
-}
-
-func fetchAttributes(client getter, url string) ([]string, error) {
-	resp, err := client.GetRetry(url)
-	if err != nil {
-		return nil, err
-	}
-	scanner := bufio.NewScanner(bytes.NewBuffer(resp))
-	data := make([]string, 0)
-	for scanner.Scan() {
-		data = append(data, strings.Split(scanner.Text(), "=")[0])
-	}
-	return data, scanner.Err()
-}
-
-func fetchAttribute(client getter, url string) (interface{}, error) {
-	if attrs, err := fetchAttributes(client, url); err == nil && len(attrs) > 0 {
-		return attrs[0], nil
-	} else {
-		return "", err
-	}
-}
-
-func fetchChildAttributes(client getter, url string) (interface{}, error) {
-	attrs := make(map[string]interface{})
-	attrList, err := fetchAttributes(client, url)
-	if err != nil {
-		return nil, err
-	}
-	for _, attr := range attrList {
-		var fetchFunc func(getter, string) (interface{}, error)
-		if strings.HasSuffix(attr, "/") {
-			fetchFunc = fetchChildAttributes
-		} else {
-			fetchFunc = fetchAttribute
-		}
-		if value, err := fetchFunc(client, url+attr); err == nil {
-			attrs[strings.TrimSuffix(attr, "/")] = value
-		} else {
-			return nil, err
-		}
-	}
-	return attrs, nil
-}

datasource/metadata_service_test.go

@@ -1,168 +0,0 @@
-package datasource
-
-import (
-	"bytes"
-	"fmt"
-	"reflect"
-	"testing"
-
-	"github.com/coreos/coreos-cloudinit/pkg"
-)
-
-type TestHttpClient struct {
-	metadata map[string]string
-	err      error
-}
-
-func (t *TestHttpClient) GetRetry(url string) ([]byte, error) {
-	if t.err != nil {
-		return nil, t.err
-	}
-	if val, ok := t.metadata[url]; ok {
-		return []byte(val), nil
-	} else {
-		return nil, pkg.ErrNotFound{fmt.Errorf("not found: %q", url)}
-	}
-}
-
-func TestFetchAttributes(t *testing.T) {
-	for _, s := range []struct {
-		metadata map[string]string
-		err      error
-		tests    []struct {
-			path string
-			val  []string
-		}
-	}{
-		{
-			metadata: map[string]string{
-				"/":      "a\nb\nc/",
-				"/c/":    "d\ne/",
-				"/c/e/":  "f",
-				"/a":     "1",
-				"/b":     "2",
-				"/c/d":   "3",
-				"/c/e/f": "4",
-			},
-			tests: []struct {
-				path string
-				val  []string
-			}{
-				{"/", []string{"a", "b", "c/"}},
-				{"/b", []string{"2"}},
-				{"/c/d", []string{"3"}},
-				{"/c/e/", []string{"f"}},
-			},
-		},
-		{
-			err: pkg.ErrNotFound{fmt.Errorf("test error")},
-			tests: []struct {
-				path string
-				val  []string
-			}{
-				{"", nil},
-			},
-		},
-	} {
-		client := &TestHttpClient{s.metadata, s.err}
-		for _, tt := range s.tests {
-			attrs, err := fetchAttributes(client, tt.path)
-			if err != s.err {
-				t.Fatalf("bad error for %q (%q): want %q, got %q", tt.path, s.metadata, s.err, err)
-			}
-			if !reflect.DeepEqual(attrs, tt.val) {
-				t.Fatalf("bad fetch for %q (%q): want %q, got %q", tt.path, s.metadata, tt.val, attrs)
-			}
-		}
-	}
-}
-
-func TestFetchAttribute(t *testing.T) {
-	for _, s := range []struct {
-		metadata map[string]string
-		err      error
-		tests    []struct {
-			path string
-			val  string
-		}
-	}{
-		{
-			metadata: map[string]string{
-				"/":      "a\nb\nc/",
-				"/c/":    "d\ne/",
-				"/c/e/":  "f",
-				"/a":     "1",
-				"/b":     "2",
-				"/c/d":   "3",
-				"/c/e/f": "4",
-			},
-			tests: []struct {
-				path string
-				val  string
-			}{
-				{"/a", "1"},
-				{"/b", "2"},
-				{"/c/d", "3"},
-				{"/c/e/f", "4"},
-			},
-		},
-		{
-			err: pkg.ErrNotFound{fmt.Errorf("test error")},
-			tests: []struct {
-				path string
-				val  string
-			}{
-				{"", ""},
-			},
-		},
-	} {
-		client := &TestHttpClient{s.metadata, s.err}
-		for _, tt := range s.tests {
-			attr, err := fetchAttribute(client, tt.path)
-			if err != s.err {
-				t.Fatalf("bad error for %q (%q): want %q, got %q", tt.path, s.metadata, s.err, err)
-			}
-			if attr != tt.val {
-				t.Fatalf("bad fetch for %q (%q): want %q, got %q", tt.path, s.metadata, tt.val, attr)
-			}
-		}
-	}
-}
-
-func TestFetchMetadata(t *testing.T) {
-	for _, tt := range []struct {
-		metadata map[string]string
-		err      error
-		expect   []byte
-	}{
-		{
-			metadata: map[string]string{
-				"http://169.254.169.254/2009-04-04/meta-data/":      "a\nb\nc/",
-				"http://169.254.169.254/2009-04-04/meta-data/c/":    "d\ne/",
-				"http://169.254.169.254/2009-04-04/meta-data/c/e/":  "f",
-				"http://169.254.169.254/2009-04-04/meta-data/a":     "1",
-				"http://169.254.169.254/2009-04-04/meta-data/b":     "2",
-				"http://169.254.169.254/2009-04-04/meta-data/c/d":   "3",
-				"http://169.254.169.254/2009-04-04/meta-data/c/e/f": "4",
-			},
-			expect: []byte(`{"a":"1","b":"2","c":{"d":"3","e":{"f":"4"}}}`),
-		},
-		{
-			metadata: map[string]string{
-				"http://169.254.169.254/openstack/2012-08-10/meta_data.json": "test",
-			},
-			expect: []byte("test"),
-		},
-		{err: pkg.ErrTimeout{fmt.Errorf("test error")}},
-		{err: pkg.ErrNotFound{fmt.Errorf("test error")}},
-	} {
-		client := &TestHttpClient{tt.metadata, tt.err}
-		metadata, err := fetchMetadata(client)
-		if err != tt.err {
-			t.Fatalf("bad error (%q): want %q, got %q", tt.metadata, tt.err, err)
-		}
-		if !bytes.Equal(metadata, tt.expect) {
-			t.Fatalf("bad fetch (%q): want %q, got %q", tt.metadata, tt.expect, metadata)
-		}
-	}
-}

datasource/proc_cmdline/proc_cmdline.go

@@ -1,4 +1,4 @@
-package datasource
+package proc_cmdline
 
 import (
 	"errors"
@@ -18,7 +18,7 @@ type procCmdline struct {
 	Location string
 }
 
-func NewProcCmdline() *procCmdline {
+func NewDatasource() *procCmdline {
 	return &procCmdline{Location: ProcCmdlineLocation}
 }
 

datasource/proc_cmdline/proc_cmdline_test.go

@@ -1,4 +1,4 @@
-package datasource
+package proc_cmdline
 
 import (
 	"fmt"
@@ -75,7 +75,7 @@ func TestProcCmdlineAndFetchConfig(t *testing.T) {
 		t.Errorf("Test produced error: %v", err)
 	}
 
-	p := NewProcCmdline()
+	p := NewDatasource()
 	p.Location = file.Name()
 	cfg, err := p.FetchUserdata()
 	if err != nil {

datasource/url/url.go

@@ -1,4 +1,4 @@
-package datasource
+package url
 
 import "github.com/coreos/coreos-cloudinit/pkg"
 
@@ -6,7 +6,7 @@ type remoteFile struct {
 	url string
 }
 
-func NewRemoteFile(url string) *remoteFile {
+func NewDatasource(url string) *remoteFile {
 	return &remoteFile{url}
 }
 

initialize/config.go

@@ -66,51 +66,62 @@ func warnOnUnrecognizedKeys(contents string, warn warner) {
 	}
 
 	// Check for unrecognized coreos options, if any are set
-	coreos, ok := c["coreos"]
+	if coreos, ok := c["coreos"]; ok {
-	if ok {
+		if set, ok := coreos.(map[interface{}]interface{}); ok {
-		set := coreos.(map[interface{}]interface{})
+			known := cc["coreos"].(map[interface{}]interface{})
-		known := cc["coreos"].(map[interface{}]interface{})
+			for k, _ := range set {
-		for k, _ := range set {
+				if key, ok := k.(string); ok {
-			key := k.(string)
+					if _, ok := known[key]; !ok {
-			if _, ok := known[key]; !ok {
+						warn("Warning: unrecognized key %q in coreos section of provided cloud config - ignoring", key)
-				warn("Warning: unrecognized key %q in coreos section of provided cloud config - ignoring", key)
+					}
+				} else {
+					warn("Warning: unrecognized key %q in coreos section of provided cloud config - ignoring", k)
+				}
 			}
 		}
 	}
 
 	// Check for any badly-specified users, if any are set
-	users, ok := c["users"]
+	if users, ok := c["users"]; ok {
-	if ok {
 		var known map[string]interface{}
 		b, _ := goyaml.Marshal(&system.User{})
 		goyaml.Unmarshal(b, &known)
 
-		set := users.([]interface{})
+		if set, ok := users.([]interface{}); ok {
-		for _, u := range set {
+			for _, u := range set {
-			user := u.(map[interface{}]interface{})
+				if user, ok := u.(map[interface{}]interface{}); ok {
-			for k, _ := range user {
+					for k, _ := range user {
-				key := k.(string)
+						if key, ok := k.(string); ok {
-				if _, ok := known[key]; !ok {
+							if _, ok := known[key]; !ok {
-					warn("Warning: unrecognized key %q in user section of cloud config - ignoring", key)
+								warn("Warning: unrecognized key %q in user section of cloud config - ignoring", key)
+							}
+						} else {
+							warn("Warning: unrecognized key %q in user section of cloud config - ignoring", k)
+						}
+					}
 				}
 			}
 		}
 	}
 
 	// Check for any badly-specified files, if any are set
-	files, ok := c["write_files"]
+	if files, ok := c["write_files"]; ok {
-	if ok {
 		var known map[string]interface{}
 		b, _ := goyaml.Marshal(&system.File{})
 		goyaml.Unmarshal(b, &known)
 
-		set := files.([]interface{})
+		if set, ok := files.([]interface{}); ok {
-		for _, f := range set {
+			for _, f := range set {
-			file := f.(map[interface{}]interface{})
+				if file, ok := f.(map[interface{}]interface{}); ok {
-			for k, _ := range file {
+					for k, _ := range file {
-				key := k.(string)
+						if key, ok := k.(string); ok {
-				if _, ok := known[key]; !ok {
+							if _, ok := known[key]; !ok {
-					warn("Warning: unrecognized key %q in file section of cloud config - ignoring", key)
+								warn("Warning: unrecognized key %q in file section of cloud config - ignoring", key)
+							}
+						} else {
+							warn("Warning: unrecognized key %q in file section of cloud config - ignoring", k)
+						}
+					}
 				}
 			}
 		}
@@ -223,16 +234,33 @@ func Apply(cfg CloudConfig, env *Environment) error {
 		cfg.Coreos.Units = append(cfg.Coreos.Units, u...)
 	}
 
+	wroteEnvironment := false
 	for _, file := range cfg.WriteFiles {
-		path, err := system.WriteFile(&file, env.Root())
+		fullPath, err := system.WriteFile(&file, env.Root())
 		if err != nil {
 			return err
 		}
-		log.Printf("Wrote file %s to filesystem", path)
+		if path.Clean(file.Path) == "/etc/environment" {
+			wroteEnvironment = true
+		}
+		log.Printf("Wrote file %s to filesystem", fullPath)
+	}
+
+	if !wroteEnvironment {
+		ef := env.DefaultEnvironmentFile()
+		if ef != nil {
+			err := system.WriteEnvFile(ef, env.Root())
+			if err != nil {
+				return err
+			}
+			log.Printf("Updated /etc/environment")
+		}
 	}
 
 	if env.NetconfType() != "" {
-		netconfBytes, err := ioutil.ReadFile(path.Join(env.ConfigRoot(), cfg.NetworkConfigPath))
+		filename := path.Join(env.ConfigRoot(), cfg.NetworkConfigPath)
+		log.Printf("Attempting to read config from %q\n", filename)
+		netconfBytes, err := ioutil.ReadFile(filename)
 		if err != nil {
 			return err
 		}
@@ -257,13 +285,23 @@ func Apply(cfg CloudConfig, env *Environment) error {
 		}
 	}
 
+	um := system.NewUnitManager(env.Root())
+	return processUnits(cfg.Coreos.Units, env.Root(), um)
+
+}
+
+// processUnits takes a set of Units and applies them to the given root using
+// the given UnitManager. This can involve things like writing unit files to
+// disk, masking/unmasking units, or invoking systemd
+// commands against units. It returns any error encountered.
+func processUnits(units []system.Unit, root string, um system.UnitManager) error {
 	commands := make(map[string]string, 0)
 	reload := false
-	for _, unit := range cfg.Coreos.Units {
+	for _, unit := range units {
-		dst := unit.Destination(env.Root())
+		dst := unit.Destination(root)
 		if unit.Content != "" {
 			log.Printf("Writing unit %s to filesystem at path %s", unit.Name, dst)
-			if err := system.PlaceUnit(&unit, dst); err != nil {
+			if err := um.PlaceUnit(&unit, dst); err != nil {
 				return err
 			}
 			log.Printf("Placed unit %s at %s", unit.Name, dst)
@@ -272,12 +310,12 @@ func Apply(cfg CloudConfig, env *Environment) error {
 
 		if unit.Mask {
 			log.Printf("Masking unit file %s", unit.Name)
-			if err := system.MaskUnit(&unit, env.Root()); err != nil {
+			if err := um.MaskUnit(&unit); err != nil {
 				return err
 			}
 		} else if unit.Runtime {
 			log.Printf("Ensuring runtime unit file %s is unmasked", unit.Name)
-			if err := system.UnmaskUnit(&unit, env.Root()); err != nil {
+			if err := um.UnmaskUnit(&unit); err != nil {
 				return err
 			}
 		}
@@ -285,7 +323,7 @@ func Apply(cfg CloudConfig, env *Environment) error {
 		if unit.Enable {
 			if unit.Group() != "network" {
 				log.Printf("Enabling unit file %s", unit.Name)
-				if err := system.EnableUnitFile(unit.Name, unit.Runtime); err != nil {
+				if err := um.EnableUnitFile(unit.Name, unit.Runtime); err != nil {
 					return err
 				}
 				log.Printf("Enabled unit %s", unit.Name)
@@ -302,14 +340,14 @@ func Apply(cfg CloudConfig, env *Environment) error {
 	}
 
 	if reload {
-		if err := system.DaemonReload(); err != nil {
+		if err := um.DaemonReload(); err != nil {
 			return errors.New(fmt.Sprintf("failed systemd daemon-reload: %v", err))
 		}
 	}
 
 	for unit, command := range commands {
 		log.Printf("Calling unit command '%s %s'", command, unit)
-		res, err := system.RunUnitCommand(command, unit)
+		res, err := um.RunUnitCommand(command, unit)
 		if err != nil {
 			return err
 		}

initialize/config_test.go

@@ -4,8 +4,38 @@ import (
 	"fmt"
 	"strings"
 	"testing"
+
+	"github.com/coreos/coreos-cloudinit/system"
 )
 
+func TestCloudConfigInvalidKeys(t *testing.T) {
+	defer func() {
+		if r := recover(); r != nil {
+			t.Fatalf("panic while instantiating CloudConfig with nil keys: %v", r)
+		}
+	}()
+
+	for _, tt := range []struct {
+		contents string
+	}{
+		{"coreos:"},
+		{"ssh_authorized_keys:"},
+		{"ssh_authorized_keys:\n  -"},
+		{"ssh_authorized_keys:\n  - 0:"},
+		{"write_files:"},
+		{"write_files:\n  -"},
+		{"write_files:\n  - 0:"},
+		{"users:"},
+		{"users:\n  -"},
+		{"users:\n  - 0:"},
+	} {
+		_, err := NewCloudConfig(tt.contents)
+		if err != nil {
+			t.Fatalf("error instantiating CloudConfig with invalid keys: %v", err)
+		}
+	}
+}
+
 func TestCloudConfigUnknownKeys(t *testing.T) {
 	contents := `
 coreos: 
@@ -332,3 +362,109 @@ users:
 		t.Errorf("Failed to parse no-log-init field")
 	}
 }
+
+type TestUnitManager struct {
+	placed   []string
+	enabled  []string
+	masked   []string
+	unmasked []string
+	commands map[string]string
+	reload   bool
+}
+
+func (tum *TestUnitManager) PlaceUnit(unit *system.Unit, dst string) error {
+	tum.placed = append(tum.placed, unit.Name)
+	return nil
+}
+func (tum *TestUnitManager) EnableUnitFile(unit string, runtime bool) error {
+	tum.enabled = append(tum.enabled, unit)
+	return nil
+}
+func (tum *TestUnitManager) RunUnitCommand(command, unit string) (string, error) {
+	tum.commands = make(map[string]string)
+	tum.commands[unit] = command
+	return "", nil
+}
+func (tum *TestUnitManager) DaemonReload() error {
+	tum.reload = true
+	return nil
+}
+func (tum *TestUnitManager) MaskUnit(unit *system.Unit) error {
+	tum.masked = append(tum.masked, unit.Name)
+	return nil
+}
+func (tum *TestUnitManager) UnmaskUnit(unit *system.Unit) error {
+	tum.unmasked = append(tum.unmasked, unit.Name)
+	return nil
+}
+
+func TestProcessUnits(t *testing.T) {
+	tum := &TestUnitManager{}
+	units := []system.Unit{
+		system.Unit{
+			Name: "foo",
+			Mask: true,
+		},
+	}
+	if err := processUnits(units, "", tum); err != nil {
+		t.Fatalf("unexpected error calling processUnits: %v", err)
+	}
+	if len(tum.masked) != 1 || tum.masked[0] != "foo" {
+		t.Errorf("expected foo to be masked, but found %v", tum.masked)
+	}
+
+	tum = &TestUnitManager{}
+	units = []system.Unit{
+		system.Unit{
+			Name: "bar.network",
+		},
+	}
+	if err := processUnits(units, "", tum); err != nil {
+		t.Fatalf("unexpected error calling processUnits: %v", err)
+	}
+	if _, ok := tum.commands["systemd-networkd.service"]; !ok {
+		t.Errorf("expected systemd-networkd.service to be reloaded!")
+	}
+
+	tum = &TestUnitManager{}
+	units = []system.Unit{
+		system.Unit{
+			Name:    "baz.service",
+			Content: "[Service]\nExecStart=/bin/true",
+		},
+	}
+	if err := processUnits(units, "", tum); err != nil {
+		t.Fatalf("unexpected error calling processUnits: %v", err)
+	}
+	if len(tum.placed) != 1 || tum.placed[0] != "baz.service" {
+		t.Fatalf("expected baz.service to be written, but got %v", tum.placed)
+	}
+
+	tum = &TestUnitManager{}
+	units = []system.Unit{
+		system.Unit{
+			Name:    "locksmithd.service",
+			Runtime: true,
+		},
+	}
+	if err := processUnits(units, "", tum); err != nil {
+		t.Fatalf("unexpected error calling processUnits: %v", err)
+	}
+	if len(tum.unmasked) != 1 || tum.unmasked[0] != "locksmithd.service" {
+		t.Fatalf("expected locksmithd.service to be unmasked, but got %v", tum.unmasked)
+	}
+
+	tum = &TestUnitManager{}
+	units = []system.Unit{
+		system.Unit{
+			Name:   "woof",
+			Enable: true,
+		},
+	}
+	if err := processUnits(units, "", tum); err != nil {
+		t.Fatalf("unexpected error calling processUnits: %v", err)
+	}
+	if len(tum.enabled) != 1 || tum.enabled[0] != "woof" {
+		t.Fatalf("expected woof to be enabled, but got %v", tum.enabled)
+	}
+}

initialize/env.go

@@ -4,6 +4,8 @@ import (
 	"os"
 	"path"
 	"strings"
+
+	"github.com/coreos/coreos-cloudinit/system"
 )
 
 const DefaultSSHKeyName = "coreos-cloudinit"
@@ -65,6 +67,26 @@ func (e *Environment) Apply(data string) string {
 	return data
 }
 
+func (e *Environment) DefaultEnvironmentFile() *system.EnvFile {
+	ef := system.EnvFile{
+		File: &system.File{
+			Path: "/etc/environment",
+		},
+		Vars: map[string]string{},
+	}
+	if ip, ok := e.substitutions["$public_ipv4"]; ok && len(ip) > 0 {
+		ef.Vars["COREOS_PUBLIC_IPV4"] = ip
+	}
+	if ip, ok := e.substitutions["$private_ipv4"]; ok && len(ip) > 0 {
+		ef.Vars["COREOS_PRIVATE_IPV4"] = ip
+	}
+	if len(ef.Vars) == 0 {
+		return nil
+	} else {
+		return &ef
+	}
+}
+
 // normalizeSvcEnv standardizes the keys of the map (environment variables for a service)
 // by replacing any dashes with underscores and ensuring they are entirely upper case.
 // For example, "some-env" --> "SOME_ENV"

initialize/env_test.go

@@ -1,8 +1,12 @@
 package initialize
 
 import (
+	"io/ioutil"
 	"os"
+	"path"
 	"testing"
+
+	"github.com/coreos/coreos-cloudinit/system"
 )
 
 func TestEnvironmentApply(t *testing.T) {
@@ -56,3 +60,47 @@ ExecStop=/usr/bin/echo $unknown`,
 		}
 	}
 }
+
+func TestEnvironmentFile(t *testing.T) {
+	subs := map[string]string{
+		"$public_ipv4":  "1.2.3.4",
+		"$private_ipv4": "5.6.7.8",
+	}
+	expect := "COREOS_PRIVATE_IPV4=5.6.7.8\nCOREOS_PUBLIC_IPV4=1.2.3.4\n"
+
+	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
+	if err != nil {
+		t.Fatalf("Unable to create tempdir: %v", err)
+	}
+	defer os.RemoveAll(dir)
+
+	env := NewEnvironment("./", "./", "./", "", "", subs)
+	ef := env.DefaultEnvironmentFile()
+	err = system.WriteEnvFile(ef, dir)
+	if err != nil {
+		t.Fatalf("WriteEnvFile failed: %v", err)
+	}
+
+	fullPath := path.Join(dir, "etc", "environment")
+	contents, err := ioutil.ReadFile(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to read expected file: %v", err)
+	}
+
+	if string(contents) != expect {
+		t.Fatalf("File has incorrect contents: %q", contents)
+	}
+}
+
+func TestEnvironmentFileNil(t *testing.T) {
+	subs := map[string]string{
+		"$public_ipv4":  "",
+		"$private_ipv4": "",
+	}
+
+	env := NewEnvironment("./", "./", "./", "", "", subs)
+	ef := env.DefaultEnvironmentFile()
+	if ef != nil {
+		t.Fatalf("Environment file not nil: %v", ef)
+	}
+}

initialize/etcd.go

@@ -39,7 +39,7 @@ func (ee EtcdEnvironment) String() (out string) {
 // Units creates a Unit file drop-in for etcd, using any configured
 // options and adding a default MachineID if unset.
 func (ee EtcdEnvironment) Units(root string) ([]system.Unit, error) {
-	if ee == nil {
+	if len(ee) < 1 {
 		return nil, nil
 	}
 

initialize/etcd_test.go

@@ -70,6 +70,8 @@ func TestEtcdEnvironmentWrittenToDisk(t *testing.T) {
 	}
 	defer os.RemoveAll(dir)
 
+	sd := system.NewUnitManager(dir)
+
 	uu, err := ee.Units(dir)
 	if err != nil {
 		t.Fatalf("Generating etcd unit failed: %v", err)
@@ -81,7 +83,7 @@ func TestEtcdEnvironmentWrittenToDisk(t *testing.T) {
 
 	dst := u.Destination(dir)
 	os.Stderr.WriteString("writing to " + dir + "\n")
-	if err := system.PlaceUnit(&u, dst); err != nil {
+	if err := sd.PlaceUnit(&u, dst); err != nil {
 		t.Fatalf("Writing of EtcdEnvironment failed: %v", err)
 	}
 
@@ -111,14 +113,27 @@ Environment="ETCD_PEER_BIND_ADDR=127.0.0.1:7002"
 	}
 }
 
-func TestEtcdEnvironmentWrittenToDiskDefaultToMachineID(t *testing.T) {
+func TestEtcdEnvironmentEmptyNoOp(t *testing.T) {
 	ee := EtcdEnvironment{}
+	uu, err := ee.Units("")
+	if err != nil {
+		t.Fatalf("Unexpected error: %v", err)
+	}
+	if len(uu) > 0 {
+		t.Fatalf("Generated etcd units unexpectedly: %v")
+	}
+}
+
+func TestEtcdEnvironmentWrittenToDiskDefaultToMachineID(t *testing.T) {
+	ee := EtcdEnvironment{"foo": "bar"}
 	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
 	if err != nil {
 		t.Fatalf("Unable to create tempdir: %v", err)
 	}
 	defer os.RemoveAll(dir)
 
+	sd := system.NewUnitManager(dir)
+
 	os.Mkdir(path.Join(dir, "etc"), os.FileMode(0755))
 	err = ioutil.WriteFile(path.Join(dir, "etc", "machine-id"), []byte("node007"), os.FileMode(0444))
 	if err != nil {
@@ -136,7 +151,7 @@ func TestEtcdEnvironmentWrittenToDiskDefaultToMachineID(t *testing.T) {
 
 	dst := u.Destination(dir)
 	os.Stderr.WriteString("writing to " + dir + "\n")
-	if err := system.PlaceUnit(&u, dst); err != nil {
+	if err := sd.PlaceUnit(&u, dst); err != nil {
 		t.Fatalf("Writing of EtcdEnvironment failed: %v", err)
 	}
 
@@ -148,6 +163,7 @@ func TestEtcdEnvironmentWrittenToDiskDefaultToMachineID(t *testing.T) {
 	}
 
 	expect := `[Service]
+Environment="ETCD_FOO=bar"
 Environment="ETCD_NAME=node007"
 `
 	if string(contents) != expect {

initialize/meta_data.go

@@ -3,8 +3,11 @@ package initialize
 import "encoding/json"
 
 // ParseMetaData parses a JSON blob in the OpenStack metadata service format, and
-// converts it to a CloudConfig
+// converts it to a partially hydrated CloudConfig
-func ParseMetaData(contents string) (cfg CloudConfig, err error) {
+func ParseMetaData(contents string) (*CloudConfig, error) {
+	if len(contents) == 0 {
+		return nil, nil
+	}
 	var metadata struct {
 		SSHAuthorizedKeyMap map[string]string `json:"public_keys"`
 		Hostname            string            `json:"hostname"`
@@ -12,17 +15,20 @@ func ParseMetaData(contents string) (cfg CloudConfig, err error) {
 			ContentPath string `json:"content_path"`
 		} `json:"network_config"`
 	}
-	if err = json.Unmarshal([]byte(contents), &metadata); err != nil {
+	if err := json.Unmarshal([]byte(contents), &metadata); err != nil {
-		return
+		return nil, err
 	}
 
-	cfg.SSHAuthorizedKeys = make([]string, 0, len(metadata.SSHAuthorizedKeyMap))
+	var cfg CloudConfig
-	for _, key := range metadata.SSHAuthorizedKeyMap {
+	if len(metadata.SSHAuthorizedKeyMap) > 0 {
-		cfg.SSHAuthorizedKeys = append(cfg.SSHAuthorizedKeys, key)
+		cfg.SSHAuthorizedKeys = make([]string, 0, len(metadata.SSHAuthorizedKeyMap))
+		for _, key := range metadata.SSHAuthorizedKeyMap {
+			cfg.SSHAuthorizedKeys = append(cfg.SSHAuthorizedKeys, key)
+		}
 	}
 	cfg.Hostname = metadata.Hostname
 	cfg.NetworkConfigPath = metadata.NetworkConfig.ContentPath
-	return
+	return &cfg, nil
 }
 
 // ExtractIPsFromMetaData parses a JSON blob in the OpenStack metadata service format,

initialize/meta_data_test.go

@@ -3,6 +3,39 @@ package initialize
 import "reflect"
 import "testing"
 
+func TestParseMetadata(t *testing.T) {
+	for i, tt := range []struct {
+		in   string
+		want *CloudConfig
+		err  bool
+	}{
+		{"", nil, false},
+		{`garbage, invalid json`, nil, true},
+		{`{"foo": "bar"}`, &CloudConfig{}, false},
+		{`{"network_config": {"content_path": "asdf"}}`, &CloudConfig{NetworkConfigPath: "asdf"}, false},
+		{`{"hostname": "turkleton"}`, &CloudConfig{Hostname: "turkleton"}, false},
+		{`{"public_keys": {"jack": "jill", "bob": "alice"}}`, &CloudConfig{SSHAuthorizedKeys: []string{"jill", "alice"}}, false},
+		{`{"unknown": "thing", "hostname": "my_host", "public_keys": {"do": "re", "mi": "fa"}, "network_config": {"content_path": "/root", "blah": "zzz"}}`, &CloudConfig{SSHAuthorizedKeys: []string{"re", "fa"}, Hostname: "my_host", NetworkConfigPath: "/root"}, false},
+	} {
+		got, err := ParseMetaData(tt.in)
+		if tt.err != (err != nil) {
+			t.Errorf("case #%d: bad error state: got %t, want %t (err=%v)", i, (err != nil), tt.err, err)
+		}
+		if got == nil {
+			if tt.want != nil {
+				t.Errorf("case #%d: unexpected nil output", i)
+			}
+		} else if tt.want == nil {
+			t.Errorf("case #%d: unexpected non-nil output", i)
+		} else {
+			if !reflect.DeepEqual(*got, *tt.want) {
+				t.Errorf("case #%d: bad output:\ngot\n%v\nwant\n%v", i, *got, *tt.want)
+			}
+		}
+	}
+
+}
+
 func TestExtractIPsFromMetadata(t *testing.T) {
 	for i, tt := range []struct {
 		in  []byte

initialize/user_data.go

@@ -9,6 +9,9 @@ import (
 )
 
 func ParseUserData(contents string) (interface{}, error) {
+	if len(contents) == 0 {
+		return nil, nil
+	}
 	header := strings.SplitN(contents, "\n", 2)[0]
 
 	// Explicitly trim the header so we can handle user-data from
@@ -19,14 +22,9 @@ func ParseUserData(contents string) (interface{}, error) {
 	if strings.HasPrefix(header, "#!") {
 		log.Printf("Parsing user-data as script")
 		return system.Script(contents), nil
-
 	} else if header == "#cloud-config" {
 		log.Printf("Parsing user-data as cloud-config")
-		cfg, err := NewCloudConfig(contents)
+		return NewCloudConfig(contents)
-		if err != nil {
-			log.Fatal(err.Error())
-		}
-		return *cfg, nil
 	} else {
 		return nil, fmt.Errorf("Unrecognized user-data header: %s", header)
 	}

initialize/user_data_test.go

@@ -37,7 +37,7 @@ func TestParseConfigCRLF(t *testing.T) {
 		t.Fatalf("Failed parsing config: %v", err)
 	}
 
-	cfg := ud.(CloudConfig)
+	cfg := ud.(*CloudConfig)
 
 	if cfg.Hostname != "foo" {
 		t.Error("Failed parsing hostname from config")
@@ -47,3 +47,12 @@ func TestParseConfigCRLF(t *testing.T) {
 		t.Error("Parsed incorrect number of SSH keys")
 	}
 }
+
+func TestParseConfigEmpty(t *testing.T) {
+	i, e := ParseUserData(``)
+	if i != nil {
+		t.Error("ParseUserData of empty string returned non-nil unexpectedly")
+	} else if e != nil {
+		t.Error("ParseUserData of empty string returned error unexpectedly")
+	}
+}

network/interface.go

@@ -3,6 +3,7 @@ package network
 import (
 	"fmt"
 	"strconv"
+	"strings"
 )
 
 type InterfaceGenerator interface {
@@ -11,6 +12,8 @@ type InterfaceGenerator interface {
 	Netdev() string
 	Link() string
 	Network() string
+	Type() string
+	ModprobeParams() string
 }
 
 type networkInterface interface {
@@ -68,6 +71,10 @@ func (i *logicalInterface) Children() []networkInterface {
 	return i.children
 }
 
+func (i *logicalInterface) ModprobeParams() string {
+	return ""
+}
+
 func (i *logicalInterface) setConfigDepth(depth int) {
 	i.configDepth = depth
 }
@@ -84,9 +91,14 @@ func (p *physicalInterface) Netdev() string {
 	return ""
 }
 
+func (p *physicalInterface) Type() string {
+	return "physical"
+}
+
 type bondInterface struct {
 	logicalInterface
-	slaves []string
+	slaves  []string
+	options map[string]string
 }
 
 func (b *bondInterface) Name() string {
@@ -97,6 +109,19 @@ func (b *bondInterface) Netdev() string {
 	return fmt.Sprintf("[NetDev]\nKind=bond\nName=%s\n", b.name)
 }
 
+func (b *bondInterface) Type() string {
+	return "bond"
+}
+
+func (b *bondInterface) ModprobeParams() string {
+	params := ""
+	for name, val := range b.options {
+		params += fmt.Sprintf("%s=%s ", name, val)
+	}
+	params = strings.TrimSuffix(params, " ")
+	return params
+}
+
 type vlanInterface struct {
 	logicalInterface
 	id        int
@@ -123,6 +148,10 @@ func (v *vlanInterface) Netdev() string {
 	return config
 }
 
+func (v *vlanInterface) Type() string {
+	return "vlan"
+}
+
 func buildInterfaces(stanzas []*stanzaInterface) []InterfaceGenerator {
 	interfaceMap := createInterfaces(stanzas)
 	linkAncestors(interfaceMap)
@@ -141,15 +170,22 @@ func createInterfaces(stanzas []*stanzaInterface) map[string]networkInterface {
 	for _, iface := range stanzas {
 		switch iface.kind {
 		case interfaceBond:
+			bondOptions := make(map[string]string)
+			for _, k := range []string{"mode", "miimon", "lacp-rate"} {
+				if v, ok := iface.options["bond-"+k]; ok && len(v) > 0 {
+					bondOptions[k] = v[0]
+				}
+			}
 			interfaceMap[iface.name] = &bondInterface{
 				logicalInterface{
 					name:     iface.name,
 					config:   iface.configMethod,
 					children: []networkInterface{},
 				},
-				iface.options["slaves"],
+				iface.options["bond-slaves"],
+				bondOptions,
 			}
-			for _, slave := range iface.options["slaves"] {
+			for _, slave := range iface.options["bond-slaves"] {
 				if _, ok := interfaceMap[slave]; !ok {
 					interfaceMap[slave] = &physicalInterface{
 						logicalInterface{
@@ -241,13 +277,17 @@ func markConfigDepths(interfaceMap map[string]networkInterface) {
 		}
 	}
 	for _, iface := range rootInterfaceMap {
-		setDepth(iface, 0)
+		setDepth(iface)
 	}
 }
 
-func setDepth(iface networkInterface, depth int) {
+func setDepth(iface networkInterface) int {
-	iface.setConfigDepth(depth)
+	maxDepth := 0
 	for _, child := range iface.Children() {
-		setDepth(child, depth+1)
+		if depth := setDepth(child); depth > maxDepth {
+			maxDepth = depth
+		}
 	}
+	iface.setConfigDepth(maxDepth)
+	return (maxDepth + 1)
 }

network/interface_test.go

@@ -36,6 +36,7 @@ func TestPhysicalInterfaceNetwork(t *testing.T) {
 					name: "testbond1",
 				},
 				nil,
+				nil,
 			},
 			&vlanInterface{
 				logicalInterface{
@@ -67,14 +68,14 @@ VLAN=testvlan2
 }
 
 func TestBondInterfaceName(t *testing.T) {
-	b := bondInterface{logicalInterface{name: "testname"}, nil}
+	b := bondInterface{logicalInterface{name: "testname"}, nil, nil}
 	if b.Name() != "testname" {
 		t.FailNow()
 	}
 }
 
 func TestBondInterfaceNetdev(t *testing.T) {
-	b := bondInterface{logicalInterface{name: "testname"}, nil}
+	b := bondInterface{logicalInterface{name: "testname"}, nil, nil}
 	netdev := `[NetDev]
 Kind=bond
 Name=testname
@@ -102,6 +103,7 @@ func TestBondInterfaceNetwork(t *testing.T) {
 						name: "testbond1",
 					},
 					nil,
+					nil,
 				},
 				&vlanInterface{
 					logicalInterface{
@@ -120,6 +122,7 @@ func TestBondInterfaceNetwork(t *testing.T) {
 			},
 		},
 		nil,
+		nil,
 	}
 	network := `[Match]
 Name=testname
@@ -218,6 +221,61 @@ Gateway=1.2.3.4
 	}
 }
 
+func TestType(t *testing.T) {
+	for _, tt := range []struct {
+		i InterfaceGenerator
+		t string
+	}{
+		{
+			i: &physicalInterface{},
+			t: "physical",
+		},
+		{
+			i: &vlanInterface{},
+			t: "vlan",
+		},
+		{
+			i: &bondInterface{},
+			t: "bond",
+		},
+	} {
+		if tp := tt.i.Type(); tp != tt.t {
+			t.Fatalf("bad type (%q): got %s, want %s", tt.i, tp, tt.t)
+		}
+	}
+}
+
+func TestModprobeParams(t *testing.T) {
+	for _, tt := range []struct {
+		i InterfaceGenerator
+		p string
+	}{
+		{
+			i: &physicalInterface{},
+			p: "",
+		},
+		{
+			i: &vlanInterface{},
+			p: "",
+		},
+		{
+			i: &bondInterface{
+				logicalInterface{},
+				nil,
+				map[string]string{
+					"a": "1",
+					"b": "2",
+				},
+			},
+			p: "a=1 b=2",
+		},
+	} {
+		if p := tt.i.ModprobeParams(); p != tt.p {
+			t.Fatalf("bad params (%q): got %s, want %s", tt.i, p, tt.p)
+		}
+	}
+}
+
 func TestBuildInterfacesLo(t *testing.T) {
 	stanzas := []*stanzaInterface{
 		&stanzaInterface{
@@ -242,7 +300,7 @@ func TestBuildInterfacesBlindBond(t *testing.T) {
 			auto:         false,
 			configMethod: configMethodManual{},
 			options: map[string][]string{
-				"slaves": []string{"eth0"},
+				"bond-slaves": []string{"eth0"},
 			},
 		},
 	}
@@ -252,16 +310,17 @@ func TestBuildInterfacesBlindBond(t *testing.T) {
 			name:        "bond0",
 			config:      configMethodManual{},
 			children:    []networkInterface{},
-			configDepth: 1,
+			configDepth: 0,
 		},
 		[]string{"eth0"},
+		map[string]string{},
 	}
 	eth0 := &physicalInterface{
 		logicalInterface{
 			name:        "eth0",
 			config:      configMethodManual{},
 			children:    []networkInterface{bond0},
-			configDepth: 0,
+			configDepth: 1,
 		},
 	}
 	expect := []InterfaceGenerator{bond0, eth0}
@@ -289,7 +348,7 @@ func TestBuildInterfacesBlindVLAN(t *testing.T) {
 			name:        "vlan0",
 			config:      configMethodManual{},
 			children:    []networkInterface{},
-			configDepth: 1,
+			configDepth: 0,
 		},
 		0,
 		"eth0",
@@ -299,7 +358,7 @@ func TestBuildInterfacesBlindVLAN(t *testing.T) {
 			name:        "eth0",
 			config:      configMethodManual{},
 			children:    []networkInterface{vlan0},
-			configDepth: 0,
+			configDepth: 1,
 		},
 	}
 	expect := []InterfaceGenerator{eth0, vlan0}
@@ -323,7 +382,9 @@ func TestBuildInterfaces(t *testing.T) {
 			auto:         false,
 			configMethod: configMethodManual{},
 			options: map[string][]string{
-				"slaves": []string{"eth0"},
+				"bond-slaves": []string{"eth0"},
+				"bond-mode":   []string{"4"},
+				"bond-miimon": []string{"100"},
 			},
 		},
 		&stanzaInterface{
@@ -332,7 +393,7 @@ func TestBuildInterfaces(t *testing.T) {
 			auto:         false,
 			configMethod: configMethodManual{},
 			options: map[string][]string{
-				"slaves": []string{"bond0"},
+				"bond-slaves": []string{"bond0"},
 			},
 		},
 		&stanzaInterface{
@@ -362,7 +423,7 @@ func TestBuildInterfaces(t *testing.T) {
 			name:        "vlan1",
 			config:      configMethodManual{},
 			children:    []networkInterface{},
-			configDepth: 2,
+			configDepth: 0,
 		},
 		1,
 		"bond0",
@@ -372,7 +433,7 @@ func TestBuildInterfaces(t *testing.T) {
 			name:        "vlan0",
 			config:      configMethodManual{},
 			children:    []networkInterface{},
-			configDepth: 1,
+			configDepth: 0,
 		},
 		0,
 		"eth0",
@@ -382,9 +443,10 @@ func TestBuildInterfaces(t *testing.T) {
 			name:        "bond1",
 			config:      configMethodManual{},
 			children:    []networkInterface{},
-			configDepth: 2,
+			configDepth: 0,
 		},
 		[]string{"bond0"},
+		map[string]string{},
 	}
 	bond0 := &bondInterface{
 		logicalInterface{
@@ -394,13 +456,17 @@ func TestBuildInterfaces(t *testing.T) {
 			configDepth: 1,
 		},
 		[]string{"eth0"},
+		map[string]string{
+			"mode":   "4",
+			"miimon": "100",
+		},
 	}
 	eth0 := &physicalInterface{
 		logicalInterface{
 			name:        "eth0",
 			config:      configMethodManual{},
 			children:    []networkInterface{bond0, vlan0},
-			configDepth: 0,
+			configDepth: 2,
 		},
 	}
 	expect := []InterfaceGenerator{eth0, bond0, bond1, vlan0, vlan1}

network/stanza.go

@@ -293,7 +293,6 @@ func parseHwaddress(options map[string][]string, iface string) (net.HardwareAddr
 }
 
 func parseBondStanza(iface string, conf configMethod, attributes []string, options map[string][]string) (*stanzaInterface, error) {
-	options["slaves"] = options["bond-slaves"]
 	return &stanzaInterface{name: iface, kind: interfaceBond, configMethod: conf, options: options}, nil
 }
 

network/stanza_test.go

@@ -129,7 +129,7 @@ func TestParseBondStanzaNoSlaves(t *testing.T) {
 	if err != nil {
 		t.FailNow()
 	}
-	if bond.options["slaves"] != nil {
+	if bond.options["bond-slaves"] != nil {
 		t.FailNow()
 	}
 }
@@ -152,9 +152,6 @@ func TestParseBondStanza(t *testing.T) {
 	if bond.configMethod != conf {
 		t.FailNow()
 	}
-	if !reflect.DeepEqual(bond.options["slaves"], options["bond-slaves"]) {
-		t.FailNow()
-	}
 }
 
 func TestParsePhysicalStanza(t *testing.T) {

pkg/http_client.go

@@ -57,6 +57,11 @@ type HttpClient struct {
 	client *http.Client
 }
 
+type Getter interface {
+	Get(string) ([]byte, error)
+	GetRetry(string) ([]byte, error)
+}
+
 func NewHttpClient() *HttpClient {
 	hc := &HttpClient{
 		MaxBackoff: time.Second * 5,

system/env_file.go

@@ -0,0 +1,100 @@
+package system
+
+import (
+	"bytes"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path"
+	"regexp"
+	"sort"
+)
+
+type EnvFile struct {
+	Vars map[string]string
+	// mask File.Content, it shouldn't be used.
+	Content interface{} `json:"-" yaml:"-"`
+	*File
+}
+
+// only allow sh compatible identifiers
+var validKey = regexp.MustCompile(`^[a-zA-Z0-9_]+$`)
+
+// match each line, optionally capturing valid identifiers, discarding dos line endings
+var lineLexer = regexp.MustCompile(`(?m)^((?:([a-zA-Z0-9_]+)=)?.*?)\r?\n`)
+
+// mergeEnvContents: Update the existing file contents with new values,
+// preserving variable ordering and all content this code doesn't understand.
+// All new values are appended to the bottom of the old, sorted by key.
+func mergeEnvContents(old []byte, pending map[string]string) []byte {
+	var buf bytes.Buffer
+	var match [][]byte
+
+	// it is awkward for the regex to handle a missing newline gracefully
+	if len(old) != 0 && !bytes.HasSuffix(old, []byte{'\n'}) {
+		old = append(old, byte('\n'))
+	}
+
+	for _, match = range lineLexer.FindAllSubmatch(old, -1) {
+		key := string(match[2])
+		if value, ok := pending[key]; ok {
+			fmt.Fprintf(&buf, "%s=%s\n", key, value)
+			delete(pending, key)
+		} else {
+			fmt.Fprintf(&buf, "%s\n", match[1])
+		}
+	}
+
+	for _, key := range keys(pending) {
+		value := pending[key]
+		fmt.Fprintf(&buf, "%s=%s\n", key, value)
+	}
+
+	return buf.Bytes()
+}
+
+// WriteEnvFile updates an existing env `KEY=value` formated file with
+// new values provided in EnvFile.Vars; File.Content is ignored.
+// Existing ordering and any unknown formatting such as comments are
+// preserved. If no changes are required the file is untouched.
+func WriteEnvFile(ef *EnvFile, root string) error {
+	// validate new keys, mergeEnvContents uses pending to track writes
+	pending := make(map[string]string, len(ef.Vars))
+	for key, value := range ef.Vars {
+		if !validKey.MatchString(key) {
+			return fmt.Errorf("Invalid name %q for %s", key, ef.Path)
+		}
+		pending[key] = value
+	}
+
+	if len(pending) == 0 {
+		return nil
+	}
+
+	oldContent, err := ioutil.ReadFile(path.Join(root, ef.Path))
+	if err != nil {
+		if os.IsNotExist(err) {
+			oldContent = []byte{}
+		} else {
+			return err
+		}
+	}
+
+	newContent := mergeEnvContents(oldContent, pending)
+	if bytes.Equal(oldContent, newContent) {
+		return nil
+	}
+
+	ef.File.Content = string(newContent)
+	_, err = WriteFile(ef.File, root)
+	return err
+}
+
+// keys returns the keys of a map in sorted order
+func keys(m map[string]string) (s []string) {
+	for k, _ := range m {
+		s = append(s, k)
+	}
+	sort.Strings(s)
+	return
+}

system/env_file_test.go

@@ -0,0 +1,426 @@
+package system
+
+import (
+	"io/ioutil"
+	"os"
+	"path"
+	"strings"
+	"syscall"
+	"testing"
+)
+
+const (
+	base          = "# a file\nFOO=base\n\nBAR= hi there\n"
+	baseNoNewline = "# a file\nFOO=base\n\nBAR= hi there"
+	baseDos       = "# a file\r\nFOO=base\r\n\r\nBAR= hi there\r\n"
+	expectUpdate  = "# a file\nFOO=test\n\nBAR= hi there\nNEW=a value\n"
+	expectCreate  = "FOO=test\nNEW=a value\n"
+)
+
+var (
+	valueUpdate = map[string]string{
+		"FOO": "test",
+		"NEW": "a value",
+	}
+	valueNoop = map[string]string{
+		"FOO": "base",
+	}
+	valueEmpty   = map[string]string{}
+	valueInvalid = map[string]string{
+		"FOO-X": "test",
+	}
+)
+
+func TestWriteEnvFileUpdate(t *testing.T) {
+	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
+	if err != nil {
+		t.Fatalf("Unable to create tempdir: %v", err)
+	}
+	defer os.RemoveAll(dir)
+
+	name := "foo.conf"
+	fullPath := path.Join(dir, name)
+	ioutil.WriteFile(fullPath, []byte(base), 0644)
+
+	oldStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	ef := EnvFile{
+		File: &File{
+			Path: name,
+		},
+		Vars: valueUpdate,
+	}
+
+	err = WriteEnvFile(&ef, dir)
+	if err != nil {
+		t.Fatalf("WriteFile failed: %v", err)
+	}
+
+	contents, err := ioutil.ReadFile(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to read expected file: %v", err)
+	}
+
+	if string(contents) != expectUpdate {
+		t.Fatalf("File has incorrect contents: %q", contents)
+	}
+
+	newStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	if oldStat.Sys().(*syscall.Stat_t).Ino == newStat.Sys().(*syscall.Stat_t).Ino {
+		t.Fatalf("File was not replaced: %s", fullPath)
+	}
+}
+
+func TestWriteEnvFileUpdateNoNewline(t *testing.T) {
+	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
+	if err != nil {
+		t.Fatalf("Unable to create tempdir: %v", err)
+	}
+	defer os.RemoveAll(dir)
+
+	name := "foo.conf"
+	fullPath := path.Join(dir, name)
+	ioutil.WriteFile(fullPath, []byte(baseNoNewline), 0644)
+
+	oldStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	ef := EnvFile{
+		File: &File{
+			Path: name,
+		},
+		Vars: valueUpdate,
+	}
+
+	err = WriteEnvFile(&ef, dir)
+	if err != nil {
+		t.Fatalf("WriteFile failed: %v", err)
+	}
+
+	contents, err := ioutil.ReadFile(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to read expected file: %v", err)
+	}
+
+	if string(contents) != expectUpdate {
+		t.Fatalf("File has incorrect contents: %q", contents)
+	}
+
+	newStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	if oldStat.Sys().(*syscall.Stat_t).Ino == newStat.Sys().(*syscall.Stat_t).Ino {
+		t.Fatalf("File was not replaced: %s", fullPath)
+	}
+}
+
+func TestWriteEnvFileCreate(t *testing.T) {
+	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
+	if err != nil {
+		t.Fatalf("Unable to create tempdir: %v", err)
+	}
+	defer os.RemoveAll(dir)
+
+	name := "foo.conf"
+	fullPath := path.Join(dir, name)
+
+	ef := EnvFile{
+		File: &File{
+			Path: name,
+		},
+		Vars: valueUpdate,
+	}
+
+	err = WriteEnvFile(&ef, dir)
+	if err != nil {
+		t.Fatalf("WriteFile failed: %v", err)
+	}
+
+	contents, err := ioutil.ReadFile(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to read expected file: %v", err)
+	}
+
+	if string(contents) != expectCreate {
+		t.Fatalf("File has incorrect contents: %q", contents)
+	}
+}
+
+func TestWriteEnvFileNoop(t *testing.T) {
+	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
+	if err != nil {
+		t.Fatalf("Unable to create tempdir: %v", err)
+	}
+	defer os.RemoveAll(dir)
+
+	name := "foo.conf"
+	fullPath := path.Join(dir, name)
+	ioutil.WriteFile(fullPath, []byte(base), 0644)
+
+	oldStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	ef := EnvFile{
+		File: &File{
+			Path: name,
+		},
+		Vars: valueNoop,
+	}
+
+	err = WriteEnvFile(&ef, dir)
+	if err != nil {
+		t.Fatalf("WriteFile failed: %v", err)
+	}
+
+	contents, err := ioutil.ReadFile(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to read expected file: %v", err)
+	}
+
+	if string(contents) != base {
+		t.Fatalf("File has incorrect contents: %q", contents)
+	}
+
+	newStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	if oldStat.Sys().(*syscall.Stat_t).Ino != newStat.Sys().(*syscall.Stat_t).Ino {
+		t.Fatalf("File was replaced: %s", fullPath)
+	}
+}
+
+func TestWriteEnvFileUpdateDos(t *testing.T) {
+	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
+	if err != nil {
+		t.Fatalf("Unable to create tempdir: %v", err)
+	}
+	defer os.RemoveAll(dir)
+
+	name := "foo.conf"
+	fullPath := path.Join(dir, name)
+	ioutil.WriteFile(fullPath, []byte(baseDos), 0644)
+
+	oldStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	ef := EnvFile{
+		File: &File{
+			Path: name,
+		},
+		Vars: valueUpdate,
+	}
+
+	err = WriteEnvFile(&ef, dir)
+	if err != nil {
+		t.Fatalf("WriteFile failed: %v", err)
+	}
+
+	contents, err := ioutil.ReadFile(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to read expected file: %v", err)
+	}
+
+	if string(contents) != expectUpdate {
+		t.Fatalf("File has incorrect contents: %q", contents)
+	}
+
+	newStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	if oldStat.Sys().(*syscall.Stat_t).Ino == newStat.Sys().(*syscall.Stat_t).Ino {
+		t.Fatalf("File was not replaced: %s", fullPath)
+	}
+}
+
+// A middle ground noop, values are unchanged but we did have a value.
+// Seems reasonable to rewrite the file in Unix format anyway.
+func TestWriteEnvFileDos2Unix(t *testing.T) {
+	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
+	if err != nil {
+		t.Fatalf("Unable to create tempdir: %v", err)
+	}
+	defer os.RemoveAll(dir)
+
+	name := "foo.conf"
+	fullPath := path.Join(dir, name)
+	ioutil.WriteFile(fullPath, []byte(baseDos), 0644)
+
+	oldStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	ef := EnvFile{
+		File: &File{
+			Path: name,
+		},
+		Vars: valueNoop,
+	}
+
+	err = WriteEnvFile(&ef, dir)
+	if err != nil {
+		t.Fatalf("WriteFile failed: %v", err)
+	}
+
+	contents, err := ioutil.ReadFile(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to read expected file: %v", err)
+	}
+
+	if string(contents) != base {
+		t.Fatalf("File has incorrect contents: %q", contents)
+	}
+
+	newStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	if oldStat.Sys().(*syscall.Stat_t).Ino == newStat.Sys().(*syscall.Stat_t).Ino {
+		t.Fatalf("File was not replaced: %s", fullPath)
+	}
+}
+
+// If it really is a noop (structure is empty) don't even do dos2unix
+func TestWriteEnvFileEmpty(t *testing.T) {
+	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
+	if err != nil {
+		t.Fatalf("Unable to create tempdir: %v", err)
+	}
+	defer os.RemoveAll(dir)
+
+	name := "foo.conf"
+	fullPath := path.Join(dir, name)
+	ioutil.WriteFile(fullPath, []byte(baseDos), 0644)
+
+	oldStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	ef := EnvFile{
+		File: &File{
+			Path: name,
+		},
+		Vars: valueEmpty,
+	}
+
+	err = WriteEnvFile(&ef, dir)
+	if err != nil {
+		t.Fatalf("WriteFile failed: %v", err)
+	}
+
+	contents, err := ioutil.ReadFile(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to read expected file: %v", err)
+	}
+
+	if string(contents) != baseDos {
+		t.Fatalf("File has incorrect contents: %q", contents)
+	}
+
+	newStat, err := os.Stat(fullPath)
+	if err != nil {
+		t.Fatalf("Unable to stat file: %v", err)
+	}
+
+	if oldStat.Sys().(*syscall.Stat_t).Ino != newStat.Sys().(*syscall.Stat_t).Ino {
+		t.Fatalf("File was replaced: %s", fullPath)
+	}
+}
+
+// no point in creating empty files
+func TestWriteEnvFileEmptyNoCreate(t *testing.T) {
+	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
+	if err != nil {
+		t.Fatalf("Unable to create tempdir: %v", err)
+	}
+	defer os.RemoveAll(dir)
+
+	name := "foo.conf"
+	fullPath := path.Join(dir, name)
+
+	ef := EnvFile{
+		File: &File{
+			Path: name,
+		},
+		Vars: valueEmpty,
+	}
+
+	err = WriteEnvFile(&ef, dir)
+	if err != nil {
+		t.Fatalf("WriteFile failed: %v", err)
+	}
+
+	contents, err := ioutil.ReadFile(fullPath)
+	if err == nil {
+		t.Fatalf("File has incorrect contents: %q", contents)
+	} else if !os.IsNotExist(err) {
+		t.Fatalf("Unexpected error while reading file: %v", err)
+	}
+}
+
+func TestWriteEnvFilePermFailure(t *testing.T) {
+	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
+	if err != nil {
+		t.Fatalf("Unable to create tempdir: %v", err)
+	}
+	defer os.RemoveAll(dir)
+
+	name := "foo.conf"
+	fullPath := path.Join(dir, name)
+	ioutil.WriteFile(fullPath, []byte(base), 0000)
+
+	ef := EnvFile{
+		File: &File{
+			Path: name,
+		},
+		Vars: valueUpdate,
+	}
+
+	err = WriteEnvFile(&ef, dir)
+	if !os.IsPermission(err) {
+		t.Fatalf("Not a pemission denied error: %v", err)
+	}
+}
+
+func TestWriteEnvFileNameFailure(t *testing.T) {
+	dir, err := ioutil.TempDir(os.TempDir(), "coreos-cloudinit-")
+	if err != nil {
+		t.Fatalf("Unable to create tempdir: %v", err)
+	}
+	defer os.RemoveAll(dir)
+
+	name := "foo.conf"
+
+	ef := EnvFile{
+		File: &File{
+			Path: name,
+		},
+		Vars: valueInvalid,
+	}
+
+	err = WriteEnvFile(&ef, dir)
+	if err == nil || !strings.HasPrefix(err.Error(), "Invalid name") {
+		t.Fatalf("Not an invalid name error: %v", err)
+	}
+}

system/networkd.go

@@ -2,10 +2,11 @@ package system
 
 import (
 	"fmt"
-	"io/ioutil"
+	"log"
 	"net"
 	"os/exec"
-	"path"
+	"strings"
+	"time"
 
 	"github.com/coreos/coreos-cloudinit/network"
 	"github.com/coreos/coreos-cloudinit/third_party/github.com/dotcloud/docker/pkg/netlink"
@@ -17,6 +18,13 @@ const (
 
 func RestartNetwork(interfaces []network.InterfaceGenerator) (err error) {
 	defer func() {
+		if e := restartNetworkd(); e != nil {
+			err = e
+			return
+		}
+		// TODO(crawford): Get rid of this once networkd fixes the race
+		// https://bugs.freedesktop.org/show_bug.cgi?id=76077
+		time.Sleep(5 * time.Second)
 		if e := restartNetworkd(); e != nil {
 			err = e
 		}
@@ -26,19 +34,18 @@ func RestartNetwork(interfaces []network.InterfaceGenerator) (err error) {
 		return
 	}
 
-	if err = probe8012q(); err != nil {
+	if err = maybeProbe8012q(interfaces); err != nil {
 		return
 	}
-	return
+	return maybeProbeBonding(interfaces)
 }
 
 func downNetworkInterfaces(interfaces []network.InterfaceGenerator) error {
 	sysInterfaceMap := make(map[string]*net.Interface)
 	if systemInterfaces, err := net.Interfaces(); err == nil {
 		for _, iface := range systemInterfaces {
-			// Need a copy of the interface so we can take the address
+			iface := iface
-			temp := iface
+			sysInterfaceMap[iface.Name] = &iface
-			sysInterfaceMap[temp.Name] = &temp
 		}
 	} else {
 		return err
@@ -46,6 +53,7 @@ func downNetworkInterfaces(interfaces []network.InterfaceGenerator) error {
 
 	for _, iface := range interfaces {
 		if systemInterface, ok := sysInterfaceMap[iface.Name()]; ok {
+			log.Printf("Taking down interface %q\n", systemInterface.Name)
 			if err := netlink.NetworkLinkDown(systemInterface); err != nil {
 				fmt.Printf("Error while downing interface %q (%s). Continuing...\n", systemInterface.Name, err)
 			}
@@ -55,26 +63,45 @@ func downNetworkInterfaces(interfaces []network.InterfaceGenerator) error {
 	return nil
 }
 
-func probe8012q() error {
+func maybeProbe8012q(interfaces []network.InterfaceGenerator) error {
-	return exec.Command("modprobe", "8021q").Run()
+	for _, iface := range interfaces {
+		if iface.Type() == "vlan" {
+			log.Printf("Probing LKM %q (%q)\n", "8021q", "8021q")
+			return exec.Command("modprobe", "8021q").Run()
+		}
+	}
+	return nil
+}
+
+func maybeProbeBonding(interfaces []network.InterfaceGenerator) error {
+	args := []string{"bonding"}
+	for _, iface := range interfaces {
+		if iface.Type() == "bond" {
+			args = append(args, strings.Split(iface.ModprobeParams(), " ")...)
+			break
+		}
+	}
+	log.Printf("Probing LKM %q (%q)\n", "bonding", args)
+	return exec.Command("modprobe", args...).Run()
 }
 
 func restartNetworkd() error {
-	_, err := RunUnitCommand("restart", "systemd-networkd.service")
+	log.Printf("Restarting networkd.service\n")
+	_, err := NewUnitManager("").RunUnitCommand("restart", "systemd-networkd.service")
 	return err
 }
 
 func WriteNetworkdConfigs(interfaces []network.InterfaceGenerator) error {
 	for _, iface := range interfaces {
-		filename := path.Join(runtimeNetworkPath, fmt.Sprintf("%s.netdev", iface.Filename()))
+		filename := fmt.Sprintf("%s.netdev", iface.Filename())
 		if err := writeConfig(filename, iface.Netdev()); err != nil {
 			return err
 		}
-		filename = path.Join(runtimeNetworkPath, fmt.Sprintf("%s.link", iface.Filename()))
+		filename = fmt.Sprintf("%s.link", iface.Filename())
 		if err := writeConfig(filename, iface.Link()); err != nil {
 			return err
 		}
-		filename = path.Join(runtimeNetworkPath, fmt.Sprintf("%s.network", iface.Filename()))
+		filename = fmt.Sprintf("%s.network", iface.Filename())
 		if err := writeConfig(filename, iface.Network()); err != nil {
 			return err
 		}
@@ -86,6 +113,7 @@ func writeConfig(filename string, config string) error {
 	if config == "" {
 		return nil
 	}
-
+	log.Printf("Writing networkd unit %q\n", filename)
-	return ioutil.WriteFile(filename, []byte(config), 0444)
+	_, err := WriteFile(&File{Content: config, Path: filename}, runtimeNetworkPath)
+	return err
 }

system/systemd.go

@@ -13,63 +13,21 @@ import (
 	"github.com/coreos/coreos-cloudinit/third_party/github.com/coreos/go-systemd/dbus"
 )
 
+func NewUnitManager(root string) UnitManager {
+	return &systemd{root}
+}
+
+type systemd struct {
+	root string
+}
+
 // fakeMachineID is placed on non-usr CoreOS images and should
 // never be used as a true MachineID
 const fakeMachineID = "42000000000000000000000000000042"
 
-// Name for drop-in service configuration files created by cloudconfig
-const cloudConfigDropIn = "20-cloudinit.conf"
-
-type Unit struct {
-	Name    string
-	Mask    bool
-	Enable  bool
-	Runtime bool
-	Content string
-	Command string
-
-	// For drop-in units, a cloudinit.conf is generated.
-	// This is currently unbound in YAML (and hence unsettable in cloud-config files)
-	// until the correct behaviour for multiple drop-in units is determined.
-	DropIn bool `yaml:"-"`
-}
-
-func (u *Unit) Type() string {
-	ext := filepath.Ext(u.Name)
-	return strings.TrimLeft(ext, ".")
-}
-
-func (u *Unit) Group() (group string) {
-	t := u.Type()
-	if t == "network" || t == "netdev" || t == "link" {
-		group = "network"
-	} else {
-		group = "system"
-	}
-	return
-}
-
-type Script []byte
-
-// Destination builds the appropriate absolute file path for
-// the Unit. The root argument indicates the effective base
-// directory of the system (similar to a chroot).
-func (u *Unit) Destination(root string) string {
-	dir := "etc"
-	if u.Runtime {
-		dir = "run"
-	}
-
-	if u.DropIn {
-		return path.Join(root, dir, "systemd", u.Group(), fmt.Sprintf("%s.d", u.Name), cloudConfigDropIn)
-	} else {
-		return path.Join(root, dir, "systemd", u.Group(), u.Name)
-	}
-}
-
 // PlaceUnit writes a unit file at the provided destination, creating
 // parent directories as necessary.
-func PlaceUnit(u *Unit, dst string) error {
+func (s *systemd) PlaceUnit(u *Unit, dst string) error {
 	dir := filepath.Dir(dst)
 	if _, err := os.Stat(dir); os.IsNotExist(err) {
 		if err := os.MkdirAll(dir, os.FileMode(0755)); err != nil {
@@ -91,7 +49,7 @@ func PlaceUnit(u *Unit, dst string) error {
 	return nil
 }
 
-func EnableUnitFile(unit string, runtime bool) error {
+func (s *systemd) EnableUnitFile(unit string, runtime bool) error {
 	conn, err := dbus.New()
 	if err != nil {
 		return err
@@ -102,7 +60,7 @@ func EnableUnitFile(unit string, runtime bool) error {
 	return err
 }
 
-func RunUnitCommand(command, unit string) (string, error) {
+func (s *systemd) RunUnitCommand(command, unit string) (string, error) {
 	conn, err := dbus.New()
 	if err != nil {
 		return "", err
@@ -131,7 +89,7 @@ func RunUnitCommand(command, unit string) (string, error) {
 	return fn(unit, "replace")
 }
 
-func DaemonReload() error {
+func (s *systemd) DaemonReload() error {
 	conn, err := dbus.New()
 	if err != nil {
 		return err
@@ -140,6 +98,57 @@ func DaemonReload() error {
 	return conn.Reload()
 }
 
+// MaskUnit masks the given Unit by symlinking its unit file to
+// /dev/null, analogous to `systemctl mask`.
+// N.B.: Unlike `systemctl mask`, this function will *remove any existing unit
+// file at the location*, to ensure that the mask will succeed.
+func (s *systemd) MaskUnit(unit *Unit) error {
+	masked := unit.Destination(s.root)
+	if _, err := os.Stat(masked); os.IsNotExist(err) {
+		if err := os.MkdirAll(path.Dir(masked), os.FileMode(0755)); err != nil {
+			return err
+		}
+	} else if err := os.Remove(masked); err != nil {
+		return err
+	}
+	return os.Symlink("/dev/null", masked)
+}
+
+// UnmaskUnit is analogous to systemd's unit_file_unmask. If the file
+// associated with the given Unit is empty or appears to be a symlink to
+// /dev/null, it is removed.
+func (s *systemd) UnmaskUnit(unit *Unit) error {
+	masked := unit.Destination(s.root)
+	ne, err := nullOrEmpty(masked)
+	if os.IsNotExist(err) {
+		return nil
+	} else if err != nil {
+		return err
+	}
+	if !ne {
+		log.Printf("%s is not null or empty, refusing to unmask", masked)
+		return nil
+	}
+	return os.Remove(masked)
+}
+
+// nullOrEmpty checks whether a given path appears to be an empty regular file
+// or a symlink to /dev/null
+func nullOrEmpty(path string) (bool, error) {
+	fi, err := os.Stat(path)
+	if err != nil {
+		return false, err
+	}
+	m := fi.Mode()
+	if m.IsRegular() && fi.Size() <= 0 {
+		return true, nil
+	}
+	if m&os.ModeCharDevice > 0 {
+		return true, nil
+	}
+	return false, nil
+}
+
 func ExecuteScript(scriptPath string) (string, error) {
 	props := []dbus.Property{
 		dbus.PropDescription("Unit generated and executed by coreos-cloudinit on behalf of user"),
@@ -178,54 +187,3 @@ func MachineID(root string) string {
 
 	return id
 }
-
-// MaskUnit masks the given Unit by symlinking its unit file to
-// /dev/null, analogous to `systemctl mask`.
-// N.B.: Unlike `systemctl mask`, this function will *remove any existing unit
-// file at the location*, to ensure that the mask will succeed.
-func MaskUnit(unit *Unit, root string) error {
-	masked := unit.Destination(root)
-	if _, err := os.Stat(masked); os.IsNotExist(err) {
-		if err := os.MkdirAll(path.Dir(masked), os.FileMode(0755)); err != nil {
-			return err
-		}
-	} else if err := os.Remove(masked); err != nil {
-		return err
-	}
-	return os.Symlink("/dev/null", masked)
-}
-
-// UnmaskUnit is analogous to systemd's unit_file_unmask. If the file
-// associated with the given Unit is empty or appears to be a symlink to
-// /dev/null, it is removed.
-func UnmaskUnit(unit *Unit, root string) error {
-	masked := unit.Destination(root)
-	ne, err := nullOrEmpty(masked)
-	if os.IsNotExist(err) {
-		return nil
-	} else if err != nil {
-		return err
-	}
-	if !ne {
-		log.Printf("%s is not null or empty, refusing to unmask", masked)
-		return nil
-	}
-	return os.Remove(masked)
-}
-
-// nullOrEmpty checks whether a given path appears to be an empty regular file
-// or a symlink to /dev/null
-func nullOrEmpty(path string) (bool, error) {
-	fi, err := os.Stat(path)
-	if err != nil {
-		return false, err
-	}
-	m := fi.Mode()
-	if m.IsRegular() && fi.Size() <= 0 {
-		return true, nil
-	}
-	if m&os.ModeCharDevice > 0 {
-		return true, nil
-	}
-	return false, nil
-}

system/systemd_test.go

@@ -25,13 +25,15 @@ Address=10.209.171.177/19
 	}
 	defer os.RemoveAll(dir)
 
+	sd := &systemd{dir}
+
 	dst := u.Destination(dir)
 	expectDst := path.Join(dir, "run", "systemd", "network", "50-eth0.network")
 	if dst != expectDst {
 		t.Fatalf("unit.Destination returned %s, expected %s", dst, expectDst)
 	}
 
-	if err := PlaceUnit(&u, dst); err != nil {
+	if err := sd.PlaceUnit(&u, dst); err != nil {
 		t.Fatalf("PlaceUnit failed: %v", err)
 	}
 
@@ -100,13 +102,15 @@ Where=/media/state
 	}
 	defer os.RemoveAll(dir)
 
+	sd := &systemd{dir}
+
 	dst := u.Destination(dir)
 	expectDst := path.Join(dir, "etc", "systemd", "system", "media-state.mount")
 	if dst != expectDst {
 		t.Fatalf("unit.Destination returned %s, expected %s", dst, expectDst)
 	}
 
-	if err := PlaceUnit(&u, dst); err != nil {
+	if err := sd.PlaceUnit(&u, dst); err != nil {
 		t.Fatalf("PlaceUnit failed: %v", err)
 	}
 
@@ -155,9 +159,11 @@ func TestMaskUnit(t *testing.T) {
 	}
 	defer os.RemoveAll(dir)
 
+	sd := &systemd{dir}
+
 	// Ensure mask works with units that do not currently exist
 	uf := &Unit{Name: "foo.service"}
-	if err := MaskUnit(uf, dir); err != nil {
+	if err := sd.MaskUnit(uf); err != nil {
 		t.Fatalf("Unable to mask new unit: %v", err)
 	}
 	fooPath := path.Join(dir, "etc", "systemd", "system", "foo.service")
@@ -175,7 +181,7 @@ func TestMaskUnit(t *testing.T) {
 	if _, err := os.Create(barPath); err != nil {
 		t.Fatalf("Error creating new unit file: %v", err)
 	}
-	if err := MaskUnit(ub, dir); err != nil {
+	if err := sd.MaskUnit(ub); err != nil {
 		t.Fatalf("Unable to mask existing unit: %v", err)
 	}
 	barTgt, err := os.Readlink(barPath)
@@ -194,8 +200,10 @@ func TestUnmaskUnit(t *testing.T) {
 	}
 	defer os.RemoveAll(dir)
 
+	sd := &systemd{dir}
+
 	nilUnit := &Unit{Name: "null.service"}
-	if err := UnmaskUnit(nilUnit, dir); err != nil {
+	if err := sd.UnmaskUnit(nilUnit); err != nil {
 		t.Errorf("unexpected error from unmasking nonexistent unit: %v", err)
 	}
 
@@ -211,7 +219,7 @@ func TestUnmaskUnit(t *testing.T) {
 	if err := ioutil.WriteFile(dst, []byte(uf.Content), 700); err != nil {
 		t.Fatalf("Unable to write unit file: %v", err)
 	}
-	if err := UnmaskUnit(uf, dir); err != nil {
+	if err := sd.UnmaskUnit(uf); err != nil {
 		t.Errorf("unmask of non-empty unit returned unexpected error: %v", err)
 	}
 	got, _ := ioutil.ReadFile(dst)
@@ -224,7 +232,7 @@ func TestUnmaskUnit(t *testing.T) {
 	if err := os.Symlink("/dev/null", dst); err != nil {
 		t.Fatalf("Unable to create masked unit: %v", err)
 	}
-	if err := UnmaskUnit(ub, dir); err != nil {
+	if err := sd.UnmaskUnit(ub); err != nil {
 		t.Errorf("unmask of unit returned unexpected error: %v", err)
 	}
 	if _, err := os.Stat(dst); !os.IsNotExist(err) {

system/unit.go

@@ -0,0 +1,67 @@
+package system
+
+import (
+	"fmt"
+	"path"
+	"path/filepath"
+	"strings"
+)
+
+// Name for drop-in service configuration files created by cloudconfig
+const cloudConfigDropIn = "20-cloudinit.conf"
+
+type UnitManager interface {
+	PlaceUnit(unit *Unit, dst string) error
+	EnableUnitFile(unit string, runtime bool) error
+	RunUnitCommand(command, unit string) (string, error)
+	DaemonReload() error
+	MaskUnit(unit *Unit) error
+	UnmaskUnit(unit *Unit) error
+}
+
+type Unit struct {
+	Name    string
+	Mask    bool
+	Enable  bool
+	Runtime bool
+	Content string
+	Command string
+
+	// For drop-in units, a cloudinit.conf is generated.
+	// This is currently unbound in YAML (and hence unsettable in cloud-config files)
+	// until the correct behaviour for multiple drop-in units is determined.
+	DropIn bool `yaml:"-"`
+}
+
+func (u *Unit) Type() string {
+	ext := filepath.Ext(u.Name)
+	return strings.TrimLeft(ext, ".")
+}
+
+func (u *Unit) Group() (group string) {
+	t := u.Type()
+	if t == "network" || t == "netdev" || t == "link" {
+		group = "network"
+	} else {
+		group = "system"
+	}
+	return
+}
+
+type Script []byte
+
+// Destination builds the appropriate absolute file path for
+// the Unit. The root argument indicates the effective base
+// directory of the system (similar to a chroot).
+func (u *Unit) Destination(root string) string {
+	dir := "etc"
+	if u.Runtime {
+		dir = "run"
+	}
+
+	if u.DropIn {
+		return path.Join(root, dir, "systemd", u.Group(), fmt.Sprintf("%s.d", u.Name), cloudConfigDropIn)
+	} else {
+		return path.Join(root, dir, "systemd", u.Group(), u.Name)
+	}
+}

test

@@ -13,12 +13,21 @@ COVER=${COVER:-"-cover"}
 
 source ./build
 
-declare -a TESTPKGS=(initialize system datasource pkg network)
+declare -a TESTPKGS=(initialize
+                     system
+                     datasource
+                     datasource/configdrive
+                     datasource/file
+                     datasource/metadata/ec2
+                     datasource/proc_cmdline
+                     datasource/url
+                     pkg
+                     network)
 
 if [ -z "$PKG" ]; then
 	GOFMTPATH="$TESTPKGS coreos-cloudinit.go"
 	# prepend repo path to each package
-	TESTPKGS=${TESTPKGS[@]/#/${REPO_PATH}/}
+	TESTPKGS="${TESTPKGS[@]/#/${REPO_PATH}/} ./"
 else
 	GOFMTPATH="$TESTPKGS"
 	# strip out slashes and dots from PKG=./foo/