Merge pull request #133 from jackgr/move-templates-support

Changes to support template moves

README.md

@@ -170,8 +170,13 @@ There is a more detailed [design document](docs/design/design.md) available.
 
 This project is still under active development, so you might run into issues. If
 you do, please don't be shy about letting us know, or better yet, contribute a
-fix or feature. We use the same [development process](CONTRIBUTING.md) as the main
-Kubernetes repository.
+fix or feature.
+
+## Contributing
+Your contributions are welcome.
+
+We use the same [workflow](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md#git-setup),
+[License](LICENSE) and [Contributor License Agreement](CONTRIBUTING.md) as the main Kubernetes repository.
 
 ## Relationship to Google Cloud Platform
 DM uses the same concepts and languages as

docs/templates/registry.md

@@ -11,6 +11,9 @@ without some organizing principles. This document defines conventions for
 template registries that store templates in Github and organize them by name
 and by version to make sharing easier.
 
+For a working example of a template registry, please see the
+[Kubernetes Template Registry](https://github.com/kubernetes/application-dm-templates).
+
 ## Template Versions
 
 Since templates referenced by configurations and by other templates may change
@@ -55,6 +58,8 @@ with any supporting files it might require, such as an optional schema file
 named `<template-name>.py.schema` or `<template-name>.jinja.schema`, respectively, 
 helper files used by the implementation, files imported by the schema, and so on.
 
+### Basic structure
+
 These constraints impose a basic level of organization on the template definition
 by ensuring that the template and all of its supporting files at least live in the
 same directory, and that the template and schema files follow well-defined naming
@@ -67,17 +72,17 @@ might be some benefits in allowing templates to share a directory, such as avoid
 the duplication of helper files, the cost of discovering and maintaining templates
 would quickly outweigh them as the number of templates in the directory increased.
 
-Every template version must therefore live in its own directory, and that
-directory must contain one and only one top-level template file and supporting
-files for one and only template version.
-
-Since it may reduce management overhead to store many different templates,
+Also, since it may reduce management overhead to store many different templates,
 and/or many versions of the same template, in a single repository, we need a way
 to organize templates within a repository.
 
-A template repository must therefore place all of the versions of a given
-template in directories named for the template versions under a directory named
-for the template.
+Therefore:
+
+* Every template version must live in its own directory named for the version.
+* The version directory must contain one and only one top-level template file 
+and supporting files for one and only template version.
+* All of the version directories for a given template must live under a single
+directory named for the template without extensions.
 
 For example:
 
@@ -98,8 +103,36 @@ templateA/
     helper.py
 ```
 
-The template directories may be organized in any way that makes sense to the
-repository maintainers.
+In this example, `templateA` is a template directory, and `v1`, `v1.01`, and 
+`v1.1` are template version directories that hold the versions of `templateA`.
+
+### Registry based template references
+
+In general, 
+[templates references](https://github.com/kubernetes/deployment-manager/blob/master/docs/design/design.md#template-references)
+are just URLs to HTTP endpoints. However, because a template registry follows
+the conventions outlined above, references to templates in a template registry
+can be shorter and simpler than generalized template references.
+
+In a registry based template reference, the scheme part of the URL and the name
+of the top level template file are omitted, and the version number is delimited
+by a colon. So for example, instead of
+
+```
+https://raw.githubusercontent.com/ownerA/repository2/master/templateA/v1/templateA.py
+```
+
+you can simply write
+
+```
+github.com/ownerA/repository2/templateA:v1
+```
+
+### Grouping templates
+
+Of course, a flat list of templates won't scale, and it's unlikely that any
+fixed taxonomy would work for all registries. Template directories may therefore
+be grouped in any way that makes sense to the repository maintainers.
 
 For example, this flat list of template directories is valid:
 
@@ -133,13 +166,70 @@ templates/
       ...
 ```
 
-## Template Registries
+### Template collections
 
-Github is a convenient place to store and manage templates. A template registry
-is a Github repository that conforms to the requirements detailed in this document.
+A side effect of allowing arbitrary grouping is that we don't know how to find
+templates when searching or listing the contents of a registry without walking
+the directory tree down to the leaves and then backtracking to identify template
+directories.
 
-For a working example of a template registry, please see the
-[Kubernetes Template Registry](https://github.com/kubernetes/deployment-manager/tree/master/templates).
+Since walking the repository is not very efficient, we introduce the concept of
+collections.
+
+#### Definition
+
+A collection is a directory that contains a flat list of templates. Deployment
+manager will only discover templates at the root of a collection.
+
+So for example, in the section above, `templateA` and `templateB` live in the
+`templates` collection in the first example, and in the `big-data` collection in
+the second example.
+
+A registry may contain any number of collections. A single, unnamed collection
+is implied at the root of every registry, but additional named collections may
+be created at other points in the directory structure.
+
+#### Usage
+
+Of course, collections are useless if we can't reference them efficiently. A
+registry based template reference may therefore include a collection name. A
+collection name is the only path segment allowed between the repository name and
+the template name. So, for example, this is a valid template reference:
+
+```
+github.com/ownerA/repository2/collectionM/templateA:v1
+```
+
+but this is not:
+
+```
+github.com/ownerA/repository2/multiple/path/segments/are/not/allowed/templateA:v1
+```
+
+Because it may appear in a template reference, a collection name must not contain
+URL path separators (i.e., slashes). However, it may contain other delimiters
+(e.g., dots). So, for example, this is a valid template reference:
+
+```
+github.com/ownerA/repository2/dot.delimited.strings.are.allowed/templateA:v1
+```
+
+#### Mapping
+
+Currently, deployment manager maps collection names to directories. This means
+that registries can be at most one level deep. Soon, however, we plan to introduce
+a metadata file at the top level that maps collection names to paths. This will
+allow registries to have arbitrary organizations, by making it possible to place
+collections anywhere in the directory tree.
+
+When the metadata file is introduced, the current behavior will be the default.
+So, if the metadata file is not found in a given registry, or if a given collection
+name is not found in the metadata file, then deployment manager will simply map
+it to a directory name by default. This approach allows us to define collections
+at the top level now, and then move them to new locations later without breaking
+existing template references.
+
+## Using Template Registries
 
 ### Accessing a template registry
 
@@ -180,8 +270,8 @@ $ dm --properties prop1=value1,prop2=value2 deploy <template-name>:<version>
 DM relies on Github to provide the tools and processes needed to add, modify or
 delete the contents of a template registry. Conventions for changing a template
 registry are defined by the registry maintainers, and should be published in the
-top level README.md or a file it references, following usual Github practices.
+top level README.md or a file it references, following standard Github practices.
 
 The [Kubernetes Template Registry](https://github.com/kubernetes/deployment-manager/tree/master/templates)
-follows the [git setup](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md#git-setup)
+follows the [workflow](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md#git-setup)
 used by Kubernetes.

examples/bootstrap/bootstrap.yaml

 resources:
 - name: expandybird
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v1/replicatedservice.py
+  type: github.com/kubernetes/application-dm-templates/common/replicatedservice:v1
   properties:
     namespace: dm
     service_port: 8081
@@ -12,7 +12,7 @@ resources:
     labels:
       app: dm
 - name: resourcifier
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v1/replicatedservice.py
+  type: github.com/kubernetes/application-dm-templates/common/replicatedservice:v1
   properties:
     namespace: dm
     service_port: 8082
@@ -24,7 +24,7 @@ resources:
     labels:
       app: dm
 - name: manager
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v1/replicatedservice.py
+  type: github.com/kubernetes/application-dm-templates/common/replicatedservice:v1
   properties:
     namespace: dm
     service_port: 8080

examples/guestbook/guestbook.yaml

 resources:
 - name: frontend
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v1/replicatedservice.py
+  type: github.com/kubernetes/application-dm-templates/common/replicatedservice:v1
   properties:
     service_port: 80
     container_port: 80
@@ -8,5 +8,5 @@ resources:
     replicas: 3
     image: gcr.io/google_containers/example-guestbook-php-redis:v3
 - name: redis
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/redis/v1/redis.jinja
+  type: github.com/kubernetes/application-dm-templates/storage/redis:v1
   properties: null

examples/wordpress/wordpress.jinja

@@ -19,14 +19,14 @@
 
 resources:
 - name: nfs
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/nfs/v1/nfs.jinja
+  type: github.com/kubernetes/application-dm-templates/storage/nfs:v1
   properties:
     ip: {{ NFS_SERVER_IP }}
     port: {{ NFS_SERVER_PORT }}
     disk: {{ NFS_SERVER_DISK }}
     fstype: {{NFS_SERVER_DISK_FSTYPE }}
 - name: nginx
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v2/replicatedservice.py
+  type: github.com/kubernetes/application-dm-templates/common/replicatedservice:v2
   properties:
     service_port: {{ NGINX_PORT }}
     container_port: {{ NGINX_PORT }}
@@ -38,7 +38,7 @@ resources:
         persistentVolumeClaim:
           claimName: nfs
 - name: mysql
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v2/replicatedservice.py
+  type: github.com/kubernetes/application-dm-templates/common/replicatedservice:v2
   properties:
     service_port: {{ MYSQL_PORT }}
     container_port: {{ MYSQL_PORT }}
@@ -53,7 +53,7 @@ resources:
           pdName: {{ MYSQL_DISK }}
           fsType: {{ MYSQL_DISK_FSTYPE }}
 - name: wordpress-php
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v2/replicatedservice.py
+  type: github.com/kubernetes/application-dm-templates/common/replicatedservice:v2
   properties:
     service_name: wordpress-php
     service_port: {{ WORDPRESS_PHP_PORT }}

templates/README.md

-# Kubernetes Template Registry
-
-Welcome to the Kubernetes Template Registry!
-
-This registry holds Deployment Manager
-[templates](https://github.com/kubernetes/deployment-manager/tree/master/docs/design/design.md#templates)
-that you can use to deploy Kubernetes resources.
-
-For more information about installing and using Deployment Manager, see its
-[README.md](https://github.com/kubernetes/deployment-manager/tree/master/README.md).
-
-## Organization
-
-The Kubernetes Template Registry is organized as a flat list of template
-directories. Templates are versioned. The versions of a template live in version
-directories under its template directory. 
-
-For more information about Deployment Manager template registries, including
-directory structure and template versions, see the
-[template registry documentation](https://github.com/kubernetes/deployment-manager/tree/master/docs/templates/registry.md)
-
-## Contributing
-
-The Kubernetes Template Registry follows the
-[git setup](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md#git-setup)
-used by Kubernetes.
-
-You will also need to have a Contributor License Agreement on file, as described
-in [CONTRIBUTING.md](../CONTRIBUTING.md).

templates/nfs/v1/nfs.jinja

-{% set PROPERTIES = properties or {} %}
-{% set SERVER_IP = PROPERTIES['ip'] or '10.0.253.247' %}
-{% set SERVER_PORT = PROPERTIES['port'] or 2049 %}
-{% set SERVER_DISK = PROPERTIES['disk'] or 'nfs-disk' %}
-{% set SERVER_DISK_FSTYPE = PROPERTIES['fstype'] or 'ext4' %}
-{% set CLAIM_NAME = PROPERTIES['claim-name'] or 'nfs' %}
-
-resources:
-- name: nfs
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v2/replicatedservice.py
-  properties:
-    service_port: {{ SERVER_PORT }}
-    container_port: {{ SERVER_PORT }}
-    replicas: 1 # Has to be 1 because of the persistent disk
-    image: gcr.io/google_containers/volume-nfs
-    privileged: true
-    cluster_ip: {{ SERVER_IP }}
-    volumes:
-      - mount_path: /mnt/data
-        gcePersistentDisk:
-          pdName: {{ SERVER_DISK }}
-          fsType: {{ SERVER_DISK_FSTYPE }}
-- name: nfs-pvc
-  type: PersistentVolumeClaim
-  properties:
-    kind: PersistentVolumeClaim
-    apiVersion: v1
-    metadata:
-      name: {{ CLAIM_NAME }}
-    spec:
-      accessModes:
-        - ReadWriteMany
-      resources:
-        requests:
-          storage: 1Mi
-- name: nfs-pv
-  type: PersistentVolume
-  properties:
-    apiVersion: v1
-    kind: PersistentVolume
-    metadata:
-      name: {{ CLAIM_NAME }}
-    spec:
-      capacity:
-        storage: 1Mi
-      accessModes:
-        - ReadWriteMany
-      nfs:
-        server: {{ SERVER_IP }}
-        path: "/"

templates/nfs/v1/nfs.jinja.schema

-info:
-  title: NFS
-  description: Defines an NFS service.
-
-properties:
-  ip:
-    type: string
-    default: 10.0.253.247
-    description: The IP of the NFS service.
-  port:
-    type: int
-    default: 2049
-    description: The port of the NFS service.
-  disk:
-    type: string
-    default: nfs-disk
-    description: The name of the persistent disk the NFS service uses.
-  fstype:
-    type: string
-    default: ext4
-    description: The filesystem the disk of the NFS service uses.
-  claim-name:
-    type: string
-    default: nfs
-    description: The name of the PersistentVolumeClaim

templates/nfs/v1/nfs.yaml

-- name: nfs
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/nfs/v1/nfs.jinja

templates/redis/v1/redis.jinja

-{% set REDIS_PORT = 6379 %}
-{% set WORKERS = properties['workers'] if properties and properties['workers'] else 2 %}
-
-resources:
-- name: redis-master
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v1/replicatedservice.py
-  properties:
-    # This has to be overwritten since service names are hard coded in the code
-    service_name: redis-master
-    service_port: {{ REDIS_PORT }}
-    target_port: {{ REDIS_PORT }}
-    container_port: {{ REDIS_PORT }}
-    replicas: 1
-    container_name: master
-    image: redis
-
-- name: redis-slave
-  type: https://raw.githubusercontent.com/kubernetes/deployment-manager/master/templates/replicatedservice/v1/replicatedservice.py
-  properties:
-    # This has to be overwritten since service names are hard coded in the code
-    service_name: redis-slave
-    service_port: {{ REDIS_PORT }}
-    container_port: {{ REDIS_PORT }}
-    replicas: {{ WORKERS }}
-    container_name: worker
-    image: kubernetes/redis-slave:v2
-    # An example of how to specify env variables.
-    env:
-    - name: GET_HOSTS_FROM
-      value: env
-    - name: REDIS_MASTER_SERVICE_HOST
-      value: redis-master

templates/redis/v1/redis.jinja.schema

-info:
-  title: Redis cluster
-  description: Defines a redis cluster, using a single replica
-    replicatedservice for master and replicatedservice for workers.
-
-properties:
-  workers:
-    type: int
-    default: 2
-    description: Number of worker replicas.

templates/redis/v1/redis.yaml

-imports:
-- path: redis.jinja
-
-resources:
-- name: redis
-  type: redis.jinja

templates/replicatedservice/v1/replicatedservice.py

-######################################################################
-# Copyright 2015 The Kubernetes Authors All rights reserved.
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#     http://www.apache.org/licenses/LICENSE-2.0
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-######################################################################
-
-"""Defines a ReplicatedService type by creating both a Service and an RC.
-
-This module creates a typical abstraction for running a service in a
-Kubernetes cluster, namely a replication controller and a service packaged
-together into a single unit.
-"""
-
-import yaml
-
-SERVICE_TYPE_COLLECTION = 'Service'
-RC_TYPE_COLLECTION = 'ReplicationController'
-
-
-def GenerateConfig(context):
-  """Generates a Replication Controller and a matching Service.
-
-  Args:
-    context: Template context. See schema for context properties.
-
-  Returns:
-    A Container Manifest as a YAML string.
-  """
-  # YAML config that we're going to create for both RC & Service
-  config = {'resources': []}
-
-  name = context.env['name']
-  container_name = context.properties.get('container_name', name)
-  namespace = context.properties.get('namespace', 'default')
-
-  # Define things that the Service cares about
-  service_name = context.properties.get('service_name', name + '-service')
-  service_type = SERVICE_TYPE_COLLECTION
-
-  # Define things that the Replication Controller (rc) cares about
-  rc_name = name + '-rc'
-  rc_type = RC_TYPE_COLLECTION
-
-  service = {
-      'name': service_name,
-      'type': service_type,
-      'properties': {
-          'apiVersion': 'v1',
-          'kind': 'Service',
-          'metadata': {
-              'name': service_name,
-              'namespace': namespace,
-              'labels': GenerateLabels(context, service_name),
-          },
-          'spec': {
-              'ports': [GenerateServicePorts(context, container_name)],
-              'selector': GenerateLabels(context, name)
-          }
-      }
-  }
-  set_up_external_lb = context.properties.get('external_service', None)
-  if set_up_external_lb:
-    service['properties']['spec']['type'] = 'LoadBalancer'
-  config['resources'].append(service)
-
-  rc = {
-      'name': rc_name,
-      'type': rc_type,
-      'properties': {
-          'apiVersion': 'v1',
-          'kind': 'ReplicationController',
-          'metadata': {
-              'name': rc_name,
-              'namespace': namespace,
-              'labels': GenerateLabels(context, rc_name),
-          },
-          'spec': {
-              'replicas': context.properties['replicas'],
-              'selector': GenerateLabels(context, name),
-              'template': {
-                  'metadata': {
-                      'labels': GenerateLabels(context, name),
-                  },
-                  'spec': {
-                      'containers': [
-                          {
-                              'env': GenerateEnv(context),
-                              'name': container_name,
-                              'image': context.properties['image'],
-                              'ports': [
-                                  {
-                                      'name': container_name,
-                                      'containerPort': context.properties['container_port'],
-                                  }
-                              ]
-                          }
-                      ]
-                  }
-              }
-          }
-      }
-  }
-
-  config['resources'].append(rc)
-  return yaml.dump(config)
-
-
-def GenerateLabels(context, name):
-  """Generates labels either from the context.properties['labels'] or 
-     generates a default label 'app':name
-
-  We make a deep copy of the context.properties['labels'] section to avoid
-  linking in the yaml document, which I believe reduces readability of the
-  expanded template. If no labels are given, generate a default 'app':name.
-
-  Args:
-    context: Template context, which can contain the following properties:
-             labels - Labels to generate
-
-  Returns:
-    A dict containing labels in a name:value format
-  """
-  tmp_labels = context.properties.get('labels', None)
-  ret_labels = {'app': name}
-  if isinstance(tmp_labels, dict):
-    for key, value in tmp_labels.iteritems():
-      ret_labels[key] = value
-  return ret_labels
-
-
-def GenerateServicePorts(context, name):
-  """Generates a ports section for a service.
-
-  Args:
-    context: Template context, which can contain the following properties:
-             service_port - Port to use for the service
-             target_port - Target port for the service
-             protocol - Protocol to use.
-
-  Returns:
-    A dict containing a port definition
-  """
-  service_port = context.properties.get('service_port', None)
-  target_port = context.properties.get('target_port', None)
-  protocol = context.properties.get('protocol')
-
-  ports = {}
-  if name:
-    ports['name'] = name
-  if service_port:
-    ports['port'] = service_port
-  if target_port:
-    ports['targetPort'] = target_port
-  if protocol:
-    ports['protocol'] = protocol
-
-  return ports
-
-def GenerateEnv(context):
-  """Generates environmental variables for a pod.
-
-  Args:
-    context: Template context, which can contain the following properties:
-             env - Environment variables to set.
-
-  Returns:
-    A list containing env variables in dict format {name: 'name', value: 'value'}
-  """
-  env = []
-  tmp_env = context.properties.get('env', [])
-  for entry in tmp_env:
-    if isinstance(entry, dict):
-      env.append({'name': entry.get('name'), 'value': entry.get('value')})
-  return env

templates/replicatedservice/v1/replicatedservice.py.schema

-info:
-  title: Replicated Service
-  description: |
-    Defines a ReplicatedService type by creating both a Service and an RC.
-
-    This module creates a typical abstraction for running a service in a
-    Kubernetes cluster, namely a replication controller and a service packaged
-    together into a single unit.
-
-required:
-- image
-
-properties:
-  container_name:
-    type: string
-    description: Name to use for container. If omitted, name is used.
-  service_name:
-    type: string
-    description: Name to use for service. If omitted, name-service is used.
-  namespace:
-    type: string
-    description: Namespace to create resources in. If omitted, 'default' is
-      used.
-    default: default
-  protocol:
-    type: string
-    description: Protocol to use for the service.
-  service_port:
-    type: int
-    description: Port to use for the service.
-  target_port:
-    type: int
-    description: Target port to use for the service.
-  container_port:
-    type: int
-    description: Port to use for the container.
-  replicas:
-    type: int
-    description: Number of replicas to create in RC.
-  image:
-    type: string
-    description: Docker image to use for replicas.
-  labels:
-    type: object
-    description: Labels to apply.
-  env:
-    type: object
-    description: Environment variables to apply.
-    properties:
-      name:
-        type: string
-      value:
-        type: string
-  external_service:
-    type: boolean
-    description: If set to true, enable external load balancer.
-

templates/replicatedservice/v2/replicatedservice.py

-"""Defines a ReplicatedService type by creating both a Service and an RC.
-
-This module creates a typical abstraction for running a service in a
-Kubernetes cluster, namely a replication controller and a service packaged
-together into a single unit.
-"""
-
-import yaml
-
-SERVICE_TYPE_COLLECTION = 'Service'
-RC_TYPE_COLLECTION = 'ReplicationController'
-
-
-def GenerateConfig(context):
-  """Generates a Replication Controller and a matching Service.
-
-  Args:
-    context: Template context. See schema for context properties
-
-  Returns:
-    A Container Manifest as a YAML string.
-  """
-  # YAML config that we're going to create for both RC & Service
-  config = {'resources': []}
-
-  name = context.env['name']
-  container_name = context.properties.get('container_name', name)
-  namespace = context.properties.get('namespace', 'default')
-
-  # Define things that the Service cares about
-  service_name = context.properties.get('service_name', name + '-service')
-  service_type = SERVICE_TYPE_COLLECTION
-
-  # Define things that the Replication Controller (rc) cares about
-  rc_name = context.properties.get('rc_name', name + '-rc')
-  rc_type = RC_TYPE_COLLECTION
-
-  service = {
-      'name': service_name,
-      'type': service_type,
-      'properties': {
-          'apiVersion': 'v1',
-          'kind': 'Service',
-          'namespace': namespace,
-          'metadata': {
-              'name': service_name,
-              'labels': GenerateLabels(context, service_name),
-          },
-          'spec': {
-              'ports': [GenerateServicePorts(context, container_name)],
-              'selector': GenerateLabels(context, name)
-          }
-      }
-  }
-  set_up_external_lb = context.properties.get('external_service', None)
-  if set_up_external_lb:
-    service['properties']['spec']['type'] = 'LoadBalancer'
-  cluster_ip = context.properties.get('cluster_ip', None)
-  if cluster_ip:
-    service['properties']['spec']['clusterIP'] = cluster_ip
-  config['resources'].append(service)
-
-  rc = {
-      'name': rc_name,
-      'type': rc_type,
-      'properties': {
-          'apiVersion': 'v1',
-          'kind': 'ReplicationController',
-          'namespace': namespace,
-          'metadata': {
-              'name': rc_name,
-              'labels': GenerateLabels(context, rc_name),
-          },
-          'spec': {
-              'replicas': context.properties['replicas'],
-              'selector': GenerateLabels(context, name),
-              'template': {
-                  'metadata': {
-                      'labels': GenerateLabels(context, name),
-                  },
-                  'spec': {
-                      'containers': [
-                          {
-                              'env': GenerateEnv(context),
-                              'name': container_name,
-                              'image': context.properties['image'],
-                              'ports': [
-                                  {
-                                      'name': container_name,
-                                      'containerPort': context.properties['container_port'],
-                                  }
-                              ],
-                          }
-                      ],
-                  }
-              }
-          }
-      }
-  }
-
-  # Set up volume mounts
-  if context.properties.get('volumes', None):
-    rc['properties']['spec']['template']['spec']['containers'][0]['volumeMounts'] = []
-    rc['properties']['spec']['template']['spec']['volumes'] = []
-    for volume in context.properties['volumes']:
-      # mountPath should be unique
-      volume_name = volume['mount_path'].replace('/', '-').lstrip('-') + '-storage'
-      rc['properties']['spec']['template']['spec']['containers'][0]['volumeMounts'].append(
-          {
-              'name': volume_name,
-              'mountPath': volume['mount_path']
-          }
-      )
-      del volume['mount_path']
-      volume['name'] = volume_name
-      rc['properties']['spec']['template']['spec']['volumes'].append(volume)
-
-  if context.properties.get('privileged', False):
-    rc['properties']['spec']['template']['spec']['containers'][0]['securityContext'] = {
-        'privileged': True
-    }
-
-  config['resources'].append(rc)
-  return yaml.dump(config)
-
-
-# Generates labels either from the context.properties['labels'] or generates
-# a default label 'name':name
-def GenerateLabels(context, name):
-  """Generates labels from context.properties['labels'] or creates default.
-
-  We make a deep copy of the context.properties['labels'] section to avoid
-  linking in the yaml document, which I believe reduces readability of the
-  expanded template. If no labels are given, generate a default 'name':name.
-
-  Args:
-    context: Template context, which can contain the following properties:
-             labels - Labels to generate
-
-  Returns:
-    A dict containing labels in a name:value format
-  """
-  tmp_labels = context.properties.get('labels', None)
-  ret_labels = {'name': name}
-  if isinstance(tmp_labels, dict):
-    for key, value in tmp_labels.iteritems():
-      ret_labels[key] = value
-  return ret_labels
-
-
-def GenerateServicePorts(context, name):
-  """Generates a ports section for a service.
-
-  Args:
-    context: Template context, which can contain the following properties:
-             service_port - Port to use for the service
-             target_port - Target port for the service
-             protocol - Protocol to use.
-
-  Returns:
-    A dict containing a port definition
-  """
-  service_port = context.properties.get('service_port', None)
-  target_port = context.properties.get('target_port', None)
-  protocol = context.properties.get('protocol')
-
-  ports = {}
-  if name:
-    ports['name'] = name
-  if service_port:
-    ports['port'] = service_port
-  if target_port:
-    ports['targetPort'] = target_port
-  if protocol:
-    ports['protocol'] = protocol
-
-  return ports
-
-def GenerateEnv(context):
-  """Generates environmental variables for a pod.
-
-  Args:
-    context: Template context, which can contain the following properties:
-             env - Environment variables to set.
-
-  Returns:
-    A list containing env variables in dict format {name: 'name', value: 'value'}
-  """
-  env = []
-  tmp_env = context.properties.get('env', [])
-  for entry in tmp_env:
-    if isinstance(entry, dict):
-      env.append({'name': entry.get('name'), 'value': entry.get('value')})
-  return env

templates/replicatedservice/v2/replicatedservice.py.schema

-info:
-  title: Replicated Service
-  description: |
-    Defines a ReplicatedService type by creating both a Service and an RC.
-
-    This module creates a typical abstraction for running a service in a
-    Kubernetes cluster, namely a replication controller and a service packaged
-    together into a single unit.
-
-required:
-- image
-
-properties:
-  container_name:
-    type: string
-    description: Name to use for container. If omitted, name is used.
-  service_name:
-    type: string
-    description: Name to use for service. If omitted, name-service is used.
-  namespace:
-    type: string
-    description: Namespace to create resources in. If omitted, 'default' is
-      used.
-    default: default
-  protocol:
-    type: string
-    description: Protocol to use for the service.
-  service_port:
-    type: int
-    description: Port to use for the service.
-  target_port:
-    type: int
-    description: Target port to use for the service.
-  container_port:
-    type: int
-    description: Port to use for the container.
-  replicas:
-    type: int
-    description: Number of replicas to create in RC.
-  image:
-    type: string
-    description: Docker image to use for replicas.
-  labels:
-    type: object
-    description: Labels to apply.
-  env:
-    type: array
-    description: Environment variables to apply.
-    properties:
-      name:
-        type: string
-      value:
-        type: string
-  external_service:
-    type: boolean
-    description: If set to true, enable external load balancer.
-  cluster_ip:
-    type: string
-    description: IP to use for the service
-  privileged:
-    type: boolean
-    description: If set to true, enable privileged container
-  volumes:
-    type: array
-    description: Volumes to mount.
-    items:
-      type: object
-      properties:
-        mounth_path:
-          type: string
-          description: Path to mount volume
-        # See https://cloud.google.com/container-engine/docs/spec-schema?hl=en for possible volumes. Since we only use gcePersistentDisk and NFS in our examples we have only added these ones.
-        oneOf:
-          gcePersistentDisk:
-            pdName:
-              type: string
-              description: Persistent's disk name
-            fsType:
-              type: string
-              description: Filesystem type of the persistent disk
-          nfs:
-            server:
-              type: string
-              description: The hostname or IP address of the NFS server
-            path:
-              type: string
-              description: The path that is exported by the NFS server
-            readOnly:
-              type: boolean
-              description: Forces the NFS export to be mounted with read-only permissions
-

templates/spark/v1/example.yaml

-imports:
-- path: spark.jinja
-
-resources:
-- name: spark
-  type: spark.jinja

templates/spark/v1/spark.jinja

-{% set PROPERTIES = properties or {} %}
-{% set WORKERS = PROPERTIES['workers'] or 2 %}
-{% set REPO = PROPERTIES['repository'] or 'gcr.io/google_containers' %}
-{% set SPARK_VERSION = PROPERTIES['spark_version'] or '1.5.1_v2' %}
-{% set ZEPPELIN_VERSION = PROPERTIES['zeppelin_version'] or 'v0.5.5_v2' %}
-{% set MASTER_CPU = PROPERTIES['master_cpu'] or '100m' %}
-{% set ZEPPELIN_CPU = PROPERTIES['zeppelin_cpu'] or '100m' %}
-{% set WORKER_CPU = PROPERTIES['worker_cpu'] or '100m' %}
-
-# TODO(zmerlynn): This can't be a standard replicatedservice (yet) because of:
-# - the double containerPort
-# - the cpu request
-# (neither are possible in the type as of v2)
-resources:
-- name: spark-master
-  type: ReplicationController
-  properties:
-    kind: ReplicationController
-    apiVersion: v1
-    metadata:
-      name: spark-master-controller
-    spec:
-      replicas: 1
-      selector:
-        component: spark-master
-      template:
-        metadata:
-          labels:
-            component: spark-master
-        spec:
-          containers:
-            - name: spark-master
-              image: {{ REPO }}/spark-master:{{ SPARK_VERSION }}
-              ports:
-                - containerPort: 7077
-                - containerPort: 8080
-              resources:
-                requests:
-                  cpu: {{ MASTER_CPU }}
-- name: spark-master-service
-  type: Service
-  properties:
-    kind: Service
-    apiVersion: v1
-    metadata:
-      name: spark-master
-    spec:
-      ports:
-        - port: 7077
-          targetPort: 7077
-      selector:
-        component: spark-master
-- name: spark-webui
-  type: Service
-  properties:
-    kind: Service
-    apiVersion: v1
-    metadata:
-      name: spark-webui
-    spec:
-      ports:
-        - port: 8080
-          targetPort: 8080
-      selector:
-        component: spark-master
-- name: spark-worker-controller
-  type: ReplicationController
-  properties:
-    kind: ReplicationController
-    apiVersion: v1
-    metadata:
-      name: spark-worker-controller
-    spec:
-      replicas: 3
-      selector:
-        component: spark-worker
-      template:
-        metadata:
-          labels:
-            component: spark-worker
-        spec:
-          containers:
-            - name: spark-worker
-              image: {{ REPO }}/spark-worker:{{ SPARK_VERSION }}
-              ports:
-                - containerPort: 8081
-              resources:
-                requests:
-                  cpu: {{ WORKER_CPU }}
-- name: zeppelin-controller
-  type: ReplicationController
-  properties:
-    kind: ReplicationController
-    apiVersion: v1
-    metadata:
-      name: zeppelin-controller
-    spec:
-      replicas: 1
-      selector:
-        component: zeppelin
-      template:
-        metadata:
-          labels:
-            component: zeppelin
-        spec:
-          containers:
-            - name: zeppelin
-              image: {{ REPO }}/zeppelin:{{ ZEPPELIN_VERSION }}
-              ports:
-                - containerPort: 8080
-              resources:
-                requests:
-                  cpu: {{ ZEPPELIN_CPU }}

templates/spark/v1/spark.jinja.schema

-info:
-  title: Spark cluster with Zeppelin front-end
-  description: |
-    Defines a Spark cluster with a single master in standalone mode,
-    and a Zeppelin web notebook front-end. After deploying the
-    cluster, the Spark WebUI can be accessed by starting:
-
-      kubectl proxy --port=8001 &
-
-    Then visiting:
-
-      http://localhost:8001/api/v1/proxy/namespaces/default/services/spark-webui/
-
-    The Zeppelin WebUI can be access by finding the Zeppelin pod and port-forwarding:
-
-      kubectl get pods -lcomponent=zeppelin # Take the pod from here
-      kubectl port-forward zeppelin-controller-abcef :8080
-
-    Then visit the forwarded port.
-
-properties:
-  master_cpu:
-    type: string
-    default: 100m
-    description: CPU request for the master (in KCUs) (see http://kubernetes.io/v1.1/docs/design/resources.html)
-
-  workers:
-    type: int
-    default: 3
-    description: Number of Spark workers.
-
-  worker_cpu:
-    type: string
-    default: 100m
-    description: CPU request for each worker (in KCUs) (see http://kubernetes.io/v1.1/docs/design/resources.html)
-
-  repository:
-    type: string
-    default: gcr.io/google_containers
-    description: Docker repo that houses the {spark-worker, spark-master, zeppelin} images.
-
-  spark_version:
-    type: string
-    default: 1.5.1_v2
-    description: spark-worker / spark-master image version to use
-
-  zeppelin_version:
-    type: string
-    default: 0.5.5_v2
-    description: zeppelin image version to use