Merge pull request #1012 from MBishop17/migration-guide

Migration guide
This commit is contained in:
Denise
2018-12-18 11:29:17 -08:00
committed by GitHub
72 changed files with 1599 additions and 145 deletions
+27 -141
View File
@@ -1,165 +1,51 @@
---
title: Migrating from Rancher v1.6 Cattle to v2.x
title: Migrating from Rancher v1.6 to v2.x
weight: 10000
---
Rancher 2.x has been rearchitected and rewritten with the goal of providing a complete management solution for Kubernetes and Docker. Due to these extensive changes, there is no direct upgrade path from 1.6.x to 2.x, but rather a migration of your 1.6 application workloads into the 2.x Kubernetes equivalent. In 1.6, the most common orchestration used was Rancher's own engine called Cattle. The following blogs (that will be converted in an official guide) explain and educate our Cattle users on running workloads in a Kubernetes environment.
Rancher v2.x has been rearchitected and rewritten with the goal of providing a complete management solution for Kubernetes and Docker. Due to these extensive changes, there is no direct upgrade path from v1.6.x to v2.x, but rather a migration of your v1.6 services into v2.x as Kubernetes workloads. In v1.6, the most common orchestration used was Rancher's own engine called Cattle. The following guide explains and educates our Cattle users on running workloads in a Kubernetes environment.
If you are an existing Kubernetes user on Rancher 1.6, you only need to review the [Get Started](#1-get-started) section to prepare you on what to expect on a new 2.x Rancher cluster.
## Video
## Kubernetes Basics
Rancher 2.x is built on the [Kubernetes](https://kubernetes.io/docs/home/?path=users&persona=app-developer&level=foundational) container orchestrator. This shift in underlying technology for 2.x is a large departure from 1.6, which supported several popular container orchestrators. Since Rancher is now based entirely on Kubernetes, it's helpful to learn the Kubernetes basics.
The following table introduces and defines some key Kubernetes concepts.
| **Concept** | **Definition** |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cluster | A collection of machines that run containerized applications managed by Kubernetes. |
| Namespace | A virtual cluster, multiple of which can be supported by a single physical cluster. |
| Node | One of the physical (or virtual) machines that make up a cluster. |
| Pod | The smallest and simplest Kubernetes object. A pod represents a set of running [containers](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/#why-containers) on your cluster. |
| Deployment | An API object that manages a replicated application. |
| Workload | Units of work that are running on the cluster, these can be pods or deployments. |
## Migration Cheatsheet
Because Rancher 1.6 defaulted to our Cattle container orchestrator, it primarily used terminology related to Cattle. However, because Rancher 2.x uses Kubernetes, it aligns with the Kubernetes naming standard. This shift could be confusing for people unfamiliar with Kubernetes, so we've created a table that maps terms commonly used in Rancher 1.6 to their equivalents in Rancher 2.x.
| **Rancher 1.6** | **Rancher 2.x** |
| --- | --- |
| Container | Pod |
| Services | Workload |
| Load Balancer | Ingress |
| Stack | Namespace |
| Environment | Project (Administration)/Cluster (Compute)
| Host | Node |
| Catalog | Helm |
<br/>
More detailed information on Kubernetes concepts can be found in the
[Kubernetes Concepts Documentation](https://kubernetes.io/docs/concepts/).
This video demonstrates a complete walk through of migration from Rancher v1.6 to v2.x.
{{< youtube OIifcqj5Srw >}}
## Migration Plan
<!-- TOC -->
- [1. Get Started](#1-get-started)
- [2. Run Migration-Tools CLI](#2-run-migration-tools-cli)
- [3. Migrate Applications](#3-migrate-applications)
- [4. Expose Your Services](#4-expose-your-services)
- [5. Monitor Your Applications](#5-monitor-your-applications)
- [6. Schedule Deployments](#6-schedule-deployments)
- [7. Service Discovery](#7-service-discovery)
- [8. Load Balancing](#8-load-balancing)
<!-- /TOC -->
## 1. Get Started
As a Rancher 1.6 user who's interested in moving to 2.x, how should you get started with migration? The following blog provides a short checklist to help with this transition.
Blog Post: [Migrating from Rancher 1.6 to Rancher 2.x—A Short Checklist](https://rancher.com/blog/2018/2018-08-09-migrate-1dot6-setup-to-2dot0/)
## 2. Run Migration-Tools CLI
The migration-tools CLI is a tool that helps you recreate your applications in Rancher v2.x. This tool exports your Rancher v1.6 applications as Compose files and converts them to a Kubernetes manifest that Rancher 2.x can consume.
This command line interface tool:
- Exports Compose files (i.e., `docker-compose.yml` and `rancher-compose.yml`) for all your stacks that are Cattle environments in your Rancher 1.6 server. For every stack, files are exported to a `<EXPORT_DIR>/<ENV_NAME>/<STACK_NAME>` folder.
- Parses Compose files that you've exported from your Rancher 1.6 stack and converts them to a Kubernetes manifest that Rancher v2.x can consume. The tool also outputs a list of constructs present in the Compose files that cannot be converted automatically to Rancher 2.x. These are directives that you'll have to manually configure in the Kubernetes YAML.
### A. Download Migration-Tools CLI
The migration-tools CLI for your platform can be downloaded from our [GitHub releases page](https://github.com/rancher/migration-tools/releases). The tools are available for Linux, Mac, and Windows platforms.
>**Want to more about Kubernetes before getting started?** Read our [Kubernetes Introduction]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/kub-intro).
### B. Configure Migration-Tools CLI
- [1. Get Started]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/get-started)
After you download migration-tools CLI, rename it and make it executable.
>**Already a Kubernetes user in v1.6?**
>
> _Get Started_ is the only section you need to review for migration to v2.x. You can skip everything else.
- [2. Migrate Your Apps]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/run-migration-tool/)
- [3. Expose Your Services]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/expose-services/)
- [4. Monitor Your Apps]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/monitor-apps)
- [5. Schedule Workloads]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/schedule-workloads/)
- [6. Service Discovery]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/discover-services/)
- [7. Load Balancing]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/load-balancing/)
1. Open Terminal and change to the directory that contains the migration-tools file.
## Migration Example Files
1. Rename the file to `migration-tools` so that it no longer includes the platform name.
Throughout this migration guide, we will reference several example services from Rancher v1.6 that we're migrating to v2.x. These services are:
1. Enter the following command to make `migration-tools` an executable:
- A service named `web`, which runs [Let's Chat](http://sdelements.github.io/lets-chat/), a self-hosted chat for small teams.
- A service named `database`, which runs [Mongo DB](https://www.mongodb.com/), an open source document database.
- A service named `webLB`, which runs [HAProxy](http://www.haproxy.org/), an open source load balancer used in Rancher v1.6.
```
chmod +x migration-tools
```
During migration, we'll export these services from Rancher v1.6. The export generates a unique directory for each Rancher v1.6 environment and stack, and two files are output into each stack's directory:
### C. Run Migration-Tools CLI
- `docker-compose.yml`
Next, use the migration-tools CLI to export all stacks in all of the Cattle environments into Compose files. Then, for stacks that you want to migrate to Rancher 2.x, convert the Compose files into Kubernetes YAML.
A file that contains standard Docker directives for each service in your stack. We'll be converting these files to Kubernetes manifests that can be read by Rancher v2.x.
>**Want full usage and options for the migration-tools CLI?** See the [Migration-Tools CLI Reference]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/migration-tools-ref).
- `rancher-compose.yml`
1. Export the Compose files for all stacks in all of the Cattle environments in your Rancher 1.6 server.
Execute the following command, replacing each placeholder with your values. The access key and secret key are Account API keys, which will allow you to export from all Cattle environments.
```
migration-tools export --url <RANCHER_URL> --access-key <RANCHER_ACCESS_KEY> --secret-key <RANCHER_SECRET_KEY> --export-dir <EXPORT_DIR>
```
**Step Result:** The migration-tools CLI exports Compose files for each stack in every Cattle environments in the `--export-dir` directory. If you omitted this option, the files are saved to your current directory.
A file for Rancher-specific functionality such as health checks and load balancers. These files cannot be read by Rancher v2.x, so don't worry about their contents—we're discarding them and recreating them using the v2.x UI.
1. Convert the exported Compose files for a stack to Kubernetes YAML.
Execute the following command, replacing each placeholder with the absolute path to your stack's Compose files. For each stack, you'll have to re-run the command for each pair of Compose files that was exported.
```
migration-tools parse --docker-file <DOCKER_COMPOSE_ABSOLUTE_PATH> --rancher-file <RANCHER_COMPOSE_ABSOLUTE_PATH>
```
>**Note:** If you omit the `--docker-file` and `--rancher-file` options from your command, the migration-tools CLI checks its home directory for these Compose files.
**Step Result:** The migration-tools CLI parses your Compose files and outputs Kubernetes YAML specs as well as an `output.txt` file. For each service in the stack, a YAML spec file is created and named the same as your service. The `output.txt` file lists all constructs for each service in `docker-compose.yml` that requires special handling to be successfully migrated to Rancher 2.x. Each construct links to the relevant blog articles on how to implement it in Rancher 2.x (these articles are also listed below).
## 3. Migrate Applications
In Rancher 1.6, you launch applications as _services_ and organize them under _stacks_ in an _environment_, which represents a compute and administrative boundary. Rancher 1.6 supports the Compose standard and provides import/export for application configurations using the following files: `docker-compose.yml` and `rancher-compose.yml`. In 2.x the environment concept doesn't exist. Instead it's replaced by:
- **Cluster:** The compute boundary.
- **Project:** An administrative boundary.
The following article explores how to map Cattle's stack and service design to Kubernetes. It also demonstrates how to migrate a simple application from Rancher 1.6 to 2.x using either the Rancher UI or Docker Compose.
Blog Post: [A Journey from Cattle to Kubernetes!](https://rancher.com/blog/2018/2018-08-02-journey-from-cattle-to-k8s/)
## 4. Expose Your Services
In Rancher 1.6, you could provide external access to your applications using port mapping. This article explores how to publicly expose your services in Rancher 2.x. It explores both UI and CLI methods to transition the port mapping functionality.
Blog Post: [From Cattle to Kubernetes—How to Publicly Expose Your Services in Rancher 2.x](https://rancher.com/blog/2018/expose-and-monitor-workloads/)
## 5. Monitor Your Applications
Rancher 1.6 provided TCP and HTTP healthchecks using its own healthcheck microservice. Rancher 2.x uses native Kubernetes healthcheck support instead. This article overviews how to configure it in Rancher 2.x.
Blog Post: [From Cattle to Kubernetes—Application Healthchecks in Rancher 2.x](https://rancher.com/blog/2018/2018-08-22-k8s-monitoring-and-healthchecks/)
## 6. Schedule Deployments
Scheduling application containers on available resources is a key container orchestration technique. The following blog reviews how to schedule containers in Rancher 2.x for those familiar with 1.6 scheduling labels (such as affinity and anti-affinity). It also explores how to launch a global service in 2.x.
Blog Post: [From Cattle to Kubernetes—Scheduling Workloads in Rancher 2.x](https://rancher.com/blog/2018/2018-08-29-scheduling-options-in-2-dot-0/)
## 7. Service Discovery
Rancher 1.6 provides service discovery within and across stacks using its own internal DNS microservice. It also supports pointing to external services and creating aliases. Moving to Rancher 2.x, you can replicate this same service discovery behavior. The following blog reviews this topic and the solutions needed to achieve service discovery parity in Rancher 2.x.
Blog Post: [From Cattle to Kubernetes—Service Discovery in Rancher 2.x](https://rancher.com/blog/2018/2018-09-04-service_discovery_2dot0/)
## 8. Load Balancing
How to achieve TCP/HTTP load balancing and configure hostname/path-based routing in Rancher 2.x.
Blog Post: [From Cattle to Kubernetes-How to Load Balance Your Services in Rancher 2.x](https://rancher.com/blog/2018/2018-09-13-load-balancing-options-2dot0/)
In Rancher 1.6, a load balancer was used to expose your applications from within the Rancher environment for external access. In Rancher 2.x, the concept is the same. There is a Load Balancer option to expose your services. In the language of Kubernetes, this function is more often referred to as an _Ingress_. In short, load balancer and Ingress play the same role.
### [Next: Get Started]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/get-started)
@@ -0,0 +1,92 @@
---
title: "6. Service Discovery"
weight: 600
---
Service discovery is one of the core functionalities of any container-based environment. Once you have packaged and launched your application, the next step is making it discoverable to other containers in your environment or the external world. This document will describe how to use the service discovery support provided by Rancher v2.x so that you can find them by name.
This document will also show you how to link the workloads and services that you migrated into Rancher v2.x. When you parsed your services from v1.6 using migration-tools CLI, it output two files for each service: one deployment manifest and one service manifest. You'll have to link these two files together before the deployment works correctly in v2.x.
<figcaption>Resolve the <code>output.txt</code> Link Directive</figcaption>
![Resolve Link Directive]({{< baseurl >}}/img/rancher/resolve-links.png)
## In This Document
<!-- TOC -->
- [Service Discovery: Rancher v1.6 vs. v2.x](#service-discovery-rancher-v16-vs-v2x)
- [Service Discovery Within and Across Stacks](#service-discovery-within-and-across-stacks)
- [Container Discovery](#container-discovery)
- [Service Name Alias Creation](#service-name-alias-creation)
<!-- /TOC -->
## Service Discovery: Rancher v1.6 vs. v2.x
For Rancher v2.x, we've replaced the Rancher DNS microservice used in v1.6 with native [Kubernetes DNS support](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/), which provides equivalent service discovery for Kubernetes workloads and pods. Former Cattle users can replicate all the service discovery features from Rancher v1.6 in v2.x. There's no loss of functionality.
Kubernetes schedules a DNS pod and service in the cluster, which is similar to the [Rancher v1.6 DNS microservice](https://rancher.com/docs/rancher/v1.6/en/cattle/internal-dns-service/#internal-dns-service-in-cattle-environments). Kubernetes then configures its kubelets to route all DNS lookups to this DNS service, which is skyDNS, a flavor of the default Kube-DNS implementation.
The following table displays each service discovery feature available in the two Rancher releases.
Service Discovery Feature | Rancher v1.6 | Rancher v2.x | Description
--------------------------|--------------|--------------|-------------
[service discovery within and across stack][1] (i.e., clusters) | ✓ | ✓ | All services in the stack are resolvable by `<service_name>` and by `<service_name>.<stack_name>` across stacks.
[container discovery][2] | ✓ | ✓ | All containers are resolvable globally by their name.
[service alias name creation][3] | ✓ | ✓ | Adding an alias name to services and linking to other services using aliases.
[discovery of external services][4] | ✓ | ✓ | Pointing to services deployed outside of Rancher using the external IP(s) or a domain name.
[1]: #service-discovery-within-and-across-stacks
[2]: #container-discovery
[3]: #service-name-alias-creation
[4]: #service-name-alias-creation
<br/>
### Service Discovery Within and Across Namespaces
When you create a _new_ workload in v2.x (not migrated, more on that [below](#linking-migrated-workloads-and-services)), Rancher automatically creates a service with an identical name, and then links the service and workload together. If you don't explicitly expose a port, the default port of `42` is used. This practice makes the workload discoverable within and across namespaces by its name.
### Container Discovery
Individual pods running in the Kubernetes cluster also get a DNS record assigned, which uses dot notation as well: `<POD_IP_ADDRESS>.<NAMESPACE_NAME>.pod.cluster.local`. For example, a pod with an IP of `10.42.2.7` in the namespace `default` with a DNS name of `cluster.local` would have an entry of `10-42-2-7.default.pod.cluster.local`.
Pods can also be resolved using the `hostname` and `subdomain` fields if set in the pod spec. Details about this resolution is covered in the [Kubernetes docs](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/).
### Linking Migrated Workloads and Services
When you migrate v1.6 services to v2.x, Rancher does not automatically create a Kubernetes service record for each migrated deployment. Instead, you'll have to link the deployment and service together manually, using any of the methods listed below.
In the image below, the `web-deployment.yml` and `web-service.yml` files [created after parsing]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/run-migration-tool/#migration-example-file-output) our [migration example services]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/#migration-example-files) are linked together.
<figcaption>Linked Workload and Kubernetes Service</figcaption>
![Linked Workload and Kubernetes Service]({{< baseurl >}}/img/rancher/linked-service-workload.png)
### Service Name Alias Creation
Just as you can create an alias for Rancher v1.6 services, you can do the same for Rancher v2.x workloads. Similarly, you can also create DNS records pointing to services running externally, using either their hostname or IP address. These DNS records are Kubernetes service objects.
Using the v2.x UI, use the context menu to navigate to the `<CLUSTER> > <PROJECT>` view and choose the **Service Discovery** tab. All existing DNS records created for your workloads are listed under each namespace.
Click **Add Record** to create new DNS records. Then view the various options supported to link to external services or to create aliases for another workload, DNS record, or set of pods.
<figcaption>Add Service Discovery Record</figcaption>
![Add Service Discovery Record]({{< baseurl >}}/img/rancher/add-record.png)
The following table indicates which alias options are implemented natively by Kubernetes and which options are implemented by Rancher leveraging Kubernetes.
Option | Kubernetes-implemented? | Rancher-implemented?
-------|-------------------------|---------------------
Pointing to an external hostname | ✓ | |
Pointing to a set of pods that match a selector | ✓ | |
Pointing to an external IP address | | ✓
Pointing to another workload | | ✓
Create alias for another DNS record | | ✓
### [Next: Load Balancing]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/load-balancing/)
@@ -0,0 +1,232 @@
---
title: "3. Expose Your Services"
weight: 400
---
In testing environments, you usually need to route external traffic to your cluster containers by using an unadvertised IP and port number, providing users access to their apps. You can accomplish this goal using port mapping, which exposes a workload (i.e., service) publicly over a specific port, provided you know your node IP address(es). You can either map a port using HostPorts (which exposes a service on a specified port on a single node) or NodePorts (which exposes a service on _all_ nodes on a single port).
Use this document to correct workloads that list `ports` in `output.txt`. You can correct it by either setting a HostPort or a NodePort.
<figcaption>Resolve <code>ports</code> for the <code>web</code> Workload</figcaption>
![Resolve Ports]({{< baseurl >}}/img/rancher/resolve-ports.png)
## In This Document
<!-- TOC -->
- [What's Different About Exposing Services in Rancher v2.x?](#whats-different-about-exposing-services-in-rancher-v2x)
- [HostPorts](#hostports)
- [Setting HostPorts](#setting-hostports)
- [NodePorts](#nodeports)
- [Setting NodePorts](#setting-nodeports)
<!-- /TOC -->
## What's Different About Exposing Services in Rancher v2.x?
In Rancher v1.6, we used the term _Port Mapping_ for exposing an IP address and port where your you and your users can access a service.
In Rancher v2.x, the mechanisms and terms for service exposure have changed and expanded. You now have two port mapping options: _HostPorts_ (which is most synonymous with v1.6 port mapping, allows you to expose your app at a single IP and port) and _NodePorts_ (which allows you to map ports on _all_ of your cluster nodes, not just one).
Unfortunately, port mapping cannot be parsed by the migration-tools CLI. If the services you're migrating from v1.6 to v2.x have port mappings set, you'll have to either set a [HostPort](#hostports) or [NodePort](#nodeports) as a replacement.
## HostPorts
A _HostPort_ is a port exposed to the public on a _specific node_ running one or more pod. Traffic to the node and the exposed port (`<HOST_IP>:<HOSTPORT>`) are routed to the requested container's private port. Using a HostPort for a Kubernetes pod in Rancher v2.x is synonymous with creating a public port mapping for a container in Rancher v1.6.
In the following diagram, a user is trying to access an instance of Nginx, which is running within a pod on port 80. However, the Nginx deployment is assigned a HostPort of 9890. The user can connect to this pod by browsing to its host IP address, followed by the HostPort in use (9890 in case).
![HostPort Diagram]({{< baseurl >}}/img/rancher/hostPort.svg)
#### HostPort Pros
- Any port available on the host can be exposed.
- Configuration is simple, and the HostPort is set directly in the Kubernetes pod specifications. Unlike NodePort, no other objects need to be created to expose your app.
#### HostPort Cons
- Limits the scheduling options for your pod, as only hosts with vacancies for your chosen port can be used.
- If the scale of your workload is larger than the number of nodes in your Kubernetes cluster, the deployment fails.
- Any two workloads that specify the same HostPort cannot be deployed to the same node.
- If the host where your pods are running becomes unavailable, Kubernetes reschedules the pods to different nodes. Thus, if the IP address for your workload changes, external clients of your application will lose access to the pod. The same thing happens when you restart your pods—Kubernetes reschedules them to a different node.
## Setting HostPorts
You can set a HostPort for migrated workloads (i.e., services) using the Rancher v2.x UI. To add a HostPort, browse to the project containing your workloads, and edit each workload that you want to expose, as shown below. Map the port that your service container exposes to the HostPort exposed on your target node.
For example, for the web-deployment.yml file parsed from v1.6 that we've been using as a sample, we would edit its Kubernetes manifest, set the publish the port that the container uses, and then declare a HostPort listening on the port of your choice (`9890`) as shown below. You can then access your workload by clicking the link created in the Rancher UI.
<figcaption>Port Mapping: Setting HostPort</figcaption>
![Set HostPort]({{< baseurl >}}/img/rancher/set-hostport.gif)
<!--
{{% tabs %}}
{{% tab "Migrated Workload" %}}
While parsing Compose files to Kubernetes manifest, our migration-tool does not convert Rancher v1.6 port mappings to v2.x HostPorts. If you want to set a HostPort to expose one of the deployments you're migrating (not services), you'll have to manually add it to the [parsed Kubernetes manifest]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/run-migration-tool/#output), as displayed in the sample below.
In this sample, we've taken the [web-deployment.yml]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/run-migration-tool/#migration-example-file-output) file parsed from [docker-compose.yml]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/run-migration-tool/#migration-example-files) specified for Let's Chat.
<figcaption>HostPort Kubernetes manifest Example: <code>web-deployment.yaml</code></figcaption>
```YAML
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
io.rancher.container.pull_image: always
io.rancher.scheduler.global: "true"
kompose.cmd: ./migration-tools parse --docker-file docker-compose.yml --rancher-file
rancher-compose.yml
kompose.version: 1.16.0 ()
creationTimestamp: null
labels:
io.kompose.service: web
name: web
spec:
replicas: 0
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: web
spec:
containers:
- image: sdelements/lets-chat
name: web
ports:
- containerPort: 8080
hostPort: 9890 # NEW HOSTPORT DECLARATION
resources: {}
stdin: true
tty: true
restartPolicy: Always
status: {}
```
{{% /tab %}}
{{% tab "New Workload" %}}
While creating a new workload using the Rancher v2.x UI, you may specify a HostPort in the **Port Mapping** section. This action adds a `hostPort` value to the `ports` section of the cluster's Kubernetes manifest automatically. For more information on using Rancher UI to set a HostPort for a workload, see [Deploying Workloads]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/workloads/deploy-workloads/).
<figcaption>Workload Deployment: Setting HostPort</figcaption>
![HostPort Configuration]({{< baseurl >}}/img/rancher/deploy-workload-hostport.png)
{{% /tab %}}
{{% /tabs %}}
-->
## NodePorts
A _NodePort_ is a port that's open to the public _on each_ of your cluster nodes. When the NodePort receives a request for any of the cluster hosts' IP address for the set NodePort value, NodePort (which is a Kubernetes service) routes traffic to a specific pod, regardless of what node it's running on. NodePort provides a static endpoint where external requests can reliably reach your pods.
NodePorts help you circumvent an IP address shortcoming. Although pods can be reached by their IP addresses, they are disposable by nature. Pods are routinely destroyed and recreated, getting a new IP address with each replication. Therefore, IP addresses are not a reliable way to access your pods. NodePorts help you around this issue by providing a static service where they can always be reached. Even if your pods change their IP addresses, external clients dependent on them can continue accessing them without disruption, all without any knowledge of the pod re-creation occurring on the back end.
In the following diagram, a user is trying to connect to an instance of Nginx running in a Kubernetes cluster managed by Rancher. Although he knows what NodePort Nginx is operating on (30216 in this case), he does not know the IP address of the specific node that the pod is running on. However, with NodePort enabled, he can connect to the pod using the IP address for _any_ node in the cluster. Kubeproxy will forward the request to the correct node and pod.
![NodePort Diagram]({{< baseurl >}}/img/rancher/nodePort.svg)
NodePorts are available within your Kubernetes cluster on an internal IP. If you want to expose pods external to the cluster, use NodePorts in conjunction with an external load balancer. Traffic requests from outside your cluster for `<NodeIP>:<NodePort>` are directed to the workload by kube-proxy. The `<NodeIP>` can be the IP address of any node in your Kubernetes cluster.
#### NodePort Pros
- Creating a NodePort service provides a static public endpoint to your workload pods. There, even if the pods are destroyed, Kubernetes can deploy the workload anywhere in the cluster without altering the public endpoint.
- The scale of the pods is not limited by the number of nodes in the cluster. NodePort allows decoupling of public access from the number and location of pods.
#### NodePort Cons
- When a NodePort is used, that `<NodeIP>:<NodePort>` is reserved in your Kubernetes cluster on all nodes, even if the workload is never deployed to the other nodes.
- You can only specify a port from the configured range (`30000-32767`) and not any random port.
- An extra Kubernetes object (a Kubernetes service of type NodePort) is needed to expose your workload. Thus, finding out how your application is exposed is not straightforward.
## Setting NodePorts
You can set a NodePort for migrated workloads (i.e., services) using the Rancher v2.x UI. To add a NodePort, browse to the project containing your workloads, and edit each workload that you want to expose, as shown below. Map the port that your service container exposes to a NodePort, which you'll be able to access from each cluster node.
For example, for the web-deployment.yml file parsed from v1.6 that we've been using as a sample, we would edit its Kubernetes manifest, set the publish the port that the container uses, and then declare a NodePort. You can then access your workload by clicking the link created in the Rancher UI.
>**Note:**
>
>- If you set a NodePort without giving it a value, Rancher chooses a port at random from the following range: `30000-32767`.
>- If you manually set a NodePort, you must assign it a value within the `30000-32767` range.
<figcaption>Port Mapping: Setting NodePort</figcaption>
![Set NodePort]({{< baseurl >}}/img/rancher/set-nodeport.gif)
<!--
{{% tabs %}}
{{% tab "Migrated Workload" %}}
You can declare a NodePort for a workload that you're migrating by editing its Kubernetes manifest. In this sample, we've taken the same [web-deployment.yml]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/run-migration-tool/#migration-example-file-output) referenced in [Setting HostPorts](#setting-hostports). However, in this manifest, we've set a NodePort instead of a HostPort.
>**Note:**
>
>- If you set the `nodePort` directive without giving it a value, Rancher chooses a port at random from the following range: `30000-32767`.
>- If you manually set the `nodePort` directive, you must assign it a value within the `30000-32767` range.
<figcaption>HostPort Kubernetes manifest Example: <code>web-deployment.yaml</code></figcaption>
```YAML
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
io.rancher.container.pull_image: always
io.rancher.scheduler.global: "true"
kompose.cmd: ./migration-tools parse --docker-file docker-compose.yml --rancher-file
rancher-compose.yml
kompose.version: 1.16.0 ()
creationTimestamp: null
labels:
io.kompose.service: web
name: web
spec:
replicas: 0
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: web
spec:
containers:
- image: sdelements/lets-chat
name: web
ports:
- containerPort: 8080
nodePort: 32767 # NEW NODEPORT DECLARATION
resources: {}
stdin: true
tty: true
restartPolicy: Always
status: {}
```
{{% /tab %}}
{{% tab "New Workload" %}}
While creating or editing a workload using the Rancher v2.x UI, you may specify a NodePort in the **Port Mapping** section. This action adds a `nodePort` value to the `ports` section of the cluster's Kubernetes manifest automatically. For more information on using Rancher UI to set a NodePort for a workload, see [Deploying Workloads]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/workloads/deploy-workloads/).
<figcaption>Workload Deployment: Setting NodePort</figcaption>
![NodePort Configuration]({{< baseurl >}}/img/rancher/deploy-workload-nodeport.png)
{{% /tab %}}
{{% /tabs %}} -->
### [Next: Configure Health Checks]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/monitor-apps)
@@ -0,0 +1,97 @@
---
title: "1. Get Started"
weight: 25
---
Get started with your migration to Rancher v2.x by installing Rancher and configuring your new Rancher environment.
## Outline
<!-- TOC -->
- [A. Install Rancher v2.x](#a-install-rancher-v2-x)
- [B. Configure Authentication](#b-configure-authentication)
- [C. Provision a Cluster and Project](#c-provision-a-cluster-and-project)
- [D. Create Stacks](#d-create-stacks)
<!-- /TOC -->
## A. Install Rancher v2.x
The first step in migrating from v1.6 to v2.x is to install the Rancher v2.x Server side-by-side with your v1.6 Server, which you'll need during the migration process. Due the architecture changes between v1.6 and v2.x, there is no direct path for upgrade. You'll have to install v2.x independently and then migrate your v1.6 services to v2.x.
New for v2.x, all communication to Rancher Server is encrypted. The procedures below instructs you on how to create and install these certificates.
Before installing v2.x, provision one host or more to function as your Rancher Server(s). You can find the requirements for these hosts in [Server Requirements]({{< baseurl >}}/rancher/v2.x/en/installation/requirements/).
After provisioning your node(s), install Rancher:
- [Single Node Install]({{< baseurl >}}/rancher/v2.x/en/installation/single-node)
For development environments, we recommend a single node install. This installation procedure deploys a single Rancher container to your host.
- [High Availability Configuration]({{< baseurl >}}/rancher/v2.x/en/installation/ha/)
For production environments where your user base requires constant access to your cluster, we recommend installing Rancher in a high availability (HA) configuration. This installation procedure provisions a three-node cluster and installs Rancher on each node using a Helm chart.
>**Important Difference:** Although you could install Rancher v1.6 in an HA configuration using an external database and a Docker command on each node, Rancher v2.x in an HA configuration requires an existing Kubernetes cluster. Review [High Availability Install](https://rancher.com/docs/rancher/v2.x/en/installation/ha/) for full requirements.
## B. Configure Authentication
After Rancher Server is installed, we recommend configuring external authentication (like Active Directory or GitHub) so that users can log into Rancher using their single sign-on. For a full list of supported authentication providers and instructions on how to configure them, see [Authentication]({{< baseurl >}}/rancher/v2.x/en/admin-settings/authentication).
<figcaption>Rancher v2.x Authentication</figcaption>
![Rancher v2.x Authentication]({{< baseurl >}}/img/rancher/auth-providers.svg)
### Local Users
Although we recommend using an external authentication provider, Rancher v1.6 and v2.x both offer support for users local to Rancher. However, these users cannot be migrated from Rancher v1.6 to v2.x. If you used local users in Rancher v1.6 and want to continue this practice in v2.x, you'll need to [manually recreate these user accounts]({{< baseurl >}}/rancher/v2.x/en/admin-settings/authentication/) and assign them access rights.
As a best practice, you should use a hybrid of external _and_ local authentication. This practice provides access to Rancher should your external authentication experience an interruption, as you can still log in using a local user account. Set up a few local accounts as administrative users of Rancher.
### SAML Authentication Providers
In Rancher v1.6, we encouraged our SAML users to use Shibboleth, as it was the only SAML authentication option we offered. However, to better support their minor differences, we've added more fully tested SAML providers for v2.x: Ping Identity, Microsoft ADFS, and FreeIPA.
## C. Provision a Cluster and Project
Begin work in Rancher by provisioning a new Kubernetes cluster, which is similar to an environment in v1.6. This cluster will host your application deployments.
A cluster and project in combined together in Rancher v2.x equivalent to a v1.6 environment. A _cluster_ is the compute boundary (i.e., your hosts) and a _project_ is an administrative boundary (i.e., a grouping of namespaces used to assign access rights to users).
### Clusters
In Rancher v1.6, compute nodes were added to an _environment_. Rancher v2.x eschews the term _environment_ for _cluster_, as Kubernetes uses this term for a team of computers instead of _environment_.
Rancher v2.x lets you launch a Kubernetes cluster anywhere. Host your cluster using:
- A [hosted Kubernetes provider]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/).
- A [pool of nodes from an infrastructure provider]({{< baseurl >}}rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/). Rancher launches Kubernetes on the nodes.
- Any [custom node(s)]({{< baseurl >}}rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/). Rancher can launch Kubernetes on the nodes, be they bare metal servers, virtual machines, or cloud hosts on a less popular infrastructure provider.
### Projects
Additionally, Rancher v2.x introduces [projects]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/projects-and-namespaces/), which are objects that divide clusters into different application groups that are useful for applying user permissions. This model of clusters and projects allow for multi-tenancy because hosts are owned by the cluster, and the cluster can be further divided into multiple projects where users can manage their apps.
When you create a cluster, two projects are automatically created:
- The `system` project, which includes namespaces for each of your Kubernetes components.
- The `default` project, which is a "hello-world" project for your cluster.
However, for production environments, we recommend [creating your own project]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/projects-and-namespaces/#creating-projects) and giving it a descriptive name.
After provisioning a new cluster and project, you can authorize your users to access and use project resources. Similarly to Rancher v1.6 environments, Rancher v2.x allows you to [assign users to projects]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/projects-and-namespaces/editing-projects/). By assigning users to projects, you can limit what applications and resources a user can access.
## D. Create Stacks
In Rancher v1.6, _stacks_ were used to group together the services that belong to your application. In v2.x, you need to [create namespaces]({{< baseurl >}}rancher/v2.x/en/k8s-in-rancher/projects-and-namespaces/#creating-namespaces), which are the v2.x equivalent of stacks, for the same purpose.
In Rancher v2.x, namespaces are child objects to projects. When you create a project, a `default` namespace is added to the project, but you can create your own to parallel your stacks from v1.6.
During migration, if you don't explicitly define which namespace a service should be deployed to, it's deployed to the `default` namespace.
Just like v1.6, Rancher v2.x supports service discovery within and across namespaces (we'll get to [service discovery]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/discover-services) soon).
### [Next: Migrate Your Services]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/run-migration-tool)
@@ -0,0 +1,37 @@
---
title: Kubernetes Introduction
weight: 1
---
Rancher v2.x is built on the [Kubernetes](https://kubernetes.io/docs/home/?path=users&persona=app-developer&level=foundational) container orchestrator. This shift in underlying technology for v2.x is a large departure from v1.6, which supported several popular container orchestrators. Since Rancher is now based entirely on Kubernetes, it's helpful to learn the Kubernetes basics.
The following table introduces and defines some key Kubernetes concepts.
| **Concept** | **Definition** |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cluster | A collection of machines that run containerized applications managed by Kubernetes. |
| Namespace | A virtual cluster, multiple of which can be supported by a single physical cluster. |
| Node | One of the physical (or virtual) machines that make up a cluster. |
| Pod | The smallest and simplest Kubernetes object. A pod represents a set of running [containers](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/#why-containers) on your cluster. |
| Deployment | An API object that manages a replicated application. |
| Workload | Units of work that are running on the cluster, these can be pods or deployments. |
## Migration Cheatsheet
Because Rancher v1.6 defaulted to our Cattle container orchestrator, it primarily used terminology related to Cattle. However, because Rancher v2.x uses Kubernetes, it aligns with the Kubernetes naming standard. This shift could be confusing for people unfamiliar with Kubernetes, so we've created a table that maps terms commonly used in Rancher v1.6 to their equivalents in Rancher v2.x.
| **Rancher v1.6** | **Rancher v2.x** |
| --- | --- |
| Container | Pod |
| Services | Workload |
| Load Balancer | Ingress |
| Stack | Namespace |
| Environment | Project (Administration)/Cluster (Compute)
| Host | Node |
| Catalog | Helm |
| Port Mapping | HostPort (Single Node)/NodePort (All Nodes) |
<br/>
More detailed information on Kubernetes concepts can be found in the
[Kubernetes Concepts Documentation](https://kubernetes.io/docs/concepts/).
@@ -0,0 +1,199 @@
---
title: "7. Load Balancing"
weight: 700
---
If your applications are public-facing and draw significant traffic from the internet, you should place a load balancer in front of your cluster so that users can always access their apps without service interruption. Typically, you can fulfill a high volume of service requests by [horizontally scaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) your deployment, which spins up additional application containers as traffic ramps up. However, this technique requires routing that distributes traffic across your nodes efficiently. In cases where you need to accommodate public traffic that scales up and down, you'll need a load balancer.
As outlined in [its documentation](https://rancher.com/docs/rancher/v1.6/en/cattle/adding-load-balancers/), Rancher v1.6 provided rich support for load balancing using its own microservice powered by HAProxy, which supports HTTP, HTTPS, TCP hostname, and path-based routing. Most of these same features are available in v2.x. However, load balancers that you used with v1.6 cannot be migrated to v2.x. You'll have to manually recreate your v1.6 load balancer in v2.x.
If you encounter the `output.txt` text below after parsing your v1.6 Compose files to Kubernetes manifests, you'll have to resolve it by manually creating a load balancer in v2.x.
<figcaption><code>output.txt</code> Load Balancer Directive</figcaption>
![Resolve Load Balancer Directive]({{< baseurl >}}/img/rancher/resolve-load-balancer.png)
## In This Document
<!-- TOC -->
- [Load Balancing Protocol Options](#load-balancing-protocol-options)
- [Load Balancer Deployment](#load-balancer-deployment)
- [Load Balancing Architecture](#load-balancing-architecture)
- [Ingress Caveats](#ingress-caveats)
- [Deploying Ingress](#deploying-ingress)
- [Rancher v2.x Load Balancing Limitations](#rancher-v2x-load-balancing-limitations)
<!-- /TOC -->
## Load Balancing Protocol Options
<!--
<figcaption>Rancher v1.6 rancher-compose.yml Load Balancer Configuration</figcaption>
```
...
webLB:
scale: 1
start_on_create: true
lb_config:
certs: []
port_rules:
- hostname: myapp.chat.com
path: /login
priority: 1
protocol: http
service: web
source_port: 80
target_port: 8080
health_check:
healthy_threshold: 2
response_timeout: 2000
port: 42
unhealthy_threshold: 3
initializing_timeout: 60000
interval: 2000
reinitializing_timeout: 60000
```
-->
By default, Rancher v2.x replaces the v1.6 load balancer microservice with the native [Kubernetes Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/), which is backed by Nginx Ingress Controller for layer 7 load balancing. By default, Kubernetes Ingress only supports the HTTP and HTTPS protocols, not TCP. Load balancing is limited to these two protocols when using Ingress.
> **TCP Required?** Rancher v2.x still supports layer 4 load balancers, although you must create the load balancer using the cloud provider in which your Kubernetes cluster is deployed.
## Load Balancer Deployment
<!-- TODO: Add table, link to relevant sections -->
In Rancher v1.6, you could add port/service rules for configuring your HAProxy to load balance for target services. You could also configure the hostname/path-based routing rules.
Rancher v2.x offers similar functionality, but load balancing is instead handled by Ingress. An Ingress is a specification of rules that a controller component applies to your load balancer. The actual load balancer can run outside of your cluster or within it.
By default, Rancher v2.x deploys Nginx Ingress Controller and load balancer on clusters provisioned using RKE (Rancher's own Kubernetes installer) to process the Kubernetes Ingress rules. The Nginx Ingress Controller is installed by default only in clusters provisioned by RKE. Clusters provisioned by cloud providers like GKE have their own Ingress Controllers that configure the load balancer. For this document, our scope is limited to the RKE-installed NGINX Ingress Controller only, but you can read about cloud providers in our documentation <!-- TODO: link -->
RKE deploys Nginx Ingress Controller as a Kubernetes DaemonSet <!-- TODO: link -->, meaning that an NGINX instance is deployed on every node in the cluster. Nginx acts like an Ingress Controller listening to Ingress creation within your entire cluster, and it also configures itself as the load balancer to satisfy the Ingress rules. The DaemonSet is configured with hostNetwork to expose two ports: 80 and 443.
For more information Nginx Ingress Controller, their deployment as DaemonSets, deployment configuration options, see the [RKE documentation](https://rancher.com/docs/rke/v0.1.x/en/config-options/add-ons/ingress-controllers/).
## Load Balancing Architecture
Deployment of Ingress Controller in v2.x as a DaemonSet brings some architectural changes that v1.6 users should know about.
In Rancher v1.6 you could deploy a scalable load balancer service within your stack. If you had four hosts in your Cattle environment, you could deploy one load balancer service with a scale of two and point to your application by appending port 80 to your two host IP Addresses. You could also launch another load balancer on the remaining two hosts to balance a different service again using port 80 because your load balancer is using different host IP Addresses).
<!-- add comparison table-->
<figcaption>Rancher v1.6 Load Balancing Architecture</figcaption>
![Rancher v1.6 Load Balancing]({{< baseurl >}}/img/rancher/cattle-load-balancer.svg)
The Rancher v2.x Ingress Controller is a DaemonSet—it is globally deployed on all schedulable nodes to serve your entire Kubernetes Cluster. Therefore, when you program the Ingress rules, you must use a unique hostname and path to point to your workloads, as the load balancer node IP addresses and ports 80 and 443 are common access points for all workloads.
<figcaption>Rancher v2.x Load Balancing Architecture</figcaption>
![Rancher v2.x Load Balancing]({{< baseurl >}}/img/rancher/kubernetes-load-balancer.svg)
## Ingress Caveats
Although Rancher v2.x supports HTTP and HTTPS hostname and path-based load balancing, you must use unique host names and paths when configuring your workloads. This limitation derives from:
- Ingress confinement to ports 80 and 443 (i.e, the ports HTTP[S] uses for routing).
- The load balancer and the Ingress Controller is launched globally for the cluster as a DaemonSet.
> **TCP Required?** Rancher v2.x still supports TCP. See [TCP Load Balancing Options](#tcp-load-balancing-options) for workarounds.
## Deploying Ingress
You can launch a new load balancer to replace your load balancer from v1.6. Using the Rancher v2.x UI, browse to the applicable project and choose **Workloads** from the main menu. Then choose the **Load Balancing** tab and begin by clicking **Deploy**. During deployment, you can choose a target project or namespace.
>**Prerequisite:** Before deploying Ingress, you must have a workload deployed that's running a scale of two or more pods.
>
>![Workload Scale]({{< baseurl >}}/img/rancher/workload-scale.png)
For balancing between these two pods, you must create a Kubernetes Ingress rule. To create this rule, navigate to your cluster and project, and then select the **Load Balancing** tab. Then click **Add Ingress**. This GIF below depicts how to add Ingress to one of your projects.
<figcaption>Browsing to Load Balancer Tab and Adding Ingress</figcaption>
![Adding Ingress]({{< baseurl >}}/img/rancher/add-ingress.gif)
Similar to a service/port rules in Rancher v1.6, here you can specify rules targeting your workload's container port. The sections below demonstrate how to create Ingress rules.
### Configuring Host- and Path-Based Routing
Using Rancher v2.x, you can add Ingress rules that are based on host names or a URL path. Based on the rules you create, your Nginx Ingress Controller routes traffic to multiple target workloads or Kubernetes services.
For example, let's say you have multiple workloads deployed to a single namespace. You can add an Ingress to route traffic to these two workloads using the same hostname but different paths, as depicted in the image below. URL requests to `foo.com/name.html` will direct users to the `web` workload, and URL requests to `foo.com/login` will direct users to the `chat` workload.
<figcaption>Ingress: Path-Based Routing Configuration</figcaption>
![Ingress: Path-Based Routing Configuration]({{< baseurl >}}/img/rancher/add-ingress-form.png)
Rancher v2.x also places a convenient link to the workloads on the Ingress record. If you configure an external DNS to program the DNS records, this hostname can be mapped to the Kubernetes Ingress address.
<figcaption>Workload Links</figcaption>
![Load Balancer Links to Workloads]({{< baseurl >}}/img/rancher/load-balancer-links.png)
The Ingress address is the IP address in your cluster that the Ingress Controller allocates for your workload. You can reach your workload by browsing to this IP address. Use `kubectl` command below to see the Ingress address assigned by the controller:
```
kubectl get ingress
```
### HTTPS/Certificates Option
Rancher v2.x Ingress functionality supports the HTTPS protocol, but if you want to use it, you need to use a valid SSL/TLS certificate. While configuring Ingress rules, use the **SSL/TLS Certificates** section to configure a certificate.
- If you're configuring Ingress for an internal audience that knows it can safely ignore a certificate warning, using the **Use default ingress controller certificate** is easiest.
- If you're configuring Ingress for a public audience, we recommend [uploading a certificate]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/certificates/) from a known certificate authority (you'll have to do this before configuring Ingress). Then, while configuring your load balancer, use the **Choose a certificate** option and select the uploaded certificate that you want to use.
<figcaption>Load Balancer Configuration: SSL/TLS Certificate Section</figcaption>
![SSL/TLS Certificates Section]({{< baseurl >}}/img/rancher/load-balancer-ssl-certs.png)
### TCP Load Balancing Options
#### Layer-4 Load Balancer
For the TCP protocol, Rancher v2.x supports configuring a Layer 4 load balancer using the cloud provider in which your Kubernetes cluster is deployed. Once this load balancer appliance is configured for your cluster, when you choose the option of a `Layer-4 Load Balancer` for port-mapping during workload deployment, Rancher automatically creates a corresponding load balancer service. This service will call the corresponding cloud provider and configure the load balancer appliance to route requests to the appropriate pods. See [Cloud Providers](https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/rke-clusters/options/cloud-providers/) for information on how to configure LoadBalancer services for your cloud provider.
For example, if we create a deployment named `myapp` and specify a Layer 4 load balancer in the **Port Mapping** section, Rancher will automatically add an entry to the **Load Balancer** tab named `myapp-loadbalancer`.
<figcaption>Workload Deployment: Layer 4 Load Balancer Creation</figcaption>
![Deploy Layer-4 Load Balancer]({{< baseurl >}}/img/rancher/deploy-workload-load-balancer.png)
Once configuration of the load balancer succeeds, the Rancher UI provides a link to your workload's public endpoint.
#### Nginx Ingress Controller TCP Support by ConfigMaps
Although Nginx supports TCP, Kubernetes Ingress itself does not support the TCP protocol. Therefore, out-of-the-box configuration of Nginx Ingress Controller for TCP balancing isn't possible.
However, there is a workaround to use Nginx's TCP balancing by creating a Kubernetes ConfigMap, as described in the [Ingress GitHub readme](https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/exposing-tcp-udp-services.md). You can create a ConfigMap object that stores pod configuration parameters as key-value pairs, separate from the pod image, as described in the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/).
To configure Nginx to expose your services via TCP, you can add the ConfigMap `tcp-services` that should exist in the `ingress-nginx` namespace. This namespace also contains the Nginx Ingress Controller pods.
![Layer-4 Load Balancer: ConfigMap Workaround]({{< baseurl >}}/img/rancher/layer-4-lb-config-map.png)
The key in the ConfigMap entry should be the TCP port that you want to expose for public access: `<namespace/service name>:<service port>`. As shown above, two workloads are listed in the `Default` namespace. For example, the first entry in the ConfigMap above instructs NGINX to expose the `myapp` workload (the one in the `default` namespace that's listening on private port 80) over external port `6790`. Adding these entries to the ConfigMap automatically updates the Nginx pods to configure these workloads for TCP balancing. The workloads exposed should be available at `<NodeIP>:<TCP Port>`. If they are not accessible, you might have to expose the TCP port explicitly using a NodePort service.
## Rancher v2.x Load Balancing Limitations
Cattle provided feature-rich load balancer support that is [well documented](https://rancher.com/docs/rancher/v1.6/en/cattle/adding-load-balancers/#load-balancers). Some of these features do not have equivalents in Rancher v2.x. This is the list of such features:
- No support for SNI in current Nginx Ingress Controller.
- TCP load balancing requires a load balancer appliance enabled by cloud provider within the cluster. There is no Ingress support for TCP on Kubernetes.
- Only ports 80 and 443 can be configured for HTTP/HTTPS routing via Ingress. Also Ingress Controller is deployed globally as a DaemonSet and not launched as a scalable service. Also, users cannot assign random external ports to be used for balancing. Therefore, users need to ensure that they configure unique hostname/path combinations to avoid routing conflicts using the same two ports.
- There is no way to specify port rule priority and ordering.
- Rancher v1.6 added support for draining backend connections and specifying a drain timeout. This is not supported in Rancher v2.x.
- There is no support for specifying a custom stickiness policy and a custom load balancer config to be appended to the default config as of now in Rancher v2.x. There is some support, however, available in native Kubernetes for customizing the Nginx configuration as noted in the [NGINX Ingress Controller Custom Conguration Documentation](https://kubernetes.github.io/ingress-nginx/examples/customization/custom-configuration/).
### Finished!
@@ -0,0 +1,211 @@
---
title: "4. Configure Health Checks"
weight: 400
---
Rancher v1.6 provided TCP and HTTP health checks on your nodes and services using its own health check microservice. These health checks monitored your containers to confirm they're operating as intended. If a container failed a health check, Rancher would destroy the unhealthy container and then replicates a healthy one to replace it.
For Rancher v2.x, we've replaced the heath check microservice, leveraging instead Kubernete's native health check support.
Use this document to correct Rancher v2.x workloads and services that list `health_check` in `output.txt`. You can correct them by configuring a liveness probe (i.e., a health check).
For example, for the image below, we would configure liveness probes for the `web` and `weblb` workloads (i.e., the Kubernetes manifests output by migration-tools CLI).
<figcaption>Resolve <code>health_check</code> for the <code>web</code> and <code>webLB</code> Workloads</figcaption>
![Resolve health_check]({{< baseurl >}}/img/rancher/resolve-health-checks.png)
## In This Document
<!-- TOC -->
- [Rancher v1.6 Health Checks](#rancher-v16-health-checks)
- [Rancher v2.x Health Checks](#rancher-v2x-health-checks)
- [Configuring Probes in Rancher v2.x](#configuring-probes-in-rancher-v2x)
<!-- /TOC -->
## Rancher v1.6 Health Checks
In Rancher v1.6, you could add health checks to monitor a particular service's operations. These checks were performed by the Rancher health check microservice, which is launched in a container on a node separate from the node hosting the monitored service (however, Rancher v1.6.20 and later also runs a local health check container as a redundancy for the primary health check container on another node). Health check settings were stored in the `rancher-compose.yml` file for your stack.
The health check microservice features two types of health checks, which have a variety of options for timeout, check interval, etc.:
- **TCP health checks**:
These health checks check if a TCP connection opens at the specified port for the monitored service. For full details, see the [Rancher v1.6 documentation](https://rancher.com/docs/rancher/v1.6/en/cattle/health-checks/).
- **HTTP health checks**:
These health checks monitor HTTP requests to a specified path and check whether the response is expected response (which is configured along with the health check).
The following diagram displays the health check microservice evaluating a container running Nginx. Notice that the microservice is making its check across nodes.
![Rancher v1.6 Health Checks]({{< baseurl >}}/img/rancher/healthcheck.svg)
## Rancher v2.x Health Checks
In Rancher v2.x, the health check microservice is replaced with Kubernete's native health check mechanisms, called _probes_. These probes, similar to the Rancher v1.6 health check microservice, monitor the health of pods over TCP and HTTP.
However, probes in Rancher v2.x have some important differences, which are described below. For full details about probes, see the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes).
### Local Health Checks
Unlike the Rancher v1.6 health checks performed across hosts, probes in Rancher v2.x occur on _same_ host, performed by the kubelet.
### Multiple Probe Types
Kubernetes includes two different _types_ of probes: liveness checks and readiness checks.
- **Liveness Check**:
Checks if the monitored container is running. If the probe reports failure, Kubernetes kills the pod, and then restarts it according to the deployment [restart policy](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy).
- **Readiness Check**:
Checks if the container is ready to accept and serve requests. If the probe reports failure, the pod is sequestered from the public until it self heals.
The following diagram displays kubelets running probes on containers they are monitoring ([kubelets](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) are the primary "agent" running on each node). The node on the left is running a liveness probe, while the one of the right is running a readiness check. Notice that the kubelet is scanning containers on its host node rather than across nodes, as in Rancher v1.6.
![Rancher v2.x Probes]({{< baseurl >}}/img/rancher/probes.svg)
## Configuring Probes in Rancher v2.x
The [migration-tool CLI]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/run-migration-tool/) cannot parse health checks from Compose files to Kubernetes manifest. Therefore, if want you to add health checks to your Rancher v2.x workloads, you'll have to add them manually.
Using the Rancher v2.x UI, you can add TCP or HTTP health checks to Kubernetes workloads. By default, Rancher asks you to configure a readiness check for your workloads and applies a liveness check using the same configuration. Optionally, you can define a separate liveness check.
If the probe fails, the container is restarted per the restartPolicy defined in the workload specs. This setting is equivalent to the strategy parameter for health checks in Rancher v1.6.
Configure probes by using the **Health Check** section while editing deployments called out in `output.txt`.
<figcaption>Edit Deployment: Health Check Section</figcaption>
![Health Check Section]({{< baseurl >}}/img/rancher/health-check-section.png)
### Configuring Readiness Checks
While you create a workload using Rancher v2.x, we recommend configuring a readiness check that monitors the health of the deployment's pods.
{{% tabs %}}
{{% tab "TCP Readiness Check" %}}
TCP readiness checks monitor your deployment's health by attempting to open a connection to the pod over a specified port. If the probe can open the port, it's considered healthy. Failure to open it is considered unhealthy, which notifies Kubernetes that it should kill the pod and then replace it according to its [restart policy](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy).
You can configure the probe along with values for specifying its behavior by selecting the **TCP connection opens successfully** option in the **Health Check** section. For more information, see [Deploying Workloads]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/workloads/deploy-workloads/). For help setting probe timeout and threshold values, see [Health Check Parameter Mappings](#health-check-parameter-mappings).
![TCP Readiness Check]({{< baseurl >}}/img/rancher/readiness-check-tcp.png)
When you configure a readiness check using Rancher v2.x, the `readinessProbe` directive and the values you've set are added to the deployment's Kubernetes manifest. Configuring a readiness check also automatically adds a liveness check to the deployment.
<!--
```YAML
...
- image: nginx
imagePullPolicy: Always
readinessProbe: # ADDED DIRECTIVE
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 2
successThreshold: 1
tcpSocket:
port: 80
timeoutSeconds: 2
livenessProbe: # ADDED DIRECTIVE
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 2
successThreshold: 1
tcpSocket:
port: 80
timeoutSeconds: 2
```
-->
{{% /tab %}}
{{% tab "HTTP Readiness Check" %}}
HTTP readiness checks monitor your deployment's health by sending an HTTP GET request to a specific URL path that you define. If the pod responds with a message range of `200`-`400`, the health check is considered successful. If the pod replies with any other value, the check is considered unsuccessful, so Kubernetes kills and replaces the pod according to its [restart policy](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy).
You can configure the probe along with values for specifying its behavior by selecting the **HTTP returns successful status** or **HTTPS returns successful status**. For more information, see [Deploying Workloads]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/workloads/deploy-workloads/). For help setting probe timeout and threshold values, see [Health Check Parameter Mappings](#healthcheck-parameter-mappings).
![HTTP Readiness Check]({{< baseurl >}}/img/rancher/readiness-check-http.png)
When you configure a readiness check using Rancher v2.x, the `readinessProbe` directive and the values you've set are added to the deployment's Kubernetes manifest. Configuring a readiness check also automatically adds a liveness check to the deployment.
<!--
, as shown below.
```YAML
...
- image: nginx
imagePullPolicy: Always
readinessProbe: # ADDED DIRECTIVE
failureThreshold: 3
httpGet:
path: /index.html
port: 80
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 2
livenessProbe: # ADDED DIRECTIVE
failureThreshold: 3
httpGet:
path: /index.html
port: 80
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 2
successThreshold: 2
timeoutSeconds: 2
```
-->
{{% /tab %}}
{{% /tabs %}}
### Configuring Separate Liveness Checks
While configuring a readiness check for either the TCP or HTTP protocol, you can configure a separate liveness check by clicking the **Define a separate liveness check**. For help setting probe timeout and threshold values, see [Health Check Parameter Mappings](#health-check-parameter-mappings).
![Separate Liveness Check]({{< baseurl >}}/img/rancher/separate-check.png)
### Additional Probing Options
Rancher v2.x, like v1.6, lets you perform health checks using the TCP and HTTP protocols. However, Rancher v2.x also lets you check the health of a pod by running a command inside of it. If the container exits with a code of `0` after running the command, the pod is considered healthy.
You can configure a liveness or readiness check that executes a command that you specify by selecting the `Command run inside the container exits with status 0` option from **Health Checks** while [deploying a workload]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/workloads/deploy-workloads/).
![Healthcheck Execute Command]({{< baseurl >}}/img/rancher/healthcheck-cmd-exec.png)
#### Health Check Parameter Mappings
While configuring readiness checks and liveness checks, Rancher prompts you to fill in various timeout and threshold values that determine whether the probe is a success or failure. The reference table below shows you the equivalent health check values from Rancher v1.6.
Rancher v1.6 Compose Parameter | Rancher v2.x Kubernetes Parameter
-------------------------------|-----------------------------------
`port` | `tcpSocket.port`
`response_timeout` | `timeoutSeconds`
`healthy_threshold` | `failureThreshold`
`unhealthy_threshold` | `successThreshold`
`interval` | `periodSeconds`
`initializing_timeout` | `initialDelaySeconds`
`strategy` | `restartPolicy`
### [Next: Schedule Workloads]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/schedule-workloads/)
@@ -0,0 +1,316 @@
---
title: 2. Migrate Your Services
weight: 100
---
Although your services from v1.6 won't work in Rancher v2.x by default, that doesn't mean you have to start again from square one, manually rebuilding your applications in v2.x. To help with migration from v1.6 to v2.x, Rancher has developed a migration-tool. The migration-tools CLI is a utility that helps you recreate your applications in Rancher v2.x. This tool exports your Rancher v1.6 services as Compose files and converts them to a Kubernetes manifest that Rancher v2.x can consume.
Additionally, for each Rancher v1.6-specific Compose directive that cannot be consumed by Kubernetes, migration-tools CLI provides instructions on how to manually recreate them in Rancher v2.x.
This command line interface tool:
- Exports Compose files (i.e., `docker-compose.yml` and `rancher-compose.yml`) for each stack in your v1.6 Cattle environment. For every stack, files are exported to a unique folder: `<EXPORT_DIR>/<ENV_NAME>/<STACK_NAME>`.
- Parses Compose files that youve exported from your Rancher v1.6 stacks and converts them to Kubernetes manifests that Rancher v2.x can consume. The tool also outputs a list of directives present in the Compose files that cannot be converted automatically to Rancher v2.x. These are directives that youll have to manually configure using the Rancher v2.x UI.
It then outputs a list of directives present in the Compose files that are unsupported in Rancher v2.x. These directives require special handling or are parameters that cannot be converted to Kubernetes manifest.
## Outline
<!-- TOC -->
- [A. Download the Migration-Tools CLI](#a-download-the-migration-tools-cli)
- [B. Configure the Migration-Tools CLI](#b-configure-the-migration-tools-cli)
- [C. Run the Migration-Tools CLI](#c-run-the-migration-tools-cli)
- [D. Deploy Services Using Rancher CLI](#d-deploy-services-using-rancher-cli)
- [What Now?](#what-now)
<!-- /TOC -->
## A. Download the Migration-Tools CLI
The migration-tools CLI for your platform can be downloaded from our [GitHub releases page](https://github.com/rancher/migration-tools/releases). The tools are available for Linux, Mac, and Windows platforms.
## B. Configure the Migration-Tools CLI
After you download migration-tools CLI, rename it and make it executable.
1. Open Terminal and change to the directory that contains the migration-tool file.
1. Rename the file to `migration-tools` so that it no longer includes the platform name.
1. Enter the following command to make `migration-tools` an executable:
```
chmod +x migration-tools
```
## C. Run the Migration-Tools CLI
Next, use the migration-tools CLI to export all stacks in all of the Cattle environments into Compose files. Then, for stacks that you want to migrate to Rancher v2.x, convert the Compose files into Kubernetes manifest.
>**Prerequisite:** Create an [Account API Key](https://rancher.com/docs/rancher/v1.6/en/api/v2-beta/api-keys/#account-api-keys) to authenticate with Rancher v1.6 when using the Migration-Tools CLI.
1. Export the Docker Compose files for your Cattle environments and stacks from Rancher v1.6.
From Terminal, execute the following command, replacing each placeholder with your values.
```
migration-tools export --url http://<RANCHER_URL:PORT> --access-key <RANCHER_ACCESS_KEY> --secret-key <RANCHER_SECRET_KEY> --export-dir <EXPORT_DIR> --all
```
**Step Result:** migration-tools exports Compose files (`docker-compose.yml` and `rancher-compose.yml`) for each in the `--export-dir` directory. If you omitted this option, Compose files are output to your current directory.
A unique directory is created for each environment and stack. For example, if we export each [environment/stack]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/#migration-example-files) from Rancher v1.6, the following directory structure is created:
```
export/ # migration-tools --export-dir
|--<ENVIRONMENT>/ # Rancher v1.6 ENVIRONMENT
|--<STACK>/ # Rancher v1.6 STACK
|--docker-compose.yml # STANDARD DOCKER DIRECTIVES FOR ALL STACK SERVICES
|--rancher-compose.yml # RANCHER-SPECIFIC DIRECTIVES FOR ALL STACK SERVICES
|--README.md # README OF CHANGES FROM v1.6 to v2.x
```
1. Convert the exported Compose files to Kubernetes manifest.
Execute the following command, replacing each placeholder with the absolute path to your Stack's Compose files. If you want to migrate multiple stacks, you'll have to re-run the command for each pair of Compose files that you exported.
```
migration-tools parse --docker-file <DOCKER_COMPOSE_ABSOLUTE_PATH> --rancher-file <RANCHER_COMPOSE_ABSOLUTE_PATH>
```
>**Note:** If you omit the `--docker-file` and `--rancher-file` options from your command, migration-tools checks its home directory for Compose files.
>**Want full usage and options for the migration-tools CLI?** See the [Migration Tools CLI Reference]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/migration-tools-ref/).
### Migration-Tools CLI Output
After you run the migration-tools parse command, the following files are output to your target directory.
| Output | Description |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `output.txt` | This file lists how to recreate your Rancher v1.6-specific functionality in Kubernetes. Each listing links to the relevant blog articles on how to implement it in Rancher v2.x. |
| Kubernetes manifest specs | Migration-tools internally invokes [Kompose](https://github.com/kubernetes/kompose) to generate a Kubernetes manifest for each service you're migrating to v2.x. Each YAML spec file is named for the service you're migrating.
#### Why are There Separate Deployment and Service Manifests?
To make an application publicly accessible by URL, a Kubernetes service is required in support of the deployment. A Kubernetes service is a REST object that abstracts access to the pods in the workload. In other words, a service provides a static endpoint to the pods by mapping a URL to pod(s) Therefore, even if the pods change IP address, the public endpoint remains unchanged. A service object points to its corresponding deployment (workload) by using selector labels.
When a you export a service from Rancher v1.6 that exposes public ports, migration-tools CLI parses those ports to a Kubernetes service spec that links to a deployment YAML spec.
#### Migration Example File Output
If we parse the two example files from [Migration Example Files]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/#migration-example-files), `docker-compose.yml` and `rancher-compose.yml`, the following files are output:
File | Description
-----|------------
`web-deployment.yaml` | A file containing Kubernetes container specs for a Let's Chat deployment.
`web-service.yaml` | A file containing specs for the Let's Chat service.
`database-deployment.yaml` | A file containing container specs for the MongoDB deployment in support of Let's Chat.
`webLB-deployment.yaml` | A file containing container specs for an HAProxy deployment that's serving as a load balancer.<sup>1</sup>
`webLB-service.yaml` | A file containing specs for the HAProxy service.<sup>1</sup>
><sup>1</sup> Because Rancher v2.x uses Ingress for load balancing, we won't be migrating our Rancher v1.6 load balancer to v2.x.
<!--
The following tabs display the contents of each parsed file. We've omitted `webLB-deployment.yaml` and `webLB-service.yaml` because we aren't migrating them to v2.x.
{{% tabs %}}
{{% tab "web-deployment.yaml" %}}
```YAML
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
io.rancher.container.pull_image: always
io.rancher.scheduler.global: "true"
kompose.cmd: ./migration-tools parse --docker-file docker-compose.yml --rancher-file
rancher-compose.yml
kompose.version: 1.16.0 ()
creationTimestamp: null
labels:
io.kompose.service: web
name: web
spec:
replicas: 0
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: web
spec:
containers:
- image: sdelements/lets-chat
name: web
ports:
- containerPort: 8080
resources: {}
stdin: true
tty: true
restartPolicy: Always
status: {}
```
{{% /tab %}}
{{% tab "web-service.yaml" %}}
```YAML
apiVersion: v1
kind: Service
metadata:
annotations:
io.rancher.container.pull_image: always
io.rancher.scheduler.global: "true"
kompose.cmd: ./migration-tools parse --docker-file docker-compose.yml --rancher-file
rancher-compose.yml
kompose.version: 1.16.0 ()
creationTimestamp: null
labels:
io.kompose.service: web
name: web
spec:
ports:
- name: "9890"
port: 9890
targetPort: 8080
selector:
io.kompose.service: web
status:
loadBalancer: {}
```
{{% /tab %}}
{{% tab "database-deployment.yaml" %}}
```YAML
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
io.rancher.container.pull_image: always
io.rancher.scheduler.affinity:host_label_soft: db=true
kompose.cmd: ./migration-tools parse --docker-file docker-compose.yml --rancher-file
rancher-compose.yml
kompose.version: 1.16.0 ()
creationTimestamp: null
labels:
io.kompose.service: database
name: database
spec:
replicas: 0
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: database
spec:
containers:
- image: mongo
name: database
resources: {}
stdin: true
tty: true
restartPolicy: Always
status: {}
```
{{% /tab %}}
{{% /tabs %}}
-->
## D. Re-Deploy Services as Kubernetes Manifests
>**Note:** Although these instructions deploy your v1.6 services in Rancher v2.x, they will not work correctly until you adjust their Kubernetes manifests.
{{% tabs %}}
{{% tab "Rancher UI" %}}
You can deploy the Kubernetes manifests created by migration-tools by importing them into Rancher v2.x.
<figcaption>Deploy Services: Import Kubernetes Manifest</figcaption>
>**Receiving an `ImportYaml Error`?**
>
>Delete the YAML directive listed in the error message. These are YAML directives from your v1.6 services that Kubernetes can't read.
![Deploy Services]({{< baseurl >}}/img/rancher/deploy-service.gif)
{{% /tab %}}
{{% tab "Rancher CLI" %}}
>**Prerequisite:** [Install Rancher CLI]({{< baseurl >}}/rancher/v2.x/en/cli/) for Rancher v2.x.
Use the following Rancher CLI commands to deploy your application using Rancher v2.x. For each Kubernetes manifest output by migration-tools CLI, enter one of the commands below to import it into Rancher v2.x.
```
./rancher kubectl create -f <DEPLOYMENT_YAML_FILE> # DEPLOY THE DEPLOYMENT YAML
./rancher kubectl create -f <SERVICE_YAML_FILE> # DEPLOY THE SERVICE YAML
```
{{% /tab %}}
{{% /tabs %}}
Following importation, you can view your v1.6 services in the v2.x UI as Kubernetes manifests by using the context menu to select `<CLUSTER> > <PROJECT>` that contains your services. The imported manifests will display on the **Workloads** and **Service Discovery** tabs.
<figcaption>Imported Services</figcaption>
![Imported Services]({{< baseurl >}}/img/rancher/imported-workloads.png)
## What Now?
Although the migration-tool CLI parses your Rancher v1.6 Compose files to Kubernetes manifests, there are discrepancies between v1.6 and v2.x that you must address by manually editing your parsed [Kubernetes manifests](#output). In other words, you need to edit each workload and service imported into Rancher v2.x, as displayed below.
<figcaption>Edit Migrated Services</figcaption>
![Edit Migrated Workload]({{< baseurl >}}/img/rancher/edit-migration-workload.gif)
As mentioned in [Migration Tools CLI Output](#migration-tools-cli-output), the `output.txt` files generated during parsing lists the manual steps you must make for each deployment. Review the upcoming topics for more information on manually editing your Kubernetes specs.
Open your `output.txt` file and take a look at its contents. When you parsed your Compose files into Kubernetes manifests, migration-tools CLI output a manifest for each workload that it creates for Kubernetes. For example, our when our [Migration Example Files]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/#migration-example-files) are parsed into Kubernetes manifests, `output.txt` lists each resultant parsed [Kubernetes manifest file](#migration-example-file-output) (i.e., workloads). Each workload features a list of action items to restore operations for the workload in v2.x.
<figcaption>Output.txt Example</figcaption>
![output.txt]({{< baseurl >}}/img/rancher/output-dot-text.png)
The following table lists possible directives that may appear in `output.txt`, what they mean, and links on how to resolve them.
Directive | Instructions
----------|--------------
[ports][4] | Rancher v1.6 _Port Mappings_ cannot be migrated to v2.x. Instead, you must manually declare either a HostPort or NodePort, which are similar to Port Mappings.
[health_check][1] | The Rancher v1.6 health check microservice has been replaced with native Kubernetes health checks, called _probes_. Recreate your v1.6 health checks in v2.0 using probes.
[labels][2] | Rancher v1.6 uses labels to implement a variety of features in v1.6. In v2.x, Kubernetes uses different mechanisms to implement these features. Click through on the links here for instructions on how to address each label.<br/><br/>[io.rancher.container.pull_image][7]: In v1.6, this label instructed deployed containers to pull a new version of the image upon restart. In v2.x, this functionality is replaced by the `imagePullPolicy` directive.<br/><br/>[io.rancher.scheduler.global][8]: In v1.6, this label scheduled a container replica on every cluster host. In v2.x, this functionality is replaced by daemonsets.<br/><br/>[io.rancher.scheduler.affinity][9]: In v2.x, affinity is applied in a different way.
[links][3] | During migration, you must create links between your Kubernetes workloads and services for them to function properly in v2.x.
[scale][5] | In v1.6, scale refers to the number of container replicas running on a single node. In v2.x, this feature is replaced by replica sets.
start_on_create | No Kubernetes equivalent. No action is required from you.
[1]:{{< baseurl >}}/rancher/v2.x/en/v1.6-migration/monitor-apps/#configuring-probes-in-rancher-v2-x
[2]:{{< baseurl >}}/rancher/v2.x/en/v1.6-migration/schedule-workloads/#scheduling-using-labels
[3]:{{< baseurl >}}/rancher/v2.x/en/v1.6-migration/discover-services
[4]:{{< baseurl >}}/rancher/v2.x/en/v1.6-migration/expose-services
[5]:{{< baseurl >}}/rancher/v2.x/en/v1.6-migration/schedule-workloads/#scheduling-pods-to-a-specific-node
<!-- MB: oops, skipped 6 -->
[7]:{{< baseurl >}}/rancher/v2.x/en/v1.6-migration/schedule-workloads/#scheduling-using-labels
[8]:{{< baseurl >}}/rancher/v2.x/en/v1.6-migration/schedule-workloads/#scheduling-global-services
[9]:{{< baseurl >}}/rancher/v2.x/en/v1.6-migration/schedule-workloads/#label-affinity-antiaffinity
### [Next: Expose Your Services]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/expose-services/)
@@ -56,15 +56,15 @@ migration-tools export --url <RANCHER_URL> --access-key <RANCHER_ACCESS_KEY> --s
### Migration-Tools Parse Reference
The `migration-tools parse` command parses the Compose files for a stack and uses [Kompose](https://github.com/kubernetes/kompose) to generate an equivalent Kubernetes YAML. It also outputs an `output.txt` file, which lists all the constructs that will need manual intervention in order to be converted to Kubernetes.
The `migration-tools parse` command parses the Compose files for a stack and uses [Kompose](https://github.com/kubernetes/kompose) to generate an equivalent Kubernetes manifest. It also outputs an `output.txt` file, which lists all the constructs that will need manual intervention in order to be converted to Kubernetes.
#### Options
| Option | Required? | Description
| ---|---|---
|`--docker-file <VALUE>` | | Parses Docker Compose file to output Kubernetes manifest (default: `docker-compose.yml`)
|`--docker-file <VALUE>` | | Parses Docker Compose file to output Kubernetes manifest(default: `docker-compose.yml`)
|`--output-file <VALUE>` | | Name of file that outputs listing checks and advice for conversion (default: `output.txt`).
|`--rancher-file <VALUE>` | | Parses Rancher Compose file to output Kubernetes manifest (default: `rancher-compose.yml`)
|`--rancher-file <VALUE>` | | Parses Rancher Compose file to output Kubernetes manifest(default: `rancher-compose.yml`)
#### Subcommands
@@ -82,4 +82,4 @@ migration-tools parse --docker-file <DOCKER_COMPOSE_ABSOLUTE_PATH> --rancher-fil
>**Note:** If you omit the `--docker-file` and `--rancher-file` options from your command, the migration-tools CLI checks its home directory for these Compose files.
**Result:** The migration-tools CLI parses your Compose files and outputs Kubernetes YAML specs as well as an `output.txt` file. For each service in the stack, a YAML spec file is created and named the same as your service. The `output.txt` file lists all constructs for each service in `docker-compose.yml` that requires special handling to be successfully migrated to Rancher 2.x. Each construct links to the relevant blog articles on how to implement it in Rancher 2.x.
**Result:** The migration-tools CLI parses your Compose files and outputs Kubernetes manifest specs as well as an `output.txt` file. For each service in the stack, a Kubernetes manifest is created and named the same as your service. The `output.txt` file lists all constructs for each service in `docker-compose.yml` that requires special handling to be successfully migrated to Rancher v2.x. Each construct links to the relevant blog articles on how to implement it in Rancher v2.x.
@@ -0,0 +1,363 @@
---
title: "5. Schedule Your Services"
weight: 500
---
In v1.6, objects called _services_ were used to schedule containers to your cluster hosts. Services included the Docker image for an application, along with configuration settings for a desired state.
In Rancher v2.x, the equivalent object is known as a _workload_. Rancher v2.x retains all scheduling functionality from v1.6, but because of the change from Cattle to Kubernetes as the default container orchestrator, the terminology and mechanisms for scheduling workloads has changed.
Workload deployment is one of the more important and complex aspects of container orchestration. Deploying pods to available shared cluster resources helps maximize performance under optimum compute resource use.
You can schedule your migrated v1.6 services while editing a deployment. Schedule services by using **Workload Type** and **Node Scheduling** sections, which are shown below.
<figcaption>Editing Workloads: Workload Type and Node Scheduling Sections</figcaption>
![Workload Type and Node Scheduling Sections]({{< baseurl >}}/img/rancher/migrate-schedule-workloads.png)
## In This Document
<!-- NEED DOCS ABOUT CHANGING DEPLOYMENTS TO DAEMONSETS -->
<!-- TOC -->
- [What's Different for Scheduling Services?](#whats-different-for-scheduling-services)
- [Node Scheduling Options](#node-scheduling-options)
- [Scheduling Pods to a Specific Node](#scheduling-pods-to-a-specific-node)
- [Scheduling Using Labels](#scheduling-using-labels)
- [Scheduling Pods Using Resource Constraints](#scheduling-pods-using-resource-constraints)
- [Scheduling Specific Services to Specific Nodes](#scheduling-specific-services-to-specific-nodes)
- [Scheduling Global Services](#scheduling-global-services)
<!-- /TOC -->
## What's Different for Scheduling Services?
Rancher v2.x retains _all_ methods available in v1.6 for scheduling your services. However, because the default container orchestration system has changed from Cattle to Kubernetes, the terminology and implementation for each scheduling option has changed.
In v1.6, you would schedule a service to a host while adding a service to a Stack. In Rancher v2.x., the equivalent action is to schedule a workload for deployment. The following composite image shows a comparison of the UI used for scheduling in Rancher v2.x versus v1.6.
![Node Scheduling: Rancher v2.x vs v1.6]({{< baseurl >}}/img/rancher/node-scheduling.png)
## Node Scheduling Options
Rancher offers a variety of options when scheduling nodes to host workload pods (i.e., scheduling hosts for containers in Rancher v1.6).
You can choose a scheduling option as you deploy a workload. The term _workload_ is synonymous with adding a service to a Stack in Rancher v1.6). You can deploy a workload by using the context menu to browse to a cluster project (`<CLUSTER> > <PROJECT> > Workloads`).
The sections that follow provide information on using each scheduling options, as well as any notable changes from Rancher v1.6. For full instructions on deploying a workload in Rancher v2.x beyond just scheduling options, see [Deploying Workloads]({{< baseurl >}}/rancher/v2.x/en/k8s-in-rancher/workloads/deploy-workloads/).
Option | v1.6 Feature | v2.x Feature
-------|------|------
[Schedule pods by scale?](#scheduling-pods-by-scale) | ✓ | ✓
[Schedule pods to specific node?](#scheduling-pods-to-a-specific-node) | ✓ | ✓
[Schedule to nodes using labels?](#scheduling-pods-using-labels) | ✓ | ✓
[Schedule to nodes using label affinity/anti-affinity rules?](#scheduling-pods-using-affinityanti-affinity-label) | ✓ | ✓
[Schedule based on resource constraints?](#scheduling-pods-using-resource-constraints) | ✓ | ✓
[Schedule specific services to specific hosts?](#scheduling-specific-services-to-specific-nodes) | ✓ | ✓
[Schedule services globally?](#scheduling-global-services) | ✓ | ✓
### Scheduling Pods by Scale
In v1.6, you could control the number of container replicas deployed for a service. You can schedule pods the same way in v2.x, but you'll have to set the scale manually while editing a workload.
![Resolve Scale]({{< baseurl >}}/img/rancher/resolve-scale.png)
During migration, you can resolve `scale` entries in `output.txt` by setting a value for the **Workload Type** option **Scalable deployment** depicted below.
<figcaption>Scalable Deployment Option</figcaption>
![Workload Scale]({{< baseurl >}}/img/rancher/workload-type-option.png)
### Scheduling Pods to a Specific Node
Just as you could schedule containers to a single host in Rancher v1.6, you can schedule pods to single node in Rancher v2.x
As you deploy a workload, use the **Node Scheduling** section to choose a node to run your pods on. The workload below is being scheduled to deploy an Nginx image with a scale of two pods on a specific node.
<!-- Question: What would be a good use case for use of a scheduling pods on the same node?-->
<figcaption>Rancher v2.x: Workload Deployment</figcaption>
![Workload Tab and Group by Node Icon]({{< baseurl >}}/img/rancher/schedule-specific-node.png)
Rancher schedules pods to the node you select if 1) there are compute resource available for the node and 2) you've configured port mapping to use the HostPort option, that there are no port conflicts.
If you expose the workload using a NodePort that conflicts with another workload, the deployment gets created successfully, but no NodePort service is created. Therefore, the workload isn't exposed outside of the cluster.
After the workload is created, you can confirm that the pods are scheduled to your chosen node. From the **Workloads** tab, click the **Group by Node** icon to sort your workloads by node. Note that both Nginx pods are scheduled to the same node.
![Pods Scheduled to Same Node]({{< baseurl >}}/img/rancher/scheduled-nodes.png)
<!--
If you export the workload's manifest for Rancher v2.x, you can see in the pod spec that the workload is scheduled to the node that you selected (`nodeName: mark-do1`).
<figcaption>Kubernetes manifest: All Pods Scheduled to Single Node</figcaption>
```YAML
...
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: nginx
ports:
- containerPort: 80
name: 80tcp01
protocol: TCP
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities: {}
privileged: false
readOnlyRootFilesystem: false
runAsNonRoot: false
stdin: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
tty: true
dnsPolicy: ClusterFirst
nodeName: mark-do1 # Scheduled Node
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
...
```
-->
### Scheduling Using Labels
In Rancher v2.x, you can constrain pods for scheduling to specific nodes (referred to as hosts in v1.6). Using [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/), which are key/value pairs that you can attach to different Kubernetes objects, you can configure your workload so that pods you've labeled are assigned to specific nodes (or nodes with specific labels are automatically assigned workload pods).
<figcaption>Label Scheduling Options</figcaption>
Label Object | Rancher v1.6 | Rancher v2.x
-------------|--------------|---------------
Schedule by Node? | ✓ | ✓
Schedule by Pod? | ✓ | ✓
#### Applying Labels to Nodes and Pods
Before you can schedule pods based on labels, you must first apply labels to your pods or nodes.
>**Hooray!**
>All the labels that you manually applied in Rancher v1.6 (but _not_ the ones automatically created by Rancher) are parsed by migration-tools CLI, meaning you don't have to manually reapply labels.
To apply labels to pods, make additions to the **Labels and Annotations** section as you configure your workload. After you complete workload configuration, you can view the label by viewing each pod that you've scheduled. The GIF below demonstrates how to apply and view pod labels.
![Add Pod Label]({{< baseurl >}}/img/rancher/add-pod-label.gif)
To apply labels to nodes, edit your node and make additions to the **Labels** section.
![Add Node Label]({{< baseurl >}}/img/rancher/add-node-label.gif)
#### Label Affinity/AntiAffinity
Some of the most-used scheduling features in v1.6 were affinity and anti-affinity rules.
<figcaption><code>output.txt</code> Affinity Label</figcaption>
![Affinity Label]({{< baseurl >}}/img/rancher/resolve-affinity.png)
- **Affinity**
Any pods that share the same label are scheduled to the same node. Affinity can be configured in one of two ways:
Affinity | Description
---------|------------
**Hard** | A hard affinity rule means that the host chosen must satisfy all the scheduling rules. If no such host can be found, the workload will fail to deploy. In the Kubernetes manifest, this rule translates to the `nodeAffinity` directive.<br/><br/>To use hard affinity, configure a rule using the **Require ALL of** section (see figure below).
**Soft** | Rancher v1.6 user are likely familiar with soft affinity rules, which try to schedule the deployment per the rule, but can deploy even if the rule is not satisfied by any host.<br/><br/>To use soft affinity, configure a rule using the **Prefer Any of** section (see figure below).
<br/>
<figcaption>Affinity Rules: Hard and Soft</figcaption>
![Affinity Rules]({{< baseurl >}}/img/rancher/node-scheduling-affinity.png)
- **AntiAffinity**
Any pods that share the same label are scheduled to different nodes. In other words, while affinity _attracts_ a specific label to each other, anti-affinity _repels_ a label from itself, so that pods are scheduled to different nodes.
You can create an anti-affinity rules using either hard or soft affinity. However, when creating your rule, you must use either the `is not set` or `not in list` operator.
For anti-affinity rules, we recommend using labels with phrases like `NotIn` and `DoesNotExist`, as these terms are more intuitive when users are applying anti-affinity rules.
<figcaption>AntiAffinity Operators</figcaption>
![AntiAffinity ]({{< baseurl >}}/img/rancher/node-schedule-antiaffinity.png)
Detailed documentation for affinity/anti-affinity is available in the [Kubernetes Documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity).
Affinity rules that you create in the UI update your workload, adding pod affinity/anti-affinity directives to the workload Kubernetes manifest specs.
<!--
The following Rancher v2.x workload is running Wordpress and mySql DB. Notice that both `containers` have constraints for a minimum (`requests`) and maximum (`limits`) for both `memory` and `cpu`.
```YAML
apiVersion: v1
kind: Pod
metadata:
name: frontend
spec:
containers:
- name: db
image: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: "password"
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
- name: wp
image: wordpress
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
```
-->
You can find more detail about these specs and how to use them in the [Kubernetes Documentation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container).
### Preventing Scheduling Specific Services to Specific Nodes
In Rancher v1.6 setups, you could prevent services from being scheduled to specific nodes with the use of labels. In Rancher v2.x, you can reproduce this behavior using native Kubernetes scheduling options.
In Rancher v2.x, you can prevent pods from being scheduled to specific nodes by applying _taints_ to a node. Think of a taint as a "stink" coming off a node. Pods will not be scheduled to a tainted node unless it has special permission, called a _toleration_. A toleration is a special label that allows a pod to be deployed to a tainted node. While editing a workload, you can apply tolerations using the **Node Scheduling** section. Click **Show advanced options**.
<figcaption>Applying Tolerations</figcaption>
![Tolerations]({{< baseurl >}}/img/rancher/node-schedule-advanced-options.png)
For more information, see the Kubernetes documentation on [taints and tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/).
As of v2.1.1, there is no UI support for scheduling workloads to specific nodes, but you can still use this option by editing the deployment's Kubernetes manifest.
<!-- TODO: Add code sample -->
<!-- Question: how are taints and tolerations different from antiaffinity? They sound synonymous -->
### Scheduling Global Services
Rancher v1.6 included the ability to deploy [global services](https://rancher.com/docs/rancher/v1.6/en/cattle/scheduling/#global-service), which are services that deploy duplicate containers to each host in the environment (i.e., nodes in your cluster using Rancher v2.x terms). If a service has the `io.rancher.scheduler.global: 'true'` label declared, then Rancher v1.6 schedules a service container on each host in the environment.
<figcaption><code>output.txt</code> Global Service Label</figcaption>
![Global Service Label]({{< baseurl >}}/img/rancher/resolve-global.png)
In Rancher v2.x, you can schedule a pod to each node using a [Kubernetes DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/), which is a specific type of workload <!-- link -->). A _DaemonSet_ functions exactly like a Rancher v1.6 global service. The Kubernetes scheduler deploys a pod on each node of the cluster, and as new nodes are added, the scheduler will start new pods on them provided they match the scheduling requirements of the workload. Additionally, in v2.x, you can also limit a DaemonSet to be deployed to nodes that have a specific label.
To create a daemonset while configuring a workload, choose **Run one pod on each node** from the the **Workload Type** options.
<figcaption>Workload Configuration: Choose run one pod on each node to configure daemonset</figcaption>
![choose Run one pod on each node]({{< baseurl >}}/img/rancher/workload-type.png)
<!--
{{% tabs %}}
{{% tab "Rancher v2.x Kubernetes Manifest" %}}
This is what the Kubernetes manifest specs look like for a _DaemonSet_:
```YAML
apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
labels:
workload.user.cattle.io/workloadselector: daemonSet-default-globalapp
name: globalapp
namespace: default
spec:
selector:
matchLabels:
workload.user.cattle.io/workloadselector: daemonSet-default-globalapp
template:
metadata:
labels:
workload.user.cattle.io/workloadselector: daemonSet-default-globalapp
spec:
affinity: {}
containers:
- image: nginx
imagePullPolicy: Always
name: globalapp
resources: {}
stdin: true
tty: true
restartPolicy: Always
```
{{% /tab %}}
{{% tab "Rancher v1.6 Compose" %}}
The sample below is an example of a global service in Rancher v1.6. Note that just placing the required label is sufficient to make a service global.
```YAML
version: '2'
services:
global:
image: nginx
stdin_open: true
tty: true
labels:
io.rancher.container.pull_image: always
io.rancher.scheduler.global: 'true'
```
{{% /tab %}}
{{% /tabs %}} -->
### Scheduling Pods Using Resource Constraints
While creating a service in the Rancher v1.6 UI, you could schedule its containers to hosts based on hardware requirements that you choose. The containers are then scheduled to hosts based on which ones have bandwidth, memory, and CPU capacity.
In Rancher v2.x, you can still specify the resources required by your pods. However, these options are unavailable in the UI. Instead, you must edit your workload's manifest file to declare these resource constraints.
To declare resource constraints, edit your migrated workloads, editing the **Security & Host** sections.
- To reserve a minimum hardware reservation available for your pod(s), edit the following sections:
- Memory Reservation
- CPU Reservation
- NVIDIA GPU Reservation
- To set a maximum hardware limit for your pods, edit:
- Memory Limit
- CPU Limit
<figcaption>Scheduling: Resource Constraint Settings</figcaption>
![Resource Constraint Settings]({{< baseurl >}}/img/rancher/resource-constraint-settings.png)
### [Next: Service Discovery]({{< baseurl >}}/rancher/v2.x/en/v1.6-migration/discover-services/)
File diff suppressed because one or more lines are too long
+1
View File
@@ -0,0 +1 @@
<mxfile modified="2018-12-05T02:40:02.031Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" etag="el7oWMVISfPZBoWDxqoZ" version="9.5.8"><diagram id="GIbyU7ykz9qA4X5SG9LT" name="Page-1">7Vxbc5s4FP41ntl9SAYkLvZj4ibb3clOM5vudPsog2KzxYgF2XH661cCARYCG9eASUr7EDgIgfR956JzhCdwvt79FqFw9SdxsT8BmrubwA8TAHR9BtgfLnlNJZZhpIJl5LmiUSF48r5jIdSEdOO5OJYaUkJ86oWy0CFBgB0qyVAUkRe52TPx5aeGaIkVwZODfFX6xXPpKpVOTa2Qf8TecpU9WdfElTXKGgtBvEIuedkTwbsJnEeE0PRovZtjn09eNi/pffc1V/MXi3BAm9ygPXynd3H4AL/46483f5uf//hkX4letsjfiAFPgOWz/m5db8tfmr6KmbD+2/A3vaV4R6+Q7y2DCbzhE88ejyN2wK4lExzQqziBkV/Vp+GuuJcdLcXf5Bm8cYKdTyLpIRMAocv/q/fOEaXslYB2F2y9iARrPnzRHxt82qX8GCZORpNJgTQwEJFN4GI+Szq7/LLyKH4KkcOvvjBSM9mKrn1xOUeRnzhk7TnsmI89phH5huf5UKA9nc/m9/mVjD2Aj9vz/b2WmmabN1MmF9P6QcwpvN3iiHqMjDfiAiWhmLZ7tPZ8rlMhCUMvYCy/9dEC+7fI+bZMRpQ9ICABG8otYUPyKL8Davks8P7xrpZQek5Tpt+YrDGNXlkTcYMhiJ1ptjh9KdTEzmSrPRWBmRAJ1VzmPRfsZQeCwCeQeVpB5hLWfGrC2rELI4IWWXPt1Dmx5DkBpjopEAJ1UnS9q0mZKZPykcRc6/ThKQJ2nrVnvczwiCwIJUK61xoaxsy8a642jShfTywVcwHyla6CrJvTCpBBVyBnb6CirKrAiPJpKGe92EBS7SGgrhq8FQojsntVQGc4hvyQXXVwHB8HfpH7kU8b6nvchSRymT0SPRrBmYPfhr9cRsj1cNG/cHWdUWBWMu7XpkoBUOXyOmOAat1HBnTKAH1oFACwNoSPQxRUxvAFtFci9uaxerRc/MKiMx7Kgzl3HTzAy4/1X9VY3FkhEUnkgXb6zJpIezAczJ1KtQ9pw7l1RcFZKeq2Ff6Zep/0U1eQL3hRGV6O6LfugirQt3pFH9Sgr4adI/rtu58K+PWMIb3gbxoKzthd4idxSiK6IksSIP+ukO4BqMmI451H/xFifvyVHzMHy84C9rbJpWs7P/8qmv6LKX0VmUO0SSAvHvxAOKo5O6QYgv/jXmwTbXM+4cC94WlDdrrwifMtFd17vl80EM8y2JmL4lV+Lz95RJStToJEwo1gRgY+LT9ABTa1ZBM5+EC7DHCKoiU+1GFd6iLCPqLeVn699qlidkKVlB6CLLpMldaZciI5hoM8uCjyaijQCvJgwNAPwS5M34ZdgH3bBYkaSbvOyVF2MW+VKRe1I6Aqv99fsWoR9VJXSiKSQ4lW154ttOLKgXj0+RlbDuunz7pSYzp3EB3LCVpQUXrRKvKzRlfBMawqrvZcjzIMpSBVkbHquSSVPU2tVqi+YKxWHGTXkItSUM1MCpjVVeMI84kwD7cqBQ0F9rEm0SkHBleWguZIgZ4pMLi6FC8klTggykVjbrgTDgysMARtBf+0NKDGeCP87XsBa6rA32tlCKqLdaH+I/69uICL14asbmpDB9K++Ul7mb2B1oay1MLR1F4WiR1N7aXu+lKpPav3JPC1qVkyXTQTZIJHHHlsYHxB986KBx0QpyYp1xNxOqot9VlaevM2xngbNkZdd7RAFX2PKPubE/Z9VUYWfSTLCWS5qF2B6sban7HWNLNsiKxGAa7umtievJ9a07Fwe2DFJqNqG3B75u78CKjef9ZVLesrnw2sVPat6NGS9rlmRtz6SLzkW8YsGWOV6FHCPbWT4qYCeqUftmaTOtKnpY7SASodJRzKh3MGrdQPix4IcpmEKToKnMTAqTEYt4UlmyOBKfR5H3khOmo+1p7rJlytMnIyfyU7V8fMRhbipI/+dLviS8iK3EtnFRhDTb2+UVuQe5V6z9TcFhyNTYzGscnskrGJoRbYVJ1UEf+5dLK88aGpUnaWDjNmZ+mg3jgd1vIK48DG64ssPxors91QmfVWIgA2Eeh1r0HIPXLchUvOPilvm0h2X0QC75VI53qFnolkqBW6v3BM/C2O+XBJslC10Job8WARhwfXp4Xkit1c/MSA9vtjuuDlgeuBpuBIU4XzHe0Yytl7eMWbOcvzNgu1tYA9slQ15cXIrGKLkFGxNaC81Ghvy/dUQbPzDwLe42dDjS1Y0/3daXLuYl8CnBch1dJCcWwHfdFIi5pa0MWWO+pOgpbdFGzupozRTXXlpkw4NDfVScHooD2SjFHasG17tGdujlqnI8CDdu1R48+Q0q1l3QXa1alY3ZJ3WMFZ6ff2Su0N82B7dpC+QbuLQ+vCnAUD33XTNmebfoJ7NmfPMmV2K9UhCZbajEIDb3EipOfSqUW8s5ThUbyNy+Jd+ass1yykYN35ZOkFCvwDTA83YFKHGWSgWUoAUhV/dFbGsdU8/6i0P6S0oKnSXnTPSfaapS3To9aetIqYdqa17LT4teY0YCt+8xre/Q8=</diagram></mxfile>
+1
View File
@@ -0,0 +1 @@
<mxfile modified="2018-12-11T19:28:00.479Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36" version="9.6.1" etag="gA-6378sX7AcZ2MzV8Cx" type="device"><diagram id="a4f168be-4a87-79f1-ee78-a1f69b98b0d0" name="Page-1">7Vrbcts2EP0azTQP4vAiUtSjJFtOM7Hrqd1p0zeYhEjUJMGAkCzn67MAwQtIUVIq1clMEz2YXNz3nD0LgBk5y3R3w1Ae39IQJyPbDHcj52pk29bEMeGPsLwqy2zmlJaIkVDZGsMD+YKVUTWMNiTEhVaRU5pwkuvGgGYZDrhmQ4zRF73amib6qDmKcM/wEKCkb/2ThDwurb5rNvb3mERxNbJlqpIUVZWVoYhRSF9aJud65CwZpbx8SndLnAjvVX4p260GSuuJMZzxUxrYZYMtSjZqbSPbS6DpIiRbMT/+qhbtfd6ISS043vExSkiUjZy58DGMhBk8QJn0ZcbHhURMlFp+vmvawlOk/soxRGUJU0KZNsjIdpxQ/Pptf0dZEMsBt5bhwZ8HjoLnqkdYadmpPhCY5Xoqq60tzWZ0k4VYuMSC4peYcPyQo0CUvgCFwRbzNFHFNWTiJaApCeBZrL7gjD7jZb0YZ+ovZ8tVXVJRxRYrJ0nSqmmaU3fug1059kp51VlsMeMEmDdXBZzmynErlJJERFBO85xkQOlFgp5wsgB3RHJF1QAZzWApCwpLIly0cMzaC22qKPaIIfGuZVLUucE0xZy9QpWq1PPKJiqQPcXqlyYo3MoWtwLCrhQAqUCM6q4brsKDousA1/dxtwOs8EN++kJrxUBPVQ/mQQe4pqU5wLK8ngdq1Wp7wLL98z3gDQbvvhho2+5Ak4+EzBPbE0R6xR8tinCwNtdWNzwYfaKcKmurtjOZzNzr02NuD43sAR71+XKcDxeIiFmPD3cRyXY9oMD3uXjMGQ1wURwH66mWlN82PCFCTaRdR1yD9DQILPE7D/TZDDlISGfEUEhw079SvbNgU6UTV4vySV/mavzaoHqXUDmrB+qbq5w109dvWe6pKncBDzg/Ve77qtzp24I34YPf48N7jBIew66w2Qc2uN6SgNECsy0J8D7Uf2rjvwT7B9DGfRugDp44hJObeqWMxzSiGUquG2sHqBay/2DOX9XhE20kRk0PH6kIF1kP7wj/q/X8ScS4MXXFawZLqsvES6uwopo8kR5Eo6AbFmCN/hyxCHPNJJZ6EDGGE8TJVj/E7gNANb2nRJ7PFNL2zNSg7kJYTkk1alCcM4ZeW9VyUaEYHsY19WF8t93b0eru4eqW5x6oDg/ldBv+1f47jZLTSzKwpk+bTp++kZ4F4MLnJceuggQVhchCpXlFkmassKqk1AEsqlxkrGDDtvUcK8qbhu22WC+KQlTEdb3BTckRRs/eiNG2zh1n1qF0Oc2zKW1PO+O45kGSeo53qP7ZLP3w683dKr575gv/j8dFkfy9+2CPbXdwo1XkKKsypm1AnVuaEU4Z4Cw3kxxB3mMSjyKnWQiu8FAqEmX2VOT65qrd1/AurF0LqA2jlNkd8gqkFfB8PVgBWd38pdgEIl2/O2sgKpaQ0U7Xa0SSDcPvjMG+OwEvbsj0OIbuyBe12xaxpPgCtd3FyL06ZXeWkjCUYjFwxwPZfi3/1TM6ed/fCdDpwXQ/Ng1zMtXpOZ6Ur2dG40zr1DJcvQO6XheYd+Lgm/XZ/4/02db02Xx7fW4kedqW5J50f7s+z77bjmM80XVz7J6mz30B9ic6uaYdcl1o79IT7sM638sLbyHzw7eGbVWzhMxrByuzc5K6nMDLAcCr5loqMM1xJlgJ0SL093F5fwFZT9EzLlTiuLl+lNz8vMGFHOL94+OZYyDhGtF/TAtBEFMmrXRfmvy/5BH/aB4xLX0vPrYukkfGfqfXTg8XSSTVUfPQ2bM64JFUfk9sIzkIS1XwUcBzTwvCCT0FN5UT5FDzIi+/egqOoOplTXZC9xdqNlcx5+Jz6Vx4wV4FYWYaBIBdE8h1zAhoKj6aIQ4BsxJ2IPUqJ5mk3BgFclr2auILFmPEgnhs2b6RZ1EZGQHJogXlHLqBFU+qibdWxMrLgJqyCV43YXDWZynHmhodZtn9awlH0aR9K+Fc4KvMvm10hxh1Dt4L0FEKDJKn48qKf+kuEh/ijSgNEyOWmt67V1L86VwgTeWvfX9lHkDyEtCZVgc6x5kY/etmx90D3gGlGAAPXptv7aUKNP9lwbn+Cg==</diagram></mxfile>
+1
View File
@@ -0,0 +1 @@
<mxfile userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36" version="9.4.2" editor="www.draw.io" type="device"><diagram id="90f814c8-1a40-e035-bd9d-a47d365d761f" name="Page-1">7Vlbc6s2EP41PNoDCGz8GDtJ2+lpmmk6056njgwyVg8gKuQY59dXAgkQCJvEPr3MHHvGhtVKYne/vQkLbNLyOwrz/U8kQonl2lFpgXvLdR0P2PxPUE41ZbkENSGmOJJMLeEFvyFJlPPiA45QoTEyQhKGc50YkixDIdNokFJy1Nl2JNF3zWGMBoSXECZD6m84YvuaGvh2S/8e4XivdnZsOZJCxSwJxR5G5NghgQcLbCghrL5Kyw1KhPKUXup5jyOjzYNRlLEpE9x6witMDlI2y10kfOo6wq/i+dhJCr346yAeas1QyWYwwXFmgTuhY74TovyCj1W6zNisqCwmRp0gL9u5/CqW/9UegrkyU0KotonlAhCJ73Duj4ctohli3P6uvUkORbV7vR6Xs15S34aTK2kU1dUEcyk5ZBESCnH48HGPGXrJYShGjxy/nLZnaSKHG4OJm5CkOOTXQvaCUfIFbRpRwDLYrDaPzYgCiivkxknS4bTtpX8XcLpU673UKVi/Isowx92dHGAkl2p7hClOhPvkJM9xxgG9TuAWJWsYfokridQGGcm4KGvCRcJMzAB2owWxPipHseM0iOSujEiKGD1xFjVBubH0Ysdd1ffH1if8leTZd/zBDSTsoPTDuFm7hSq/kGg1I3dlQG7PsEIP+aigMjjArWK3362Ala4AVymkowDPpADHBXP/ehUsR53X5AVd2hMPyRecZksNbqQz/tf8CIU7e+f0HYSSLWFEUjvcwPNW/oPJ6/aE4jfODNXD9t0wxVEkpJ7mRatpIOqAxvEMoAns6xETfEPM/x8xwP8HEWOsEIxxlqtbpJgs/kU+gd2SfhV56/5s3pkSjs/rZdbLRzNnoaJsR1OubQrI3g1UtfjmXP+Oc9VV0STPqtE7hJCCjDt0LTNgFjcAjONcQIy5Mt5Vn2Fl/BTjrJyKDm7wXFzmlISoKC4jZNsUlj8fWIJFTXk+0o0jp4eTxvIxhRFGLSpk5aoDWUPqNGQ54jtArHMtZuQMX486XjC33WHYcezl3PF4j7hUv0NQ+WC+WPp2+/FuEL+9AcZyQlndnakEcXXUsM9GDWnHiwFjF4QoDPtm3ZADxVWH+YSOpvhwCyP2UgcAhsyxNLUyqxuEgWEr81By0TKYVC2ucIlLOZcTZ+5ZRF/f7iyadKqgPtTRChhgfYtQOYSxKaS1aIxgsa9gbPcga0JmD4e+Lb6qo34mBWaYGPvyTz0GHoAYSUdTFKkj56Y5lrJNeFahOS1jcXw2J7sdDtH8UCBaVL+T2/eReqnbni6+kr3AO+zlXGmvJoHc2l4D0xgN2LfXK0IwnScwrxfRI2o3Sk7KXo/V51qTq1wFNAc2VDumPgLcoo8w1To9QKAoRi/ylueoPYkJj4EPLbWnyw5M/kSMneRBMTxUemxX+ESq9kPwcUXR0+9C/zyWydvPVrc1EQ8xrl75zDwlhYpLqpRBGiOVFYDZChQlkOFXfX2TVuXUZ4KrMrBUubx34Gb3DFM/g5zV2uaOUnjqsOWCoRjfp3+uFdg9S9cLtnZvRJwGBWOjBFPhLIlqh/744bkh1hVsVbGsgtWwZhFH0jocKCrwm0x1wrRSYs7try3/3uTao3XsyLFqW4iPObLW+3JIXJmcL2TnGUe0o5mtaWauhZ2+rNsDHU9PBWJWPxa8ExP+LaMBKjHrODm/+zzB/6fGkLPVaZPHnA9HFNcUUbwrI8rkOG1K3FpPuoOh/kZIV8GgLd2Tgock1XLUDvw1TzBu04tEPgoib9S3P9aEXHid0OsknaWpCwkMCfoDp1f8tn3LWDts+64WPPwN</diagram></mxfile>
@@ -0,0 +1 @@
<mxfile modified="2018-12-05T04:59:51.427Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" etag="RO8xbcgUHuFQ74JOJZEx" version="9.5.8"><diagram id="GIbyU7ykz9qA4X5SG9LT" name="Page-1">7Zxbc5s4FIB/jWd2H5JBXO3HxE3a7ma3mc3udPsoGxWzlRGL5djpr68E4iIEGMdg0wzpTANHQkI6n86RjkQmxny9fx/BcPUHcRGe6Jq7nxjvJro+szT2Pxe8JALLmCYCL/LdRARywZP/HQmheM7b+i7aSBkpIZj6oSxckiBASyrJYBSRnZztK8FyrSH0kCJ4WkKsSj/7Ll0l0mnaLC7/gHxvldYMNJGyhmlmIdisoEt2BZFxNzHmESE0uVrv5wjzvkv7JXnuviY1e7EIBbTNA9rDd3q3CR+Mz3j94eYf6+/fPjlXopRniLeiwRPdxqy8W9d/5i9NX0RP2P9v+ZveUrSnVxD7XjAxbnjHs+pRxC5YWtzBAb3axGrkqWAa7vNn2ZUnfsd18Myx7jCJpEomumG4/J/67O/bBYoCRBkUujbH201ce1Iea3xSpFwNE8etSaW61DA9ItvARbyXAEverXyKnkK45Kk7xjSTregai+RMi/xmSdb+kl3ztm9oRL6hedYUw5nOZ/P7LCWlR+ft9jEu5NQ0x7qZMrno1neiT43bZxRRn8F4IxIoCUW33cO1j/mYCkkY+gGj/BbDBcK3cPnNi1uUVhCQgDXllrAm+ZQ/YWhZL/Dy0b4WKJBhyoY3ImtEoxeWRTxgCrDFyE453+XDxLGFbFUYImY6QqAYml5Wck4vuxAAHwGzUUVzSdm8b8LaxgsrAhdpdu3oTin3iqVdW0rHGIaudgwAvXWMrnTMn8xQM4kxvNGAll+1r6CMeUQWhBIhLeQ2THNm3bUfO624b6BLVbxQ9BVQ+QfWtELNem9qNurUbI5qPlXNaTGOLo3vIajdrHXiwsGqXrzspkGVm/4YeBHaxH6WZY8Ixg2utsQXQybkl2FElryMg4wtMsf1aUuxz31WLJdBlUhsRU7GWRcO2oug66O8fOFb+6NtVnImFb4kA6tIm9EbbNYI25uFDQyONluhbbmCnDJ1ZjcYCDJfVe2auvCZvTEwk2evtqMAYIFz6t9R9L9Di8p566j+7v2NPVXUb59V/dO64T/q/ywuoGL8g5SRsxCQ4lZQNHI99CRuSURXxCMBxHe5tKBBTVY52vv0Xy5mji25+5KmsCnHSyGJ334RBfyHKH0RkUm4jVWe1/tAuFYzOiQ/zn/4gmkbPWc8ocC94WFJdrvAZPktEd37GOcZRF0mu3PhZpU9y28eIWXrniCWsKlUBgPvldegwLqWbKMlasoo1hcURh5qLNGuhitCGFL/WX6/7lFRpwQdoAIKoGjXjiWxIpESgzOy0paV6SVZUf2Kwk7fgUJbiRMqptbRKuba/UUJZ0qniOiRaoPH6FETWEOOEaZvoGpZHQKjlo/T8nBDhKDK4I1Rm58ianMAtsFFCIHqR0bW3ghrgwsQ6uqWV4rEJoRBJWy5aq/EyQPOXOQtftH5Coa96py7Q76Rm12DX1UeRSgCFBhM6hw6g4OOUDQjOLD4ZNoqJT6pTplH7Xfu7Cq0f9bwZLpqVLQ/bk6cw/1cPDhpqccqughOShEnOeBUCEC98eikWLAcDDilCj8YcKoLx7QOOMWPst6BL4UMIfEDuimU/MgF9cxapTOapey21ZCbXSTV58Rm7TgBYqsXiM8aYX/L2OonYnuShXPU/VeFltdFibRDUaKZ7RjQPuh4xJKn2vcpgaAWmj9zrKgUBdbVGc2sYj1l9uXSzNlJ1gC0tQYde60G/3gR29B2u8R0WpoBcOp2yau8l6mX8ATN7gvYs6b8PTkwbUR2kMjO+kVWsnMdcGSqvu4vtCH4Of7ShOv7ZqLbcM3dQbDYJF4hifwsovInJ7nkij2c76NpHx+TcNNUa8yqt89qtMs6iU+BN+bsyLUf3ADKhsQJrr313k/r0XHStzCGJVtKp2LLx6xw5P3t5qYbTGec1+tvcHHauVlMTkicahYV12s54Noo/MgLSeBoUmp6rjItP2meKLJ702qdNqscWeyLRasXFm2zBxZPnrtqB0IvNmjK39Pc1e5nYDj1I0MaFskwGfDI0LsdGa3DLgmUFwu7VO1qdrfGaWGNjtTi6QG8+iOV9eZe4q5DShzQkhLzVEqqTZE+lU1XeY7Ys7dOmy9va12zKT0rDhPPDxQY+R8FKK0GpEl/OpVXZ/cHz2qtfdeNca5afsimUAoutoC8jsYO1gO6ZisLgKr5f28nvpyqj0KPNiHSCPsZ7Qk4xp50aUL0y5oQO+VPmJBZibPkvfozIbpCHz8tM9qQY2yINb20DXnN7LSdyWi12S0t4YZpaDo0Ga0/nUjOvJ19L6C8VSV2tuvWU+X8xuzA3oFjN+XvZ/2VTrTaekmEF2TXbh/hUhPoa40tcpQNBXbziCKf9Rc3kLGwfHwHuBZyFKtb3ImNW89kqXk9iv+DWDv9eEJQigMAp91s+nwbD07FhzfMQe6Z6D0mC8gLeCCQR/CZq4PBkh97Hp1ns/N0Slo3KwLyVZ9XvcKZstv8r7glUOR/Cs+4+wE=</diagram></mxfile>
+1
View File
@@ -0,0 +1 @@
<mxfile userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36" version="9.4.2" editor="www.draw.io" type="device"><diagram id="90f814c8-1a40-e035-bd9d-a47d365d761f" name="Page-1">7VtRc+I2EP41zLQPMLJlG3g8SHLt9JpmJg/tPXWELYx6xqK2SMj9+kq2ZFuWDCYQyE0DM8GsJEva/b7V7poM4Hy9+5yhzep3GuFk4IJoN4A3A9d1PAfwDyF5kRJ/Mi0lcUYiKasFj+Q7lkI5MN6SCOdaR0ZpwshGF4Y0TXHINBnKMvqsd1vSRJ91g2JsCB5DlJjSP0nEVqV04oNa/gsm8UrN7ADZskaqsxTkKxTR54YI3g7gPKOUlVfr3RwnQntKL+W4u47WamEZTlmfAW454AklW7m3gRskfOgsIk9ifexFbjr4dysWNWN4x4YoIXE6gJ+EjvlMOOMXvK3QZcqGeWEx0epMNrt6LL+K5Wcxh+hcmCmhmTbJwIUwEm9z7G/bBc5SzLj9XTBPtnkxe3k/vs/ylvo0XFzsRkldbWNuRrdphIVCHN78vCIMP25QKFqfOYC5bMXWiWyuDCa+hHRNQn4t9p6zjH7D82orcDyZT+d3VYsCiiv2TZKk0ROAsf9pwuVSrTdSp3D2hDNGOO4+yQZGN1Jtd2hNEsGfDd1sSMoBPUvQAiczFH6Lix2pCVKa8q3MKN8SYWIEBJUWmkCR2BFT4l1DJIHzGdM1ZtkL76JaoQSxpLGC/3NNibEC/qpBh4r/SNIwrm5dI5VfSLB2IH1qIPd2x5WWoqTABRH7altaKEYoMBe6SGMuHLrOUcqofApaqJuCvUoKdB0NXVNJU2jqyD+HijwLuVsaqeAsMBGhfFXwALQwr0Fb4qmFYR+It0LhA80JI9SK5S+tDgvKGF13gp1uWUJSPrty5cBGFL6PjdjSeheLM2dEl0sS4tE2x1le/LVbedphZtOcDXt5wRvZCx5hL+dEe0HPm/q3b2AvwzRWA7bt9YQxWo8StClvorvkppvVfF9G+WKolDY2d1e8TjW5bPWhxmALgT2Lk4NnAAR0DgNCuTSl9gQv60Out/uyOL9ut+9PWwqZGhpxbX7fOYffH58csBQa6gxJ+NGQKtk9j1wbMUSz6UcJLXC4BEunL28qp2CQeEUz8p13RmqxdvKbWIL9w4qL4Efx0gKgfdHosniZsPnGo9HhJqO7l84Y1PTghefjg0Kc54dhsagiuj9KT9phgTWJIjHJHrh0nQHcB0ciYGodGTp6NXj2g5Mj3gZM64jL4p9M7HQ6Ii8Y+Zongs4IuN60eo3HhmNywHjkeDxVG6u/lrMcjoKxD+qXdwbY2RKtq7vyAOjR6WVdObTFO00qWpNF08HXBBlK7go3n8WLn1wYiMTU5UsBxSldXTs/m1xuuXvbGhaZ5QDYS/f/5wlwiNjuNU8AW1r0cQK8zxOgAyg/4gngnxV2KXdWDzQT4x5x9sST3bf1SL1zr5bJD/sseDwsq8RXZcEPiInaTyFxgZDGCRLUugF9oXYw6WlBrarpNrAV2DLB4Aw+y7dFrS274ijGj/IrB8aKxjRFyW0tbbG5YeV/MGMvssyOtoVV6zt8ocKx6wWHmsBi1v065Yuk2yyUvVx56DOUxVjLlU3dZzhBjDzp9z9Fje6hiAMsUagHF3O+dlLUt+/xs0nDjaBgEW9A4DrB23OwOyqQ3DgYEEQ+nkReJ52Ng7/TrP1LKFUpUnHHmxjcmVqo457D73ZXDD5svt/mJ3lMD3ptj3lBq3s9aqk/hsOEgcVhehdymGryD/JcljyBfz3qQPPJ2jWowzWVvfwlDDdyffX9qxbRHUslxxZ7+CdSSQ59oKTggzrxoP7UwPH1O5RLkINatqlW0S80PHtMwz/RWlCquMn9Hze3f//6UAnLLsczWJThdSRkOCffZfUKFK6Ba6PQjz8b+Dc2anVmvB3PvusE6sQ6WkmKTr4OwciZAmnj10JHdaHLZY5PhYVyGO+r4AgnoBUU9C45VrOd4tsOHWe2iuNHcfANHw9VMD398dBZAOLaaPNRHHyXxcESO3uKg5qjecelQcf2nP/1oLuPSbr7wNvxeHulY7LjzfF4/tsEnBnBXwtvKkto4K1OyybAwMg1k6/lJMRh2DaxHsH2Klodb9DAKGC4lUT7jaPlLJqebiWFp1emXa3fhOEdYWU25ctvXwdGomXkWW9V51DG0CrDUtbMztxLFTq895H0nqNeNLaoseOkPLsafeecWuuDWQ2xRb9L6hpcEbL+WRHaQ9fOdXXtXU7X/Gv9Lxhl8l3/Jwu8/Q8=</diagram></mxfile>
+1
View File
@@ -0,0 +1 @@
<mxfile modified="2018-12-11T19:43:39.200Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36" version="9.6.1" etag="2DlBBRwyfcv_srXDQ_dR" type="device"><diagram id="a4f168be-4a87-79f1-ee78-a1f69b98b0d0" name="Page-1">7Vrdc9o4EP9rmLYPMLKFAzwCCb2Za5tM25lrnzrCFkYXY7m2HMj99ae1JX8bTELczF2TB+zV12o/frsra4CXu8P7kATbj9yh3sBEzmGArwemaYwxkj9AeUwp0xlOCW7IHNUpJ3xh/1BFVOPcmDk0KnUUnHuCBWWizX2f2qJEI2HI9+VuG+6VVw2IS2uELzbx6tS/mCO2ahcWyul/UOZu9coGUi07ojsrQrQlDt8XSPhmgJch5yJ92h2W1APhabmk41YtrRljIfVFlwFmOuCBeLHa28C88uTQhcMegD/xqDZ99TMGphaCHsSQeMz1B3gOMpYr0VA+yLZElr4YRonGoNWYBod8rHxy1W+yBnRO1OTxsLTIwMTYgf/62M/Et7fJgg/mSG4O/RmvaehTIQ3CREsvjhJ20gXkxtM1yutKcrI9TTVLOzVDHvsOBQkZsnm/ZYJ+CYgNrXtp0JK2FTtPNWcahBeb75gtn0EYkQj5PV1me8OT6XK2XGUt2nJMEATzvEJPhCbWfCrpSs7XSsh48UBDwaQhzlWD4IGS44rsmAf+FPAgYL608IVH1tRbEPveTXakF/C5L7ey4HJLTMAIjDIpFC1HGRMsSQ8FkrKk95TvqAgfZRfVao6VVSu3vlKv+9xHLO0j24J/YD2OKL90s6lz05UPynpbTP+qwZQriqWOdGP1ykOx5S73iXeTUxdl1RfU/DcV4lEhEYkFBwFmM3zgoIakHz0w8a3w/B2sYTSx4NWXW8ra4KXQKM0oSOYGeDqmDtjDUWWE1COCPZShqkmyaugdZ4kXKiWOK0o0ZrPyFBGPQ5uqUUVoOXciQUKXitpE8zAkj4VuAXSI2hmeSPkV1zGRdZSvan9jcrx/bR/l/vIh5Ti30kwZnQxXB8NjhgsOHHT30CzykbWeAQ0ajUV7blUmZt11s+hbdF3TuoDr4tYg1ATeRdonmVqcwPp12ID+5Y6vDf6pvUEbo4rrIV/zBHeAWuiNx+OZddM9WNTNKLPA00jfiz2M8SWRWyMyGllFTC4CsmrSkJzoVaKTmKdgfG17JIpAsyl5xbxsct/RnVRYlRTVjs4IG+1KSeFWu2lKS5FT0WaXjgtd1aS5KfjtJ5f5hybPq6hTR7sg5DaNotNuts6ymNtYeAwkndDLWi85YzfnMeD/ee46mxFMIFtzQ+Iwms+vLOJZDqeDUAWfrWkNnzPMLvrj1SXcEdX0dxF3nBT98fX63MX9qznNMHCLip+cLpX0e15Oojd9VkhuruNUMKvVcfeybPOo+O9F7ywenwdAtejdgibHi7oXwBpsjayOYDNrN/rOueD0ydlw607PT4crGT+e1QTQVMdeoozF9aB6IRNHR01c2U9vuWmLGcsxN8sVWrXg87RFw3VNKtU12W6T6sxLmK51WnUlbTgk2iZqRBWVNWmmogcLwb8W5B2PmGC8Mfv/UOkgNSX4rrU84Gl+tcyOTlGT+nQCtzu4cMQ74psNs+lI+plNAxGNHphD+Y/Ag7r4xXXdh1sap3X70kW6VT2I6LdIbz8q/l2l91Kldz+P7adKN2o6+R9W6VoIhYoBq+OLYpWOx4NfVKXj+una7yq9W5V+9geQX1ql1wPSy1TpBX+clBwSvVqHHDc4JP5lDjl+QiD9XVu/ytr6uQjRb229+/rIr76xu2+O9fE+Nva33z/dDsfdqpYkELBdcjGiqOXm7OVk7dFeEmySP9klWWweBYUqRL9s2AGscKH4ud4KATc/5iAJc2U7vjliMrneMGmtoSxKdvDBnwgif4AegaCk8PgwZj6wOnS4fS+JhikBe3XH9zT8MfzxmSawNQp8tzWFr1tDq+px9VDBaKpNcV31mvas6NAOOl0venh0I+pQY4xk060P6LplcAPDT5L9ZUKQDyjDIhTGIPnmGxiXYEX+kh1Yn7+O4IdAT4ncPuQuUlUSJoAhDsxCW8CdhEsi3kTVsS/Jp5SDz3wXBPc1EVHKCGUivdcSyVgIDEM21StjJAl3USA9BNh6G8U25H3vQGjAmMNp5L8BVb7dEObFIX3Xlb0jmQn11nzfKSkpJRfZgVWeWaByZpGnH3ly8ZSrHGhQu8qRndzUAl3lVs25EaUhe9Gf2U5mKmVk0jCjr5498/OEiarwVclZWz5P1CYaGpWZcoKeim82ERU1eDvvE0ZjXLuEl5gdES/KIA/Qpj/YC6nMbbphHmJ+n/iSikdXk5KrTQo44MJlQEzZ7o8zyBQiJZhMj6msEhWv40j6xjLig/J9w/60Ctib2JpQEQJYg2fip1XKz1gKMho1SK0/gG5GzDpg9ovjL4zPnSvJXvG5dqZwodt2llFeB6PKBePq7Tk0LfeflfqXb8/1FSzka37FOu2eX1THN/8C</diagram></mxfile>
Binary file not shown.

After

Width:  |  Height:  |  Size: 93 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.9 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 136 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 282 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 114 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 68 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 17 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 11 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 183 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 104 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 191 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 145 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 106 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 119 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 79 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 102 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 63 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 182 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 100 KiB

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 81 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 87 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 118 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 138 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 61 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.9 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 78 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB