Kubernetes Cluster On-Prem: Kubernetes Cluster Setup

Updated 1 day ago by Michael Cretzman

This topic covers installing Harness Kubernetes Cluster On-Prem in an existing Kubernetes cluster.

We assume that you are very familiar with Kubernetes, and can perform the standard Kubernetes and managing configurations using Kustomize overlays.

Harness Kubernetes Cluster On-Prem uses the KOTS kubectl plugin for installation. This topic covers installing KOTS in your existing cluster as part of setting up Harness On-Prem.

Installing Harness On-Prem into an existing Kubernetes cluster is a simple process where you prepare your existing cluster and network, and use the KOTS admin tool and Kustomize to complete the installation and deploy Harness.

In this topic:

Cluster Requirements

Do not perform any of the steps in this topic until you have set up the requirements in the Kubernetes Cluster On-Prem: Infrastructure Requirements topic.

Summary

Installing Harness in an existing cluster is performed as a KOTS Existing Cluster Online Install.

This simply means that you are using an existing Kubernetes cluster, as opposed to bare metal or VMs, and that your cluster can make outbound internet requests for an online installation.

Step 1: Set up Cluster Requirements

As stated earlier, follow the steps in the Kubernetes Cluster On-Prem: Infrastructure Requirements topic to ensure you have your cluster set up correctly.

These requirements also include RBAC settings that might require your IT administrator to assist you unless your user account is bound to the cluster-admin Cluster Role.

Specifically, you need to create a KOTS admin Role and bind it to the user that will install Harness. You also need to create a Harness ClusterRole.

Step 2: Set Up Networking Requirements

Perform the following steps to ensure that you have the Load Balancer set up for Harness On-Prem.

Later, when you set up the kustomization for Harness On-Prem, you will provide an IP address for the cluster load balancer settings.

Finally, when you configure the Harness On-Prem application, you will provide the Load Balancer URL. This URL is what Harness On-Prem users will use.

Using NodePort?

If you are creating the load balancer's Service type using NodePort, create a load balancer that points to any port in range 30000-32767 on the node pool on which the Kubernetes cluster is running.

If you are using NodePort, you can skip to Step 3: Install KOTS.

Set Up Static External IP

You should have a static IP reserved to expose Harness outside of the Kubernetes cluster.

For example, in the GCP console, click VPC network, and then click External IP Addresses.

For more information, see Reserving a static external IP address.

For GCP, the External IP address must be Premium Tier.

Set Up DNS

Set up DNS to resolve the domain name you want to use for Harness On-Prem to the static IP address you reserved in the previous step.

For example, the domain name harness.abc.com would resolve to the static IP:

host harness.abc.com
harness.abc.com has address 192.0.2.0

The above DNS setup can be tested by running host <domain_name>.

Option 1: Disconnected Installation (Airgap)

The following steps will install KOTS from your private repo and the Harness On-Prem license and airgap file you obtain from Harness.

  1. Download the latest KOTS (kotsadm.tar.gz) release from https://github.com/replicatedhq/kots/releases.
  2. Push KOTS images to your private registry:
    kubectl kots admin-console push-images ./kotsadm.tar.gz <private.registry.host>/harness \
    --registry-username <rw-username> \
    --registry-password <rw-password>
  3. Obtain the Harness license file from your Harness Customer Success contact or email support@harness.io.
  4. Obtain the Harness airgap file from Harness.
  5. Log into your cluster.
  6. Install KOTS and Harness using the following command:
kubectl kots install harness 
--namespace harness
--shared-password <password>
--license-file <path to license.yaml>
--config-values <path to configvalues.yaml>
--airgap-bundle <path to harness-<version>.airgap>
--kotsadm-registry <private.registry.host>/harness
--kotsadm-namespace harness-kots
--registry-username <rw-username>
--registry-password <rw-password>

Notes:

  • The --namespace parameter uses the namespace you created in Kubernetes Cluster On-Prem: Infrastructure Requirements. in this documentation, we use the namespace harness.
  • For the --shared-password parameter, enter a password for the KOTS admin console. You will use this password to log into the KOTS admin tool.
  • The --config-values parameter is only needed if you use a config-values files, as described in Config Values from KOTS.

In the terminal, it will look like this:

  • Deploying Admin Console
• Creating namespace ✓
• Waiting for datastore to be ready ✓

The KOTS admin tool URL is provided:

  • Waiting for Admin Console to be ready ✓  

• Press Ctrl+C to exit
• Go to http://localhost:8800 to access the Admin Console

Use the URL provided in the output to open the KOTS admin console in a browser.

Enter the password you provided earlier, and click Log In.

You might be prompted to allow a port-forward connection into the cluster.

Now that KOTS and Harness are installed, you can perform the necessary configurations.

Go to Patch Resource Config using Overlay.

Option 2: Connected Installation

The following steps will install KOTS and Harness On-Prem online. There is also an option to use a Harness On-Prem airgap file instead of downloading Harness On-Prem.

Install KOTS Plugin

  1. Log into your cluster.
  2. Install the KOTS kubectl plugin using the following command:

curl https://kots.io/install | bash

The output of the command is similar to this:

Installing replicatedhq/kots v1.16.1
(https://github.com/replicatedhq/kots/releases/download/v1.16.1/kots_darwin_amd64.tar.gz)...
############################################# 100.0%#=#=-# #
############################################# 100.0%
Installed at /usr/local/bin/kubectl-kots

To test the installation, run this command:

kubectl kots --help

The KOTS help appears.

Now that KOTS is installed, you can install Harness On-Prem into your cluster.

Install KOTS

To install the KOTS Admin tool, enter the following command:

kubectl kots install harness

You are prompted to enter the namespace for the Harness installation. This is the namespace you created in Kubernetes Cluster On-Prem: Infrastructure Requirements.

In this documentation, we use the namespace harness.

In the terminal, it will look like this:

Enter the namespace to deploy to: harness
• Deploying Admin Console
• Creating namespace ✓
• Waiting for datastore to be ready ✓

Enter a password for the KOTS admin console and hit Enter. You will use this password to log into the KOTS admin tool.

The KOTS admin tool URL is provided:

Enter a new password to be used for the Admin Console: ••••••••
• Waiting for Admin Console to be ready ✓

• Press Ctrl+C to exit
• Go to http://localhost:8800 to access the Admin Console

Use the URL provided in the output to open the KOTS admin console in a browser.

Enter the password you provided earlier, and click Log In.

You might be prompted to allow a port-forward connection into the cluster.

Upload Your Harness License

Once you are logged into the KOTS admin console, you can upload your Harness license.

Obtain the Harness license file from your Harness Customer Success contact or email support@harness.io.

Drag your license YAML file into the KOTS admin tool:

Next, upload the license file:

Now that license file is uploaded, you can install Harness.

Download Harness over the Internet

If you are installing Harness over the Internet, click the download Harness from the Internet link.

KOTS begins installing Harness into your cluster.

Next, you will update the Kubernetes manifest using a kustomize overlay.

Go to Patch Resource Config using Overlay.

Following that, you will provide KOTS with the Harness configuration information (Load Balancer URL). 

Step 3: Patch Resource Config using Overlay

Next, you will see the Configure Harness page.

Before you can provide the Load Balancer URL for Harness, you need to update the Harness manifests using kustomize.

  1. If you are using a single terminal, close the KOTS admin tool (Ctrl+C).
  2. Ensure kubectl is pointing to the cluster.
  3. Run the following command:
kubectl kots download --namespace harness --slug harness
This example assumes we are installing Harness in a namespace named harness. Please change the namespace according to your configuration.

This command will download a folder named harness in your current directory.

  1. In the harness folder, open the file kustomization.yaml:
vi harness/overlays/midstream/kustomization.yaml 
  1. In the patchesStrategicMerge section add nginx-service.yaml.
  2. Save the file.
  3. Next, we will define the load balancer Service.

In the same folder, midstream, create a new file named nginx-service.yaml:

vi harness/overlays/midstream/nginx-service.yaml

Edit the file using one of the following options.

Any annotation can be added in metadata.annotations.

Option: Create LoadBalancer

Edit nginx-service.yaml by pasting the following YAML:

apiVersion: v1
kind: Service
metadata:
name: nginx-ingress-controller-chart-controller
annotations:
harness.io/managed: "true"
#You can add custom annotations here

spec:
type: LoadBalancer
loadBalancerIP: <static IP reserved earlier>

In loadBalancerIP, enter the static IP address reserved in Step 2: Set Up Networking Requirements.

On some cloud platforms, like GCP, you can omit the IP address and GCP will generate one automatically. For simplicity, we recommend that you enter the static IP address you reserved here, instead. However, you can let the cloud platform generate an IP and then use it for your load balancer after Harness is installed.

Option: Create NodePort

Edit nginx-service.yaml by pasting the following YAML:

apiVersion: v1
kind: Service
metadata:
name: nginx-ingress-controller-chart-controller
annotations:
harness.io/managed: "true"
#You can add custom annotations here

spec:
type: NodePort
ports:
# By default and for convenience, the `targetPort` is set to the same value as the `port` field.
- port: 80
# Optional field
# By default and for convenience, the Kubernetes control plane will allocate a port from a range (default: 30000-32767)
nodePort: 30007

Review: Configuring Load Balancer Overlay Post-Installation

If you are changing the load balancer settings after you have already installed Harness, you need to delete the Harness Manager pods. Kubernetes will recreate them automatically and they will pick up the changes.

Find the name of Harness manager pods:

kubectl get pods -o=name -n harness | grep harness-manager

We use the namespace harness in this example.

The output will be something like this:

pod/harness-manager-79fdf979f7-5gpfj
pod/harness-manager-79fdf979f7-l7fcc

You can also find these in your cloud platform's console:

In the following command, replace <pod_name> with each pod name and run the command:

kubectl delete pod <pod_name> <pod_name> -n harness

For example:

kubectl delete pod harness-manager-79fdf979f7-5gpfj harness-manager-79fdf979f7-l7fcc -n harness

Upload the Load Balancer Settings

  1. Run the following command to upload the Kubernetes manifests from the local filesystem to create a new version of the application:
kubectl kots upload --namespace harness --slug harness ./harness
• Uploading local application to Admin Console ✓

Now your load balancer configuration is uploaded to the KOTS admin tool as part of a new Harness version.

Step 4: Configure Harness

Now that you have uploaded the new manifests, you can configure Harness.

  1. Point kubectl to the cluster where Harness is deployed and run the following command:

kubectl kots admin-console --namespace harness

  1. In the KOTS admin tool, go to Version History. You will see a new version of Harness with Source: Config Change displayed.
  2. Select the version and click Configure. The Configure Harness settings appear.

Kubernetes Cluster On-Prem requires that you provide a Storage Class name and Load Balancer URL.

The name of the Storage Class depends on the provider where you are hosting your Kubernetes cluster. See Storage Classes from Kubernetes. You can just run the following command to get the Storage Classes in the namespace (in this example, harness):

kubectl get storageclass -n harness

Enter the name of the Storage Class.

In Load Balancer URL, you can enter the IP address of the LoadBalancer Service, such as http://35.194.31.219.

This is the static IP address you reserved and provided when setting up the kustomization.

To view and get the load balancer IP address in your cluster, run the following command:

kubectl get svc -n harness

You will see the the Load Balancer IP address in the harness-ingress-controller row under EXTERNAL-IP in the output:

NAME			    TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
delegate-proxy ClusterIP 10.44.4.186 <none> 80/TCP 2m50s
harness-ingress-controller LoadBalancer 10.44.5.226 35.194.31.219 80:30068/TCP 2m50s
harness-manager ClusterIP 10.44.0.234 <none> 9090/TCP 2m50s
harness-ui ClusterIP 10.44.9.247 <none> 80/TCP 2m50s
kotsadm ClusterIP 10.44.4.244 <none> 3000/TCP 3d

Enter the IP address in Load Balancer URL, using http://. For example:

http://35.194.31.219

When you are done, click Continue.

Option: Advanced Configurations

In the Advanced Configurations section, there are a number of advanced settings you can configure. If this is the first time you are setting up Harness On-Prem, there's no reason to fine tune the installation with these settings.

You can change the settings later in the KOTS admin console's Config tab:

Step 5: Perform Preflight Checks

Preflight checks run automatically and verify that your setup meets the minimum requirements.

You can skip these checks, but we recommend you let them run.

Fix any issues in the preflight steps. A common example is the message:

Your cluster meets the minimum version of Kubernetes, but we recommend you update to 1.15.0 or later.

You can update your cluster's version of Kubernetes if you like.

When you are finished pre-flight checks, click Continue.

The Harness application appears.

Step 6: Deploy Harness

In the KOTS admin console, in the Version history tab, click Deploy. The new version is displayed in Deployed version.

In a new browser tab, go to the following URL, replacing <LB_URL> with the URL you entered in the Load Balancer URL setting in the KOTS admin console:

<LB_URL>/#/onprem-signup

For example:

http://35.194.31.219/#/onprem-signup

The Harness sign up page appears.

Sign up with a new account and then log in. Your new account will be added to the Harness Account Administrators User Group.

See Managing Users and Groups (RBAC).

Important Next Steps

Important: You cannot invite other users to Harness until a Harness Delegate is installed and a Harness SMTP Collaboration Provider is configured.
  1. Install the Harness Delegate: Delegate Installation and Management.
  2. Set up an SMTP Collaboration Provider in Harness for email notifications from the Harness Manager: Add SMTP Collaboration Provider.
    Ensure you open the correct port for your SMTP provider, such as Office 365.
  3. Add a Harness Secrets Manager. By default, On-Prem installations use the local Harness MongoDB for the default Harness Secrets Manager. This is not recommended.After On-Prem installation, configure a new Secret Manager (Vault, AWS, etc). You will need to open your network for the Secret Manager connection.

Updating Harness

Please follow these steps to update your Harness On-Prem installation.

The steps are very similar to how you installed Harness initially.

For more information on updating KOTS and applications, see Using CLI and Updating the Admin Console from KOTS.

Disconnected (Airgap)

The following steps require a private registry, just like the initial installation of Harness.

Upgrade Harness
  1. Download the latest release from Harness.
  2. Run the following command on the cluster hosting Harness, replacing the placeholders:
kubectl kots upstream upgrade harness \ 
--airgap-bundle <path to harness-<version>.airgap> \
--kotsadm-namespace harness-kots \
--kotsadm-registry <private.registry.host>/harness \
--registry-username <username> \
--registry-password <password> \
--deploy \
-n harness
Upgrade KOTS Admin Tool

To upgrade the KOTS admin tool, first you will push images to your private Docker registry.

  1. Run the following command to push the images, replacing the placeholders:
kubectl kots admin-console push-images ./<new-kotsadm>.tar.gz \
<private.registry.host>/harness \
--registry-username rw-username \
--registry-password rw-password
  1. Next, run the following command on the cluster hosting Harness, replacing the placeholders:
kubectl kots admin-console upgrade \ 
--kotsadm-registry <private.registry.host>/harness \
--registry-username rw-username \
--registry-password rw-password \
-n harness

Connected

The following steps require a secure connection to the Internet, just like the initial installation of Harness.

Upgrade Harness
  1. Run the following command on the cluster hosting Harness:
kubectl kots upstream upgrade harness --deploy -n harness
Upgrade KOTS Admin Tool
  1. Run the following command on the cluster hosting Harness:
kubectl kots admin-console upgrade -n harness

Monitoring Harness

Harness monitoring is performed using the built in monitoring tools.

For steps on using the monitoring tools, see Prometheus from KOTS.

License Expired

If your license has expired, you will see something like the following:

Contact your Harness Customer Success representative or support@harness.io.

Notes

Harness On-Prem installations do not currently support the Harness Helm Delegate.


How did we do?