On-Prem Existing Kubernetes Cluster Setup

Updated 6 days ago by Michael Cretzman

This topic covers installing Harness Kubernetes On-Prem in an existing Kubernetes cluster.

We assume that you are very familiar with Kubernetes, and can perform the standard Kubernetes and managing configurations using Kustomize overlays.

Harness On-Prem uses the KOTS kubectl plugin for installation. This topic covers installing KOTS in your existing cluster as part of setting up Harness On-Prem.

Installing Harness On-Prem into an existing Kubernetes cluster is a simple process where you prepare your existing cluster and network, and use the KOTS admin tool and Kustomize to complete the installation and deploy Harness.

In this topic:

Cluster Requirements

Do not perform any of the steps in this topic until you have set up the requirements in the Existing Cluster On-Prem Infrastructure Requirements topic.

Summary

Installing Harness in an existing cluster is performed as a KOTS Existing Cluster Online Install.

This simply means that you are using an existing Kubernetes cluster, as opposed to bare metal or VMs, and that your cluster can make outbound internet requests for an online installation.

Step 1: Set up Cluster Requirements

As stated earlier, follow the steps in the Existing Cluster On-Prem Infrastructure Requirements topic to ensure you have your cluster set up correctly.

These requirements also include RBAC settings that might require your IT administrator to assist you unless your user account is bound to the cluster-admin Cluster Role.

Specifically, you need to create a KOTS admin Role and bind it to the user that will install Harness. You also need to create a Harness ClusterRole.

Step 2: Set Up Networking Requirements

Perform the following steps to ensure that you have the Load Balancer set up for Harness On-Prem.

Later, when you set up the kustomization for Harness On-Prem, you will provide an IP address for the cluster load balancer settings.

Finally, when you configure the Harness On-Prem application, you will provide the Load Balancer URL. This URL is what Harness On-Prem users will use.

Using NodePort?

If you are creating the load balancer's Service type using NodePort, create a load balancer that points to any port in range 30000-32767 on the node pool on which the Kubernetes cluster is running.

If you are using NodePort, you can skip to Step 3: Install KOTS.

Set Up Static External IP

You should have a static IP reserved to expose Harness outside of the Kubernetes cluster.

For example, in the GCP console, click VPC network, and then click External IP Addresses.

For more information, see Reserving a static external IP address.

For GCP, the External IP address must be Premium Tier.

Set Up DNS

Set up DNS to resolve the domain name you want to use for Harness On-Prem to the static IP address you reserved in the previous step.

For example, the domain name harness.abc.com would resolve to the static IP:

host harness.abc.com
harness.abc.com has address 192.0.2.0

The above DNS setup can be tested by running host <domain_name>.

Step 3: Install KOTS

  1. Log into your cluster using a user account that is bound to the KOTS admin role you created in Existing Cluster On-Prem Infrastructure Requirements.
  2. Install the KOTS kubectl plugin using the following command:

curl https://kots.io/install | bash

The output of the command is similar to this:

Installing replicatedhq/kots v1.16.1
(https://github.com/replicatedhq/kots/releases/download/v1.16.1/kots_darwin_amd64.tar.gz)...
############################################# 100.0%#=#=-# #
############################################# 100.0%
Installed at /usr/local/bin/kubectl-kots

To test the installation, run this command:

kubectl kots --help

The KOTS help appears.

Now that KOTS is installed, you can install Harness On-Prem into your cluster.

Step 4: Install the KOTS Admin Tool

To install the KOTS Admin tool, enter the following command:

kubectl kots install harness

You are prompted to enter the namespace for the Harness installation. This is the namespace you created in Existing Cluster On-Prem Infrastructure Requirements.

In this documentation, we use the namespace harness.

In the terminal, it will look like this:

Enter the namespace to deploy to: harness
• Deploying Admin Console
• Creating namespace ✓
• Waiting for datastore to be ready ✓

Enter a password for the KOTS admin console and hit Enter. You will use this password to log into the KOTS admin tool.

The KOTS admin tool URL is provided:

Enter a new password to be used for the Admin Console: ••••••••
• Waiting for Admin Console to be ready ✓

• Press Ctrl+C to exit
• Go to http://localhost:8800 to access the Admin Console

Use the URL provided in the output to open the KOTS admin console in a browser.

Enter the password you provided earlier, and click Log In.

You might be prompted to allow a port-forward connection into the cluster.

Step 5: Upload Your Harness License

Once you are logged into the KOTS admin console, you can upload your Harness license.

Obtain the Harness license file from your Harness Customer Success contact or email support@harness.io.

Drag your license YAML file into the KOTS admin tool:

Next, upload the license file:

Now that license file is uploaded, you can install Harness.

Option 1: Download Harness using Airgap Bundle

Installing Harness using an airgap bundle requires a Docker registry. You provide the KOTS admin tool with your registry login information and the airgap bundle. KOTS expands the bundle, pushes it to your registry, and then pulls the bundle from the registry on behalf of the Kubernetes cluster.

The username and password you provide to the KOTS admin tool should have push and pull permissions on the registry.

You will download the bundle from Harness, upload it into a Docker registry, and then provide the registry hostname and login credentials in the KOTS admin console.

To see if your license supports an airgap bundle installation, open it and look for isSnapshotSupported: true.
  1. Get airgap bundle from Harness
  2. Drag the file into the Drag your airgap bundle here or choose a bundle to upload section.

  1. In Hostname, enter the repo hostname. For example, if the Docker login is docker login mycompany-harness.jfrog.io, then the hostname is mycompany-harness.jfrog.io.
  2. Enter the username and password for the registry log in. This account should have permissions to push and pull from the registry.
  3. In Registry Namespace, enter the namespace identified in your repository path. For example, in the following repo, the Repository Path is kots.
  4. Click Upload airgap bundle. Kots will unpack the bundle and upload it to your repo.

Next, you will update the Kubernetes manifest using a kustomize overlay.

Go to Step 6: Patch Resource Config using Overlay.

Following that, you will provide KOTS with the Harness configuration information (Load Balancer URL).

Option 2: Download Harness over the Internet

If you are installing Harness over the Internet, click the download Harness from the Internet link.

KOTS begins installing Harness into your cluster.

Next, you will update the Kubernetes manifest using a kustomize overlay.

Go to Step 6: Patch Resource Config using Overlay.

Following that, you will provide KOTS with the Harness configuration information (Load Balancer URL). 

Step 6: Patch Resource Config using Overlay

Next, you will see the Configure Harness page.

Before you can provide the Load Balancer URL for Harness, you need to update the Harness manifests using kustomize.

  1. If you are using a single terminal, close the KOTS admin tool (Ctrl+C).
  2. Ensure kubectl is pointing to the cluster.
  3. Run the following command:
kubectl kots download --namespace harness --slug harness
This example assumes we are installing Harness in a namespace named harness. Please change the namespace according to your configuration.

This command will download a folder named harness in your current directory.

  1. In the harness folder, open the file kustomization.yaml:
vi harness/overlays/midstream/kustomization.yaml 
  1. In the patchesStrategicMerge section add nginx-service.yaml.
  2. Save the file.
  3. Next, we will define the load balancer Service.

In the same folder, midstream, create a new file named nginx-service.yaml:

vi harness/overlays/midstream/nginx-service.yaml

Edit the file using one of the following options.

Any annotation can be added in metadata.annotations.

Option: Create LoadBalancer

Edit nginx-service.yaml by pasting the following YAML:

apiVersion: v1
kind: Service
metadata:
name: nginx-ingress-controller-chart-controller
annotations:
harness.io/managed: "true"
#You can add custom annotations here

spec:
type: LoadBalancer
loadBalancerIP: <static IP reserved earlier>

In loadBalancerIP, enter the static IP address reserved in Step 2: Set Up Networking Requirements.

On some cloud platforms, like GCP, you can omit the IP address and GCP will generate one automatically. For simplicity, we recommend that you enter the static IP address you reserved here, instead. However, you can let the cloud platform generate an IP and then use it for your load balancer after Harness is installed.

Option: Create NodePort

Edit nginx-service.yaml by pasting the following YAML:

apiVersion: v1
kind: Service
metadata:
name: nginx-ingress-controller-chart-controller
annotations:
harness.io/managed: "true"
#You can add custom annotations here

spec:
type: NodePort
ports:
# By default and for convenience, the `targetPort` is set to the same value as the `port` field.
- port: 80
# Optional field
# By default and for convenience, the Kubernetes control plane will allocate a port from a range (default: 30000-32767)
nodePort: 30007

Review: Configuring Load Balancer Overlay Post-Installation

If you are changing the load balancer settings after you have already installed Harness, you need to delete the Harness Manager pods. Kubernetes will recreate them automatically and they will pick up the changes.

Find the name of Harness manager pods:

kubectl get pods -o=name -n harness | grep harness-manager

We use the namespace harness in this example.

The output will be something like this:

pod/harness-manager-79fdf979f7-5gpfj
pod/harness-manager-79fdf979f7-l7fcc

You can also find these in your cloud platform's console:

In the following command, replace <pod_name> with each pod name and run the command:

kubectl delete pod <pod_name> <pod_name> -n harness

For example:

kubectl delete pod harness-manager-79fdf979f7-5gpfj harness-manager-79fdf979f7-l7fcc -n harness

Upload the Load Balancer Settings

  1. Run the following command to upload the Kubernetes manifests from the local filesystem to create a new version of the application:
kubectl kots upload --namespace harness --slug harness ./harness
• Uploading local application to Admin Console ✓

Now your load balancer configuration is uploaded to the KOTS admin tool as part of a new Harness version.

Step 7: Configure Harness

Now that you have uploaded the new manifests, you can configure Harness.

  1. Point kubectl to the cluster where Harness is deployed and run the following command:

kubectl kots admin-console --namespace harness

  1. In the KOTS admin tool, go to Version History. You will see a new version of Harness with Source: Config Change displayed.
  2. Select the version and click Configure. The Configure Harness settings appear.

Kubernetes On-Prem Existing Cluster requires that you provide a Storage Class name and Load Balancer URL.

The name of the Storage Class depends on the provider where you are hosting your Kubernetes cluster. See Storage Classes from Kubernetes. You can just run the following command to get the Storage Classes in the namespace (in this example, harness):

kubectl get storageclass -n harness

Enter the name of the Storage Class.

In Load Balancer URL, you can enter the IP address of the LoadBalancer Service, such as http://35.194.31.219.

This is the static IP address you reserved and provided when setting up the kustomization.

To view and get the load balancer IP address in your cluster, run the following command:

kubectl get svc -n harness

You will see the the Load Balancer IP address in the harness-ingress-controller row under EXTERNAL-IP in the output:

NAME			    TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
delegate-proxy ClusterIP 10.44.4.186 <none> 80/TCP 2m50s
harness-ingress-controller LoadBalancer 10.44.5.226 35.194.31.219 80:30068/TCP 2m50s
harness-manager ClusterIP 10.44.0.234 <none> 9090/TCP 2m50s
harness-ui ClusterIP 10.44.9.247 <none> 80/TCP 2m50s
kotsadm ClusterIP 10.44.4.244 <none> 3000/TCP 3d

Enter the IP address in Load Balancer URL, using http://. For example:

http://35.194.31.219

When you are done, click Continue.

Option: Advanced Configurations

In the Advanced Configurations section, there are a number of advanced settings you can configure. If this is the first time you are setting up Harness On-Prem, there's no reason to fine tune the installation with these settings.

You can change the settings later in the KOTS admin console's Config tab:

Step 8: Perform Preflight Checks

Preflight checks run automatically and verify that your setup meets the minimum requirements.

You can skip these checks, but we recommend you let them run.

Fix any issues in the preflight steps. A common example is the message:

Your cluster meets the minimum version of Kubernetes, but we recommend you update to 1.15.0 or later.

You can update your cluster's version of Kubernetes if you like.

When you are finished pre-flight checks, click Continue.

The Harness application appears.

Step 9: Deploy Harness

In the KOTS admin console, in the Version history tab, click Deploy. The new version is displayed in Deployed version.

In a new browser tab, go to the following URL, replacing <LB_URL> with the URL you entered in the Load Balancer URL setting in the KOTS admin console:

<LB_URL>/#/onprem-signup

For example:

http://35.194.31.219/#/onprem-signup

The Harness sign up page appears.

Sign up with a new account and then log in. Your new account will be added to the Harness Account Administrators User Group.

See Managing Users and Groups (RBAC).

Important Next Steps

Important: You cannot invite other users to Harness until a Harness Delegate is installed and a Harness SMTP Collaboration Provider is configured.
  1. Install the Harness Delegate: Delegate Installation and Management.
  2. Set up an SMTP Collaboration Provider in Harness for email notifications from the Harness Manager: Add SMTP Collaboration Provider.
    Ensure you open the correct port for your SMTP provider, such as Office 365.
  3. Add a Harness Secrets Manager. By default, On-Prem installations use the local Harness MongoDB for the default Harness Secrets Manager. This is not recommended.After On-Prem installation, configure a new Secret Manager (Vault, AWS, etc). You will need to open your network for the Secret Manager connection.

Updating Harness

Updating Harness over the Internet follows the standard KOTS updating method described in Updating a KOTS application from KOTS.

Please follow those steps to update your Harness On-Prem installation.

Updating an air gapped installation is the same as the original installation. Harness provides you with a new airgap bundle, which you drag into the Kots admin console.

Monitoring Harness

Harness monitoring is performed using the built in monitoring tools.

For steps on using the monitoring tools, see Prometheus from KOTS.

License Expired

If your license has expired, you will see something like the following:

Contact your Harness Customer Success representative or support@harness.io.

Notes

Harness On-Prem installations do not currently support the Harness Helm Delegate.


How did we do?