ARO-RP

Azure Red Hat OpenShift Resource Provider

Welcome!

For information relating to the generally available Azure Red Hat OpenShift v4 service, please see the following links:

• https://azure.microsoft.com/en-us/services/openshift/
• https://www.openshift.com/products/azure-openshift
• https://docs.microsoft.com/en-us/azure/openshift/
• https://docs.openshift.com/aro/4/welcome/index.html

Quickstarts

• If you are an end user and want to create an Azure Red Hat OpenShift 4 cluster, follow Create, access, and manage an Azure Red Hat OpenShift 4 Cluster.

• If you want to deploy a development RP, follow deploy development RP.

Contributing

Azure Red Hat OpenShift (ARO) is a jointly engineered, managed, and supported service from Microsoft and Red Hat, delivered as an Azure service with shared responsibilities between Microsoft, Red Hat, and customers. The shared responsibility model is documented in the Azure Red Hat OpenShift responsibility assignment matrix.

This repository is the public home for the ARO resource provider code, but product development is primarily driven by Microsoft and Red Hat engineering teams and is not run as a traditional community‑driven open source project. Pull requests are usually created and reviewed in internal systems, then synchronized and published here, rather than being accepted directly from the public.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Development

• The main development workflow, CI/CD, and release management are operated by Microsoft and Red Hat employees.
• Changes are authored, reviewed, and tested internally, then mirrored to this repository as pull requests and commits.

Support and feedback

For customers and partners, the preferred way to request help is through official support channels, not via pull requests on this repository.

Recommended options:

• Azure portal: In the Azure portal, use Help + support → Create a support request for issues related to your ARO resources.
• Red Hat Customer Portal: If applicable, you can also open a case via the Red Hat Customer Portal, where Microsoft and Red Hat jointly handle ARO support.

For non‑urgent questions or suggestions, you may open an issue in this repository or use your existing Microsoft or Red Hat account channels.

References:

• https://learn.microsoft.com/en-us/azure/openshift/openshift-service-definitions
• https://azure.microsoft.com/en-us/support/create-ticket

Repository map

• .pipelines: CI workflows using Azure pipelines.

• cmd/aro: RP entrypoint.

• deploy: ARM templates to deploy RP in development and production.

• docs: Documentation.

• hack: Build scripts and utilities.

• pkg: RP source code:

  • pkg/api: RP internal and external API definitions.

  • pkg/backend: RP backend workers.

  • pkg/bootstraplogging: Bootstrap logging configuration

  • pkg/client: Autogenerated ARO service Go client.

  • pkg/cluster: Cluster create/update/delete operations wrapper for OCP installer.

  • pkg/database: RP CosmosDB wrapper layer.

  • pkg/deploy: /deploy ARM template generation code.

  • pkg/env: RP environment-specific shims for running in production, development or test

  • pkg/frontend: RP frontend webserver.

  • pkg/metrics: Handles RP metrics via statsd.

  • pkg/mirror: OpenShift release mirror tooling.

  • pkg/monitor: Monitors running clusters.

  • pkg/operator/controllers: A list of controllers instantiated by the operator component.

    • alertwebhook: Ensures that the receiver endpoint defined in the alertmanager-main secret matches the webserver endpoint at aro-operator-master.openshift-azure-operator:8080, to avoid the AlertmanagerReceiversNotConfigured warning.

    • checker: Watches the Cluster resource for changes and updates conditions of the resource based on checks mentioned below

      • internetchecker: validate outbound internet connectivity to the nodes

      • serviceprincipalchecker: validate cluster service principal has the correct role/permissions

    • clusteroperatoraro: Ensures that the ARO cluster object is consistent and immutable

    • dnsmasq: Ensures that a dnsmasq systemd service is defined as a machineconfig for all nodes. The dnsmasq config contains records for azure load balancers such as api, api-int and *.apps domains so they will resolve even if custom DNS on the VNET is set.

    • genevalogging: Ensures all the Geneva logging resources in the openshift-azure-logging namespace matches the pre-defined specification found in pkg/operator/controllers/genevalogging/genevalogging.go.

    • imageconfig: Ensures that required registries are not blocked in image.config

    • machine: validate machine objects have the correct provider spec, vm type, vm image, disk size, three master nodes exist, and the number of worker nodes match the desired worker replicas

    • machineset: Ensures that a minimum of two worker replicas are met.

    • machinehealthcheck: Ensures the MachineHealthCheck resource is running as configured. See machinehealthcheck/doc.go

      • More information around the MHC CR can be found in openshift documentation of MHC

    • monitoring: Ensures that the OpenShift monitoring configuration in the openshift-monitoring namespace is consistent and immutable.

    • node: Force deletes pods when a node fails to drain for 1 hour. It should clear up any pods that refuse to be evicted on a drain due to violating a pod disruption budget.

    • pullsecret: Ensures that the ACR credentials in the openshift-config/pull-secret secret match those in the openshift/azure-operator/cluster secret.

    • rbac: Ensures that the aro-sre clusterrole and clusterrolebinding exist and are consistent.

    • routefix: Ensures all the routefix resources in the namespace openshift-azure-routefix remain on the cluster.

    • subnets: Ensures that the Network Security Groups (NSGs) are correct, and updates the Azure Machine Provider spec with subnet, vnet, and Network Resource Group.

    • workaround: Applies a set of temporary workarounds to the ARO cluster.

    • previewfeature: Allows toggling certain features that are not yet enabled by default.

  • pkg/portal: Portal for running promql queries against a cluster or requesting a kubeconfig for a cluster.

  • pkg/proxy: Proxy service for portal kubeconfig cluster access.

  • pkg/swagger: Swagger specification generation code.

  • pkg/util: Utility libraries.

• python: Autogenerated ARO service Python client and az aro client extension.

• swagger: Autogenerated ARO service Swagger specification.

• test: End-to-end tests.

Basic architecture

• pkg/frontend is intended to become a spec-compliant RP web server. It is backed by CosmosDB. Incoming PUT/DELETE requests are written to the database with an non-terminal (Updating/Deleting) provisioningState.

• pkg/backend reads documents with non-terminal provisioningStates, asynchronously updates them and finally updates document with a terminal provisioningState (Succeeded/Failed). The backend updates the document with a heartbeat - if this fails, the document will be picked up by a different worker.

• As CosmosDB does not support document patch, care is taken to correctly pass through any fields in the internal model which the reader is unaware of (see github.com/ugorji/go/codec.MissingFielder). This is intended to help in upgrade cases and (in the future) with multiple microservices reading from the database in parallel.

• Care is taken to correctly use optimistic concurrency to avoid document corruption through concurrent writes (see RetryOnPreconditionFailed).

• The pkg/api architecture differs somewhat from github.com/openshift/openshift-azure: the intention is to fix the broken merge semantics and try pushing validation into the versioned APIs to improve error reporting.

• Everything is intended to be crash/restart/upgrade-safe, horizontally scaleable, upgradeable...

Useful links

• https://github.com/AzureExpert/azure-resource-manager-rpc/tree/master

• https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/azure-resource-manager/management/overview.md

• https://github.com/cloud-and-ai-microsoft/resource-provider-contract

• https://github.com/microsoft/api-guidelines

• https://docs.microsoft.com/en-gb/rest/api/cosmos-db

• https://github.com/jewzaam/go-cosmosdb

• https://www.redhat.com/en/technologies/cloud-computing/openshift/azure