Implementing a CAPI IPAM Provider


Implementing a CAPI IPAM Provider

One of the oldest problems of Cluster API and Kubernetes cluster creation is managing the Nodes IPs.

While it is possible to rely on DHCP for the node IP allocation, and some other cloud providers have their own way of managing the IPs, for some enterprises and onpremises environments the dynamic IP approach doesn’t applies, and there’s a need for a more strict control on “who is using what IP”.

Additionally, using dynamic IPs may be challenging, as for instance:

  • Some DHCP implementations are not strict on the process of renewal
  • The Kubernetes APIServer endpoint may be static, relying on a static IP or an FQDN associated with an IP
  • Implementing a LoadBalancer controller that uses DHCP is kind of…problematic and complex (but kube-vip does it!!!)

The only other way of doing on Cluster API was to use static IPs on the Machine spec, but this approach reduces the possibility of templating and automating the Cluster creation.

Note: The source code for this PoC is available on

This first part of the blog post will just implement the IPAM Provider :)

I will probably update it to do some tests against CAPV soon :)

The introduction of Cluster API IPAM

With the challenges above, the community created a new concept on Cluster API core called “IPAM Provider” that allows a user to specify “where should my nodes get their IPs from”.

The following new resources where introduced:

  • - A request for a new IP address
    • Contains just an object reference, for “which pool should the IP be allocated”
  • - The allocated IP Address
    • Contains a “response” from the IPAM Provider, with the allocated IP address, mask and gateway

So now, basically once you have an IPAM provider running on your cluster, you can create “IPAddressClaims” and receive “IPAddress” back.

So with an IPAM provider running on the cluster, every new Node creation now should, instead of relying on DHCP, request for a new IP address claim, and use the response IPAddress as the node IP.

participant A as Machine
participant B as Infrastructure Provider
participant K as Kubernetes API
participant D as IPAM Provider
  A-->>B: Reconcile Machine request
  B->>K: Create a new claim with a Pool reference
  Note over B, K: IPAddressClaim
  K-->>D: Reconcile IPAddress Claim
  Note over K, D: IPAddressClaim 
  D-->D: Allocate IP based on Pool configuration
  D->>K: Create the allocated IPAddress
  Note over D, K: IPAddress
  D->>A: Add allocated IP Address to underlying infrastructure

Developing a new IPAM Provider

Based on the workflow below (and some additional assumptions), we need to support at least the following workflows:

  • Allow a cluster admin to create a new IP pool to be consumed by the infrastructure provider
  • Allow users (or infrastructure provider) to allocate a new IP address from the configured IPPool, to be added to the nodes (or consumed somewhere else)
  • Allow users (or infrastructure provider) to deallocate previously allocated IP addresses

Besides this seems a “simple” workflow, there’s a bunch of logics behind to make it work and adhere with Cluster API contracts.

With this complexity in mind, Cluster API community did a “Reference and Generic implementation” called InCluster provider that can be used both as a real IPAM implementation, as also as a reference library for other providers.

We will be using this library for our example

Our IPAM software

For our implementation, we will use phpIPAM. It is a widely used IPAM, and it can be ran easily with docker-compose. Also, phpIPAM has a simple API that can be used for our needs.

We will not cover phpIPAM installation, but it should be straightforward to get it running.

In the end of phpIPAM installation and configuration we should have a subnet configured and ready to be used as:

Note The API management needs to be enabled on phpIPAM, under Administration / phpIPAM settings. Then a new API Key should be created under Administration / API, with the type “User token”. Note2 If you are using docker-compose, inside the container, on the file /phpipam/config.dist.php change the directive api_allow_unsafe to true

Creating our phpIPAM Go client

As we will need to allocate and deallocate IPAddress, let’s write our phpIPAM Client:

  1package ipamclient
  3import (
  4	"fmt"
  6	""
  7	""
  8	""
  9	""
 10	""
 13type IPAMClient struct {
 14	subnetid int
 15	ctrl     *addresses.Controller
 16	subctrl  *subnets.Controller
 19type Subnet struct {
 20	Mask    string `json:"mask,omitempty"`
 21	Gateway struct {
 22		IPAddress string `json:"ip_addr,omitempty"`
 23	} `json:"gateway,omitempty"`
 26type addrId struct {
 27	ID        int    `json:"id,omitempty"`
 28	SubnetID  int    `json:"subnetId,omitempty"`
 29	IPAddress string `json:"ip,omitempty"`
 32// We create our new client with all the controllers already encapsulated
 33// TODO: Should support HTTPs and skip insecure :)
 34func NewIPAMClient(cfg phpipam.Config, subnetid int) *IPAMClient {
 35	sess := session.NewSession(cfg)
 36	return &IPAMClient{
 37		subnetid: subnetid,
 38		ctrl:     addresses.NewController(sess),
 39		subctrl:  subnets.NewController(sess),
 40	}
 43func (i *IPAMClient) GetAddress(hostname string) (string, error) {
 44	myaddr := make([]addrId, 0)
 46    // We just return a new address if it doesn't already exists
 47	err := i.ctrl.SendRequest("GET", fmt.Sprintf("/addresses/search_hostname/%s", hostname), &struct{}{}, &myaddr)
 48	if err == nil && len(myaddr) > 0 && myaddr[0].SubnetID == i.subnetid {
 49		return myaddr[0].IPAddress, nil
 50	}
 52	addr, err := i.ctrl.CreateFirstFreeAddress(i.subnetid, addresses.Address{Description: hostname, Hostname: hostname})
 53	if err != nil {
 54		return "", err
 55	}
 56	return addr, nil
 59func (i *IPAMClient) ReleaseAddress(hostname string) error {
 60	// The library is broken on addrStruct so we need a simple one just to get the allocated ID
 61	// TODO: Improve error handling, being able to check if the error is something like "not found"
 62	myaddr, err := i.searchForAddress(hostname)
 63	if err != nil {
 64		return fmt.Errorf("failed to find the address, maybe it doesn't exist anymore? %w", err)
 65	}
 67	_, err = i.ctrl.DeleteAddress(myaddr.ID, false)
 68	if err != nil {
 69		return err
 70	}
 71	return nil
 74func (i *IPAMClient) GetSubnetConfig() (*Subnet, error) {
 75	var subnet Subnet
 76	err := i.subctrl.SendRequest("GET", fmt.Sprintf("/subnets/%d/", i.subnetid), &struct{}{}, &subnet)
 77	if err != nil {
 78		return nil, err
 79	}
 80	return &subnet, nil
 83func (i *IPAMClient) searchForAddress(hostname string) (*addrId, error) {
 84	myaddr := make([]addrId, 0)
 86	err := i.ctrl.SendRequest("GET", fmt.Sprintf("/addresses/search_hostname/%s", hostname), &struct{}{}, &myaddr)
 87	if err == nil && len(myaddr) > 0 && myaddr[0].SubnetID == i.subnetid {
 88		return &myaddr[0], nil
 89	}
 90	return nil, err
 93func SpecToClient(spec *v1alpha1.PHPIPAMPoolSpec) (*IPAMClient, error) {
 94	if spec == nil {
 95		return nil, fmt.Errorf("spec cannot be null")
 96	}
 98	if spec.SubnetID < 0 || spec.Credentials == nil {
 99		return nil, fmt.Errorf("subnet id and credentials are required")
100	}
102	return NewIPAMClient(phpipam.Config{
103		AppID:    spec.Credentials.AppID,
104		Username: spec.Credentials.Username,
105		Password: spec.Credentials.Password,
106		Endpoint: spec.Credentials.Endpoint,
107	}, spec.SubnetID), nil
110This library will be responsible to allocate / deallocate addresses and get the right configurations from our subnet.
111**Note** This code may have changed over time, look at the repo for the latest available version
112### Writing the CAPI IPAM controller
113Writing the IPAM controller starts with the definition of the Pool API and controller, which will be the resource where the cluster admin will define the IPAM software configuration.
115After that, CAPI IPAM requires two more implementations: the Claim Handler, responsible for getting and releasing an IP address from our IPAM, and the IPAM Adapter, that will serve as a middle layer between CAPI IPAM generic controller and our specific IPAM implementation. 
117#### The Pool controller
118As part of the implementation, we need a "way" to let the controller know which IPAddress it owns, and also how it should allocate IP Addresses and communicate with the underlying IPAM. Let's take as an example the definition of an in-cluster IPPool:
122kind: InClusterIPPool
124  name: inclusterippool-sample
126  addresses:
127    -
128  prefix: 24
129  gateway:

Looking at this IPPool example, we can see the definition of “how the controller should behave when allocating IPs”. Later on, we refer inclusterippool-sample when requesting new IPs.

For phpIPAM we have already configured our IPPool, but we need to tell the controller some other informations, like “what credentials to use” and “what subnet should be consumed”. Something as below (not ideal, I know, but good for our experiment)

 2kind: PHPIPAMPool
 4  name: mypool
 6  subnetid: 7 # You get this from phpipam UI
 7  appid: capi123 # Created with the APIKey when configuring PHPIPAM
 8  username: admin # You shouldn't do it, but I'm lazy!
 9  password: password
10  endpoint:

I’m not going to write the whole go code for the API spec, but it should be available on Github. On the implementation of the PHPIPAMIPPool reconciliation, what matters to us is:

 1// Reconcile the IPPool and set it as ready
 2func (r *PHPIPAMIPPoolReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 4	ippoollogger = log.FromContext(ctx).WithName("PHPIPAM ippool")
 5	ippoollogger.Info("received reconciliation", "request", req.NamespacedName.String())
 7	var ippool ipamv1alpha1.PHPIPAMIPPool
 8	if err := r.Get(ctx, req.NamespacedName, &ippool); err != nil {
 9		if k8serrors.IsNotFound(err) {
10			return ctrl.Result{}, nil
11		}
12		ippoollogger.Error(err, "unable to get ippool")
13		return ctrl.Result{}, err
14	}
16	ipamcl, err := ipamclient.SpecToClient(&ippool.Spec)
17	if err != nil {
18		return r.ConditionsWithErrors(ctx, req, &ippool, ipamv1alpha1.ConditionReasonInvalidPHPIPam, "PHPIPAMconfig configuration is invalid: "+err.Error(), true)
20	}
22	subnetCfg, err := ipamcl.GetSubnetConfig()
23	if err != nil {
24		return r.ConditionsWithErrors(ctx, req, &ippool, ipamv1alpha1.ConditionReasonInvalidCreds, "failed to login to phpipam: "+err.Error(), false)
25	}
26	ippool.Status.Gateway = subnetCfg.Gateway.IPAddress
27	ippool.Status.Mask = subnetCfg.Mask
28	r.ipamcl = ipamcl
29	return r.SetReady(ctx, req, &ippool)

After installing the CRDs, building and running the controller, we can create the following IPPool and check if the controller is able to reconcile it:

 2kind: PHPIPAMIPPool
 4  name: my-ipam-test
 6  subnetid: 7
 7  credentials:
 8    username: admin
 9    password: "12qw!@QW"
10    app_id: capi123
11    endpoint: ""

And getting its status should return:

2  conditions:
3  - lastTransitionTime: "2024-02-20T17:15:05Z"
4    message: IPPool is ready
5    reason: IPPoolReady
6    status: "True"
7    type: Ready
8  gateway:
9  mask: "24"

The Claim Handler

The Claim Handler is the responsible for the real communication between the controller and the IPAM software.

It will be called for every new IPAddress/IPAddressClaim reconciliation

This interface implementation requires that 3 methods are defined:

  • A method called “FetchPool” that will be the first method called, to get the Pool requested from IPAddressClaim and populate the required structures
  • A method called EnsureAddress that is responsible to check or allocate a new address for the claim
  • A method called ReleaseAddress that is responsible to release the address back to the Pool

The handler implementation will look something like:

 1package ipaddress
 3import (
 4	"context"
 5	"fmt"
 6	"strconv"
 8	""
 9	""
10	""
11	ipamv1 ""
12	ctrl ""
13	""
15	""
16	""
19// IPAddressClaimHandler reconciles an IPAddress Claim getting the right address from the right pool
20type IPAddressClaimHandler struct {
21	client.Client
22	claim   *ipamv1.IPAddressClaim
23	mask    int
24	gateway string
25	ipamcl  *ipamclient.IPAMClient
28var _ ipamutil.ClaimHandler = &IPAddressClaimHandler{}
30// FetchPool fetches the PHPIPAM Pool.
31func (h *IPAddressClaimHandler) FetchPool(ctx context.Context) (client.Object, *ctrl.Result, error) {
33	var err error
34	phpipampool := &v1alpha1.PHPIPAMIPPool{}
36	if err = h.Client.Get(ctx, types.NamespacedName{Namespace: h.claim.Namespace, Name: h.claim.Spec.PoolRef.Name}, phpipampool); err != nil {
37		return nil, nil, errors.Wrap(err, "failed to fetch pool")
38	}
40	if phpipampool.Status.Mask == "" || phpipampool.Status.Gateway == "" || !v1alpha1.PoolHasReadyCondition(phpipampool.Status) {
41		return nil, nil, fmt.Errorf("IPPool is not ready yet")
42	}
44	h.mask, err = strconv.Atoi(phpipampool.Status.Mask)
45	if err != nil {
46		return nil, nil, fmt.Errorf("pool contains invalid network mask")
47	}
48	h.gateway = phpipampool.Status.Gateway
50    ipamcl, err := ipamclient.SpecToClient(&phpipampool.Spec)
51	if err != nil {
52		return nil, nil, err
53	}
54	h.ipamcl = ipamcl
55	return phpipampool, nil, nil
58// EnsureAddress ensures that the IPAddress contains a valid address.
59func (h *IPAddressClaimHandler) EnsureAddress(ctx context.Context, address *ipamv1.IPAddress) (*ctrl.Result, error) {
60	hostname := fmt.Sprintf("%s.%s", h.claim.GetName(), h.claim.GetNamespace())
61	ipv4, err := h.ipamcl.GetAddress(hostname)
62	if err != nil {
63		return nil, errors.Wrap(err, "failed to get an IP Address")
64	}
66	address.Spec.Address = ipv4
67	address.Spec.Gateway = h.gateway
68	address.Spec.Prefix = h.mask
69	return nil, nil
72// ReleaseAddress releases the ip address.
73func (h *IPAddressClaimHandler) ReleaseAddress(ctx context.Context) (*ctrl.Result, error) {
74	hostname := fmt.Sprintf("%s.%s", h.claim.GetName(), h.claim.GetNamespace())
75	err := h.ipamcl.ReleaseAddress(hostname)
76	return nil, err

The Provider Adapter

The Provider Adapter implements the middle layer between the generic IPAM reconciler and our specific IPAM Handler.

Its function is to, first allow the Generic controller to setup a new manager for the IPAddressClaim and IPAddress and also, to get the proper ClaimHandler.

The Provider Adapter also has some specific needs (like setting a new Index and some common functions) that will not be part of the snippet, and hopefully we can integrate and migrate into the Generic reconciler

 1// PHPIPAMProviderAdapter is used as middle layer for provider integration.
 2type PHPIPAMProviderAdapter struct {
 3	Client     client.Client
 4	IPAMClient *ipamclient.IPAMClient
 7var _ ipamutil.ProviderAdapter = &PHPIPAMProviderAdapter{}
 9// SetupWithManager sets up the controller with the Manager.
10func (v *PHPIPAMProviderAdapter) SetupWithManager(_ context.Context, b *ctrl.Builder) error {
11	b.
12		For(&ipamv1.IPAddressClaim{}, builder.WithPredicates(
13			ipampredicates.ClaimReferencesPoolKind(metav1.GroupKind{
14				Group: v1alpha1.GroupVersion.Group,
15				Kind:  v1alpha1.PHPIPAMPoolKind,
16			}),
17		)).
18		WithOptions(controller.Options{
19			// To avoid race conditions when allocating IP Addresses, we explicitly set this to 1
20			MaxConcurrentReconciles: 1,
21		}).
22		Watches(
23			&v1alpha1.PHPIPAMIPPool{},
24			handler.EnqueueRequestsFromMapFunc(v.IPPoolToIPClaims()),
25			builder.WithPredicates(resourceTransitionedToUnpaused()),
26		).
27		Owns(&ipamv1.IPAddress{}, builder.WithPredicates(
28			ipampredicates.AddressReferencesPoolKind(metav1.GroupKind{
29				Group: v1alpha1.GroupVersion.Group,
30				Kind:  v1alpha1.PHPIPAMPoolKind,
31			}),
32		))
33	return nil
36// ClaimHandlerFor returns a claim handler for a specific claim.
37func (v *PHPIPAMProviderAdapter) ClaimHandlerFor(_ client.Client, claim *ipamv1.IPAddressClaim) ipamutil.ClaimHandler {
38	return &IPAddressClaimHandler{
39		Client: v.Client,
40		claim:  claim,
41	}

As we can see, it is a simple middle layer between a reconciler and the real IPAM logic code.

Testing if it works

Putting all together, and with the proper “main.go” file (look at the repo!), we can test if the IP allocation works

Note Don’t forget to install the Cluster API core CRDs first with:

1kubectl apply -f
2kubectl apply -f
3kubectl apply -f

With the controller running and able to reach phpIPAM, and with the PHPIPAMIPPool properly defined, we can create a new IPAddressClaim using our pool, and check if an IP Address is allocated:

 2kind: IPAddressClaim
 4  name: first-ip
 5  namespace: default
 7  poolRef:
 8    apiGroup:
 9    kind: PHPIPAMIPPool
10    name: my-ipam-test

After we apply this object, the controller should go to phpIPAM and gets us in return an IPAddress with the same name and an IP allocated, which we can later verify on phpIPAM:

 1# kubectl get ipaddress first-ip -o yaml
 2kind: IPAddress
 4  name: first-ip
 5  namespace: default
 8  address: # This was filled by our controller
 9  claimRef:
10    name: first-ip
11  gateway:
12  poolRef:
13    apiGroup:
14    kind: PHPIPAMIPPool
15    name: my-ipam-test
16  prefix: 24

And on phpIPAM:

Allocated IPs