Implementing a CAPI IPAM Provider
Overview
Implementing a CAPI IPAM Provider
One of the oldest problems of Cluster API and Kubernetes cluster creation is managing the Nodes IPs.
While it is possible to rely on DHCP for the node IP allocation, and some other cloud providers have their own way of managing the IPs, for some enterprises and onpremises environments the dynamic IP approach doesn’t applies, and there’s a need for a more strict control on “who is using what IP”.
Additionally, using dynamic IPs may be challenging, as for instance:
- Some DHCP implementations are not strict on the process of renewal
- The Kubernetes APIServer endpoint may be static, relying on a static IP or an FQDN associated with an IP
- Implementing a LoadBalancer controller that uses DHCP is kind of…problematic and complex (but kube-vip does it!!!)
The only other way of doing on Cluster API was to use static IPs on the Machine spec, but this approach reduces the possibility of templating and automating the Cluster creation.
Note: The source code for this PoC is available on https://github.com/rikatz/capi-phpipam/
This first part of the blog post will just implement the IPAM Provider :)
I will probably update it to do some tests against CAPV soon :)
The introduction of Cluster API IPAM
With the challenges above, the community created a new concept on Cluster API core called “IPAM Provider” that allows a user to specify “where should my nodes get their IPs from”.
The following new resources where introduced:
- ipaddressclaims.ipam.cluster.x-k8s.io/v1alpha1 - A request for a new IP address
- Contains just an object reference, for “which pool should the IP be allocated”
- ipaddress.ipam.cluster.x-k8s.io/v1alpha1 - The allocated IP Address
- Contains a “response” from the IPAM Provider, with the allocated IP address, mask and gateway
So now, basically once you have an IPAM provider running on your cluster, you can create “IPAddressClaims” and receive “IPAddress” back.
So with an IPAM provider running on the cluster, every new Node creation now should, instead of relying on DHCP, request for a new IP address claim, and use the response IPAddress as the node IP.
sequenceDiagram participant A as Machine participant B as Infrastructure Provider participant K as Kubernetes API participant D as IPAM Provider A-->>B: Reconcile Machine request B->>K: Create a new claim with a Pool reference Note over B, K: IPAddressClaim K-->>D: Reconcile IPAddress Claim Note over K, D: IPAddressClaim D-->D: Allocate IP based on Pool configuration D->>K: Create the allocated IPAddress Note over D, K: IPAddress D->>A: Add allocated IP Address to underlying infrastructure
Developing a new IPAM Provider
Based on the workflow below (and some additional assumptions), we need to support at least the following workflows:
- Allow a cluster admin to create a new IP pool to be consumed by the infrastructure provider
- Allow users (or infrastructure provider) to allocate a new IP address from the configured IPPool, to be added to the nodes (or consumed somewhere else)
- Allow users (or infrastructure provider) to deallocate previously allocated IP addresses
Besides this seems a “simple” workflow, there’s a bunch of logics behind to make it work and adhere with Cluster API contracts.
With this complexity in mind, Cluster API community did a “Reference and Generic implementation” called InCluster provider that can be used both as a real IPAM implementation, as also as a reference library for other providers.
We will be using this library for our example
Our IPAM software
For our implementation, we will use phpIPAM. It is a widely used IPAM, and it can be ran easily with docker-compose. Also, phpIPAM has a simple API that can be used for our needs.
We will not cover phpIPAM installation, but it should be straightforward to get it running.
In the end of phpIPAM installation and configuration we should have a subnet configured and ready to be used as:
/phpipam/config.dist.php change the directive api_allow_unsafe to true
Creating our phpIPAM Go client
As we will need to allocate and deallocate IPAddress, let’s write our phpIPAM Client:
1package ipamclient
2
3import (
4 "fmt"
5
6 "github.com/pavel-z1/phpipam-sdk-go/controllers/addresses"
7 "github.com/pavel-z1/phpipam-sdk-go/controllers/subnets"
8 "github.com/pavel-z1/phpipam-sdk-go/phpipam"
9 "github.com/pavel-z1/phpipam-sdk-go/phpipam/session"
10 "github.com/rikatz/capi-phpipam/api/v1alpha1"
11)
12
13type IPAMClient struct {
14 subnetid int
15 ctrl *addresses.Controller
16 subctrl *subnets.Controller
17}
18
19type Subnet struct {
20 Mask string `json:"mask,omitempty"`
21 Gateway struct {
22 IPAddress string `json:"ip_addr,omitempty"`
23 } `json:"gateway,omitempty"`
24}
25
26type addrId struct {
27 ID int `json:"id,omitempty"`
28 SubnetID int `json:"subnetId,omitempty"`
29 IPAddress string `json:"ip,omitempty"`
30}
31
32// We create our new client with all the controllers already encapsulated
33// TODO: Should support HTTPs and skip insecure :)
34func NewIPAMClient(cfg phpipam.Config, subnetid int) *IPAMClient {
35 sess := session.NewSession(cfg)
36 return &IPAMClient{
37 subnetid: subnetid,
38 ctrl: addresses.NewController(sess),
39 subctrl: subnets.NewController(sess),
40 }
41}
42
43func (i *IPAMClient) GetAddress(hostname string) (string, error) {
44 myaddr := make([]addrId, 0)
45
46 // We just return a new address if it doesn't already exists
47 err := i.ctrl.SendRequest("GET", fmt.Sprintf("/addresses/search_hostname/%s", hostname), &struct{}{}, &myaddr)
48 if err == nil && len(myaddr) > 0 && myaddr[0].SubnetID == i.subnetid {
49 return myaddr[0].IPAddress, nil
50 }
51
52 addr, err := i.ctrl.CreateFirstFreeAddress(i.subnetid, addresses.Address{Description: hostname, Hostname: hostname})
53 if err != nil {
54 return "", err
55 }
56 return addr, nil
57}
58
59func (i *IPAMClient) ReleaseAddress(hostname string) error {
60 // The library is broken on addrStruct so we need a simple one just to get the allocated ID
61 // TODO: Improve error handling, being able to check if the error is something like "not found"
62 myaddr, err := i.searchForAddress(hostname)
63 if err != nil {
64 return fmt.Errorf("failed to find the address, maybe it doesn't exist anymore? %w", err)
65 }
66
67 _, err = i.ctrl.DeleteAddress(myaddr.ID, false)
68 if err != nil {
69 return err
70 }
71 return nil
72}
73
74func (i *IPAMClient) GetSubnetConfig() (*Subnet, error) {
75 var subnet Subnet
76 err := i.subctrl.SendRequest("GET", fmt.Sprintf("/subnets/%d/", i.subnetid), &struct{}{}, &subnet)
77 if err != nil {
78 return nil, err
79 }
80 return &subnet, nil
81}
82
83func (i *IPAMClient) searchForAddress(hostname string) (*addrId, error) {
84 myaddr := make([]addrId, 0)
85
86 err := i.ctrl.SendRequest("GET", fmt.Sprintf("/addresses/search_hostname/%s", hostname), &struct{}{}, &myaddr)
87 if err == nil && len(myaddr) > 0 && myaddr[0].SubnetID == i.subnetid {
88 return &myaddr[0], nil
89 }
90 return nil, err
91}
92
93func SpecToClient(spec *v1alpha1.PHPIPAMPoolSpec) (*IPAMClient, error) {
94 if spec == nil {
95 return nil, fmt.Errorf("spec cannot be null")
96 }
97
98 if spec.SubnetID < 0 || spec.Credentials == nil {
99 return nil, fmt.Errorf("subnet id and credentials are required")
100 }
101
102 return NewIPAMClient(phpipam.Config{
103 AppID: spec.Credentials.AppID,
104 Username: spec.Credentials.Username,
105 Password: spec.Credentials.Password,
106 Endpoint: spec.Credentials.Endpoint,
107 }, spec.SubnetID), nil
108}```
109
110This library will be responsible to allocate / deallocate addresses and get the right configurations from our subnet.
111**Note** This code may have changed over time, look at the repo for the latest available version
112### Writing the CAPI IPAM controller
113Writing the IPAM controller starts with the definition of the Pool API and controller, which will be the resource where the cluster admin will define the IPAM software configuration.
114
115After that, CAPI IPAM requires two more implementations: the Claim Handler, responsible for getting and releasing an IP address from our IPAM, and the IPAM Adapter, that will serve as a middle layer between CAPI IPAM generic controller and our specific IPAM implementation.
116
117#### The Pool controller
118As part of the implementation, we need a "way" to let the controller know which IPAddress it owns, and also how it should allocate IP Addresses and communicate with the underlying IPAM. Let's take as an example the definition of an in-cluster IPPool:
119
120```yaml
121apiVersion: ipam.cluster.x-k8s.io/v1alpha2
122kind: InClusterIPPool
123metadata:
124 name: inclusterippool-sample
125spec:
126 addresses:
127 - 10.0.0.0/24
128 prefix: 24
129 gateway: 10.0.0.1
Looking at this IPPool example, we can see the definition of “how the controller should behave when allocating IPs”. Later on, we refer inclusterippool-sample when requesting new IPs.
For phpIPAM we have already configured our IPPool, but we need to tell the controller some other informations, like “what credentials to use” and “what subnet should be consumed”. Something as below (not ideal, I know, but good for our experiment)
1apiVersion: ipam.cluster.x-k8s.io/v1alpha2
2kind: PHPIPAMPool
3metadata:
4 name: mypool
5spec:
6 subnetid: 7 # You get this from phpipam UI
7 appid: capi123 # Created with the APIKey when configuring PHPIPAM
8 username: admin # You shouldn't do it, but I'm lazy!
9 password: password
10 endpoint: http://127.0.0.1/api
I’m not going to write the whole go code for the API spec, but it should be available on Github. On the implementation of the PHPIPAMIPPool reconciliation, what matters to us is:
1// Reconcile the IPPool and set it as ready
2func (r *PHPIPAMIPPoolReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
3
4 ippoollogger = log.FromContext(ctx).WithName("PHPIPAM ippool")
5 ippoollogger.Info("received reconciliation", "request", req.NamespacedName.String())
6
7 var ippool ipamv1alpha1.PHPIPAMIPPool
8 if err := r.Get(ctx, req.NamespacedName, &ippool); err != nil {
9 if k8serrors.IsNotFound(err) {
10 return ctrl.Result{}, nil
11 }
12 ippoollogger.Error(err, "unable to get ippool")
13 return ctrl.Result{}, err
14 }
15
16 ipamcl, err := ipamclient.SpecToClient(&ippool.Spec)
17 if err != nil {
18 return r.ConditionsWithErrors(ctx, req, &ippool, ipamv1alpha1.ConditionReasonInvalidPHPIPam, "PHPIPAMconfig configuration is invalid: "+err.Error(), true)
19
20 }
21
22 subnetCfg, err := ipamcl.GetSubnetConfig()
23 if err != nil {
24 return r.ConditionsWithErrors(ctx, req, &ippool, ipamv1alpha1.ConditionReasonInvalidCreds, "failed to login to phpipam: "+err.Error(), false)
25 }
26 ippool.Status.Gateway = subnetCfg.Gateway.IPAddress
27 ippool.Status.Mask = subnetCfg.Mask
28 r.ipamcl = ipamcl
29 return r.SetReady(ctx, req, &ippool)
30}
After installing the CRDs, building and running the controller, we can create the following IPPool and check if the controller is able to reconcile it:
1apiVersion: ipam.cluster.x-k8s.io/v1alpha1
2kind: PHPIPAMIPPool
3metadata:
4 name: my-ipam-test
5spec:
6 subnetid: 7
7 credentials:
8 username: admin
9 password: "12qw!@QW"
10 app_id: capi123
11 endpoint: "http://127.0.0.1/api"
And getting its status should return:
1status:
2 conditions:
3 - lastTransitionTime: "2024-02-20T17:15:05Z"
4 message: IPPool is ready
5 reason: IPPoolReady
6 status: "True"
7 type: Ready
8 gateway: 192.168.0.1
9 mask: "24"
The Claim Handler
The Claim Handler is the responsible for the real communication between the controller and the IPAM software.
It will be called for every new IPAddress/IPAddressClaim reconciliation
This interface implementation requires that 3 methods are defined:
- A method called “FetchPool” that will be the first method called, to get the Pool requested from IPAddressClaim and populate the required structures
- A method called EnsureAddress that is responsible to check or allocate a new address for the claim
- A method called ReleaseAddress that is responsible to release the address back to the Pool
The handler implementation will look something like:
1package ipaddress
2
3import (
4 "context"
5 "fmt"
6 "strconv"
7
8 "github.com/pkg/errors"
9 "k8s.io/apimachinery/pkg/types"
10 "sigs.k8s.io/cluster-api-ipam-provider-in-cluster/pkg/ipamutil"
11 ipamv1 "sigs.k8s.io/cluster-api/exp/ipam/api/v1beta1"
12 ctrl "sigs.k8s.io/controller-runtime"
13 "sigs.k8s.io/controller-runtime/pkg/client"
14
15 "github.com/rikatz/capi-phpipam/api/v1alpha1"
16 "github.com/rikatz/capi-phpipam/pkg/ipamclient"
17)
18
19// IPAddressClaimHandler reconciles an IPAddress Claim getting the right address from the right pool
20type IPAddressClaimHandler struct {
21 client.Client
22 claim *ipamv1.IPAddressClaim
23 mask int
24 gateway string
25 ipamcl *ipamclient.IPAMClient
26}
27
28var _ ipamutil.ClaimHandler = &IPAddressClaimHandler{}
29
30// FetchPool fetches the PHPIPAM Pool.
31func (h *IPAddressClaimHandler) FetchPool(ctx context.Context) (client.Object, *ctrl.Result, error) {
32
33 var err error
34 phpipampool := &v1alpha1.PHPIPAMIPPool{}
35
36 if err = h.Client.Get(ctx, types.NamespacedName{Namespace: h.claim.Namespace, Name: h.claim.Spec.PoolRef.Name}, phpipampool); err != nil {
37 return nil, nil, errors.Wrap(err, "failed to fetch pool")
38 }
39
40 if phpipampool.Status.Mask == "" || phpipampool.Status.Gateway == "" || !v1alpha1.PoolHasReadyCondition(phpipampool.Status) {
41 return nil, nil, fmt.Errorf("IPPool is not ready yet")
42 }
43
44 h.mask, err = strconv.Atoi(phpipampool.Status.Mask)
45 if err != nil {
46 return nil, nil, fmt.Errorf("pool contains invalid network mask")
47 }
48 h.gateway = phpipampool.Status.Gateway
49
50 ipamcl, err := ipamclient.SpecToClient(&phpipampool.Spec)
51 if err != nil {
52 return nil, nil, err
53 }
54 h.ipamcl = ipamcl
55 return phpipampool, nil, nil
56}
57
58// EnsureAddress ensures that the IPAddress contains a valid address.
59func (h *IPAddressClaimHandler) EnsureAddress(ctx context.Context, address *ipamv1.IPAddress) (*ctrl.Result, error) {
60 hostname := fmt.Sprintf("%s.%s", h.claim.GetName(), h.claim.GetNamespace())
61 ipv4, err := h.ipamcl.GetAddress(hostname)
62 if err != nil {
63 return nil, errors.Wrap(err, "failed to get an IP Address")
64 }
65
66 address.Spec.Address = ipv4
67 address.Spec.Gateway = h.gateway
68 address.Spec.Prefix = h.mask
69 return nil, nil
70}
71
72// ReleaseAddress releases the ip address.
73func (h *IPAddressClaimHandler) ReleaseAddress(ctx context.Context) (*ctrl.Result, error) {
74 hostname := fmt.Sprintf("%s.%s", h.claim.GetName(), h.claim.GetNamespace())
75 err := h.ipamcl.ReleaseAddress(hostname)
76 return nil, err
77}
The Provider Adapter
The Provider Adapter implements the middle layer between the generic IPAM reconciler and our specific IPAM Handler.
Its function is to, first allow the Generic controller to setup a new manager for the IPAddressClaim and IPAddress and also, to get the proper ClaimHandler.
The Provider Adapter also has some specific needs (like setting a new Index and some common functions) that will not be part of the snippet, and hopefully we can integrate and migrate into the Generic reconciler
1// PHPIPAMProviderAdapter is used as middle layer for provider integration.
2type PHPIPAMProviderAdapter struct {
3 Client client.Client
4 IPAMClient *ipamclient.IPAMClient
5}
6
7var _ ipamutil.ProviderAdapter = &PHPIPAMProviderAdapter{}
8
9// SetupWithManager sets up the controller with the Manager.
10func (v *PHPIPAMProviderAdapter) SetupWithManager(_ context.Context, b *ctrl.Builder) error {
11 b.
12 For(&ipamv1.IPAddressClaim{}, builder.WithPredicates(
13 ipampredicates.ClaimReferencesPoolKind(metav1.GroupKind{
14 Group: v1alpha1.GroupVersion.Group,
15 Kind: v1alpha1.PHPIPAMPoolKind,
16 }),
17 )).
18 WithOptions(controller.Options{
19 // To avoid race conditions when allocating IP Addresses, we explicitly set this to 1
20 MaxConcurrentReconciles: 1,
21 }).
22 Watches(
23 &v1alpha1.PHPIPAMIPPool{},
24 handler.EnqueueRequestsFromMapFunc(v.IPPoolToIPClaims()),
25 builder.WithPredicates(resourceTransitionedToUnpaused()),
26 ).
27 Owns(&ipamv1.IPAddress{}, builder.WithPredicates(
28 ipampredicates.AddressReferencesPoolKind(metav1.GroupKind{
29 Group: v1alpha1.GroupVersion.Group,
30 Kind: v1alpha1.PHPIPAMPoolKind,
31 }),
32 ))
33 return nil
34}
35
36// ClaimHandlerFor returns a claim handler for a specific claim.
37func (v *PHPIPAMProviderAdapter) ClaimHandlerFor(_ client.Client, claim *ipamv1.IPAddressClaim) ipamutil.ClaimHandler {
38 return &IPAddressClaimHandler{
39 Client: v.Client,
40 claim: claim,
41 }
42}
As we can see, it is a simple middle layer between a reconciler and the real IPAM logic code.
Testing if it works
Putting all together, and with the proper “main.go” file (look at the repo!), we can test if the IP allocation works
Note Don’t forget to install the Cluster API core CRDs first with:
1kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/main/config/crd/bases/ipam.cluster.x-k8s.io_ipaddressclaims.yaml
2kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/main/config/crd/bases/ipam.cluster.x-k8s.io_ipaddresses.yaml
3kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api/main/config/crd/bases/cluster.x-k8s.io_clusters.yaml
With the controller running and able to reach phpIPAM, and with the PHPIPAMIPPool properly defined, we can create a new IPAddressClaim using our pool, and check if an IP Address is allocated:
1apiVersion: ipam.cluster.x-k8s.io/v1alpha1
2kind: IPAddressClaim
3metadata:
4 name: first-ip
5 namespace: default
6spec:
7 poolRef:
8 apiGroup: ipam.cluster.x-k8s.io
9 kind: PHPIPAMIPPool
10 name: my-ipam-test
After we apply this object, the controller should go to phpIPAM and gets us in return an IPAddress with the same name and an IP allocated, which we can later verify on phpIPAM:
1# kubectl get ipaddress first-ip -o yaml
2kind: IPAddress
3metadata:
4 name: first-ip
5 namespace: default
6.....
7spec:
8 address: 192.168.0.11 # This was filled by our controller
9 claimRef:
10 name: first-ip
11 gateway: 192.168.0.1
12 poolRef:
13 apiGroup: ipam.cluster.x-k8s.io
14 kind: PHPIPAMIPPool
15 name: my-ipam-test
16 prefix: 24
And on phpIPAM:
