k8s controller watch other controller CR - amazon-web-services

Question

I’ve k8s operator which works as expected, I need to add a “watch” to other operator CRD (not mine), to make it simple lets call it extCR and our operator cr called inCR,
I tried the following but there is an issue how its right to trigger the reconcile.
func (r *Insiconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&Inv1alpha1.Iget{}}).
Watches(&source.Kind{Type: &ext.Se{}}, handler.EnqueueRequestsFromMapFunc(r.FWatch)).
Complete(r)
}
func (r *Insiconciler) FWatch(c client.Object) []reconcile.Request {
val := c.(*ivi.Srv)
req := reconcile.Request{NamespacedName: types.NamespacedName{Name: val.Name, Namespace: val.Namespace}}
return []reconcile.Request{req}
}
The problem here that I trigger the reconcile with the extCR , I want inside the FWatch to update the inCR and start the reconcile with inCR and not with extCR, how can I do it ?
I mean, to avoid something like the following code as sometimes the reconcile is done for the inCR and sometimes for the extCR and than I can get some ugly if's
func (r *Insiconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
var inCR FOO
var extCR BAR
if err := r.Get(ctx, req.NamespacedName, &inCR); err != nil {
return ctrl.Result{}, err
}
if err := r.Get(ctx, req.NamespacedName, &extCR); err != nil {
return ctrl.Result{}, err
}
I want to know what is the right/clean way to handle such case
case when you need to listen to externalCR (not part of your controller) and also internalCR (from your controller) .
One more thing - the CR are different GVK but the exteranlCR contain lot of fields which is not required, just some of them. but the required fields is having the same names on both cr's
update
type inCR struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec inSpec `json:"spec,omitempty"` / / ————————here is the difference
Status InsightTargetStatus `json:"status,omitempty"`
}
//————— This is defined on other program which is not owned by us, therefore cannot “reuse”
type Bar struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec extSpec `json:"spec,omitempty"` // ———————here is the difference
Status ServiceStatus `json:"status,omitempty"`
}
And inSpec is having the following fields (subset of extSpec)
type inSpec struct {
name string
age int
}
and extSpec have those fields and many more which is not related
type extSpec struct {
name string
age int
foo string // not relevant
bar string // not relevant
bazz string // not relevant
}
at the end, Inside the reconcile I need to move the relevant fields to some functions. exactly same functions just take sometime the fields from extCR and sometimes for inCR, according to the event that happens (like updating the extCR or update the inCR by users )
Update2
func sharedLogic(r reconciler, ctx context.Context, c client.Object) (ctrl.Result, error) {
in := c.(*inCR)
vPass , e := vps.Get(ctx, r.Client, in.Spec.foo, in.Spec.bar)
return ctrl.Result{}, nil
}
But for extCR I should do the following
func sharedLogic(r reconciler, ctx context.Context, c client.Object) (ctrl.Result, error) {
ext := c.(*extCR)
vPass , e := vps.Get(ctx, r.Client, ext.Spec.val.foo, ext.Spec.val.bar)
return ctrl.Result{}, nil
}

Answer 1

Few things to keep in mind:
Each controller is responsible for exactly one resource.
Reconcile request contains the information necessary to reconcile a Kubernetes object. This includes the information to uniquely identify the object - its Name and Namespace. It does NOT contain information about any specific Event or the object contents itself.
You can create a second controller without the resource definition. In your main file, both controllers will be registered.
This could be useful if the CRDs are not related at all or if the external resource references the internal one, so you can make changes to the internal resource in the external reconciler.
kubebuilder create api --group other --version v2 --kind External \
--resource=false --controller=true
This gives you a controller with a SetupWithManager method that looks like the below.
func (r *ExternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
// Uncomment the following line adding a pointer to an instance of the controlled resource as an argument
// For().
Complete(r)
}
Note how the For method is commented out because you need to import the resource to watch from somewhere else and reference it.
import (
...
otherv2 "other.io/external/api/v2"
)
...
func (r *ExternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&otherv2.External{}).
Complete(r)
}
If you cannot import the external resource you could fall back to mocking it yourself but this is probably not a very clean way. You should really try to import it from the other controller project.
kubebuilder edit --multigroup=true
kubebuilder create api --group=other --version v2 --kind External \
--resource --controller
Another way is when the resources are related to each other such that the internal resource has a reference in its spec to the external resource and knows how to get the external resource in its spec, when it reconciles. An example of this can be found here https://book.kubebuilder.io/reference/watching-resources/externally-managed.html
type InternalSpec struct {
// Name of an external resource
ExternalResource string `json:"externalResource,omitempty"`
}
This means that in each reconciliation loop, the controller will look up the external resource and use it to manage the internal resource.
func (r *InternalReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
internal := examplev1.Internal{}
if err := r.Get(context.TODO(), types.NamespacedName{
Name: req.Name,
Namespace: req.Namespace,
}, &internal); err != nil {
return ctrl.Result{}, err
}
external := otherv2.External{}
if err := r.Get(context.TODO(), types.NamespacedName{
// note how the name is taken from the internal spec
Name: internal.Spec.ExternalResource,
Namespace: req.Namespace,
}, &internal); err != nil {
return ctrl.Result{}, err
}
// do something with internal and external here
return ctrl.Result{}, nil
}
The problem with this is, that when the internal resource does not change, no reconciliation event will be triggered, even when the external resource has changed. To work around that, we can trigger the reconciliation by watching the external resource. Note the Watches method:
func (r *InternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&examplev1.Main{}).
Watches(
&source.Kind{Type: &otherv2.ExternalResource{}},
handler.EnqueueRequestsFromMapFunc(r.triggerReconcileBecauseExternalHasChanged),
builder.WithPredicates(predicate.ResourceVersionChangedPredicate{}),
).
Complete(r)
}
In order to know for which internal object we should trigger an event, we use a mapping function to look up all the internal that have a reference to the external resource.
func (r *InternalReconciler) triggerReconcileBecauseExternalHasChanged(o client.Object) []reconcile.Request {
usedByInternals := &examplev1.InternalList{}
listOps := &client.ListOptions{
FieldSelector: fields.OneTermEqualSelector(".spec.ExternalResource", o.GetName()),
Namespace: o.GetNamespace(),
}
err := r.List(context.TODO(), usedByInternals, listOps)
if err != nil {
return []reconcile.Request{}
}
requests := make([]reconcile.Request, len(usedByInternals.Items))
for i, item := range usedByInternals.Items {
requests[i] = reconcile.Request{
NamespacedName: types.NamespacedName{
Name: item.GetName(),
Namespace: item.GetNamespace(),
},
}
}
return requests
}
Since you updated your question, I suggest doing something like below.
I am creating a new project and 2 controllers. Note on the second controller command no resource is created along with the controller. this is because the controller
will watch an external resource.
mkdir demo && cd demo
go mod init example.io/demo
kubebuilder init --domain example.io --repo example.io/demo --plugins=go/v4-alpha
kubebuilder create api --group=demo --version v1 --kind Internal --controller --resource
kubebuilder create api --group=other --version v2 --kind External --controller --resource=false
$ tree controllers
controllers
├── external_controller.go
├── internal_controller.go
└── suite_test.go
Now we need some shared logic, for example by adding this to the controllers package. We will call this from both reconcilers.
// the interface may need tweaking
// depending on what you want to do with
// the reconiler
type reconciler interface {
client.Reader
client.Writer
client.StatusClient
}
func sharedLogic(r reconciler, kobj *demov1.Internal) (ctrl.Result, error) {
// do your shared logic here operating on the internal object struct
// this works out because the external controller will call this passing the
// internal object
return ctrl.Result{}, nil
}
Here is an example for the internal reconciler.
func (r *InternalReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
obj := demov1.Internal{}
if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
return ctrl.Result{}, err
}
return sharedLogic(r, &obj)
}
And in the external reconciler we do the same.
func (r *ExternalReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
// note, we can use the internal object here as long as the external object
// does contain the same fields we want. That means when unmarshalling the extra
// fields are dropped. If this cannot be done, you could first unmarshal into the external
// resource and then assign the fields you need to the internal one, before passing it down
obj := demov1.Internal{}
if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
return ctrl.Result{}, err
}
return sharedLogic(r, &obj)
}
func (r *ExternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
// note the external resource is imported from another project
// you may be able to watch this without import by creating a minimal
// type with the right GKV
For(otherv2.External{}).
Complete(r)
}