Engine Interactions

Overall Interaction Overview

Cache Integration

The Engine relies heavily on the Package Cache as its primary interface to repositories:

Opening Repositories

All repository access goes through the cache:

Engine
     ↓
OpenRepository(repositorySpec)
     ↓
Package Cache
     ↓
Repository Adapter (Git)

Process:

1. Engine receives a Repository CR specification
2. Calls cache.OpenRepository(ctx, repositorySpec)
3. Cache returns a Repository interface implementation
4. Engine uses the repository abstraction for all operations

Benefits of cache-mediated access:

• Caching: Frequently accessed repositories remain in memory
• Synchronization: Cache handles background repository sync
• Abstraction: Engine doesn’t need to know if it’s Git or another backend
• Connection pooling: Cache manages repository connections efficiently

Repository Operations

Once a repository is opened, the engine delegates all storage operations:

Package Revision Operations:

• ListPackageRevisions: Query package revisions with filtering
• CreatePackageRevisionDraft: Create mutable draft
• UpdatePackageRevision: Open existing revision as draft
• ClosePackageRevisionDraft: Commit draft to create immutable revision
• DeletePackageRevision: Remove package revision from storage

Package Operations:

• ListPackages: Query packages
• CreatePackage: Initialize new package
• DeletePackage: Remove package and all revisions

Repository Metadata:

• Version: Get repository version for change detection
• Refresh: Trigger repository sync
• Close: Clean up repository resources

The engine never directly manipulates Git repositories or storage - all operations go through the repository abstraction provided by the cache.

Cache Invalidation

The engine doesn’t directly invalidate cache entries. Instead:

• Cache monitors Repository CRs for changes
• Background sync jobs refresh repository state periodically
• Repository operations (create, update, delete) trigger cache updates automatically
• Engine operations are always performed on the latest cached state

Task Handler Invocation

The Engine delegates all package transformation operations to the Task Handler:

Task Execution During Creation

When creating a package revision:

Engine
     ↓
ApplyTask(draft, repositoryObj, packageRevision, packageConfig)
     ↓
Task Handler
     ↓
Task Mutations (init, clone, render, etc.)

Process:

1. Engine creates a draft package revision
2. Calls taskHandler.ApplyTask() with:
   • Draft to modify
   • Repository object for context
   • PackageRevision spec with task definition
   • Package configuration (path, name, etc.)
3. Task handler executes the task (init, clone, upgrade)
4. Task handler applies builtin functions (package context generation)
5. Returns modified draft or error

Task types handled:

• Init: Create new package with Kptfile
• Clone: Copy package from upstream
• Upgrade: Merge changes from new upstream version
• Edit: Create new revision from existing one

Mutations During Updates

When updating a package revision:

Engine
     ↓
DoPRMutations(repoPR, oldObj, newObj, draft)
     ↓
Task Handler
     ↓
Apply new tasks to draft

Process:

1. Engine opens draft from existing package revision
2. Calls taskHandler.DoPRMutations() with old and new specs
3. Task handler compares task lists
4. Applies any new tasks to the draft
5. Returns modified draft

Resource Mutations

When updating package resources directly:

Engine
     ↓
DoPRResourceMutations(pr, draft, oldRes, newRes)
     ↓
Task Handler
     ↓
Update resources + render

Process:

1. Engine opens draft from package revision
2. Calls taskHandler.DoPRResourceMutations() with resource changes
3. Task handler updates package resources
4. Task handler executes render task (runs function pipeline)
5. Returns render status with function results

Key interaction points:

• Engine provides the draft workspace
• Task handler modifies resources in the draft
• Engine commits the draft after task completion
• Task handler has no direct repository access

Function Runtime Integration

The task handler uses function runtimes configured in the engine:

• Builtin Runtime: For built-in functions (set-namespace, etc.)
• gRPC Runtime: For external function runner service
• Multi-Runtime: Chains multiple runtimes together

The engine configures these runtimes during initialization and passes them to the task handler.

Repository Access

The Engine accesses repositories through a layered abstraction:

Repository Abstraction Layers

Engine
     ↓
Repository Interface (from cache)
     ↓
Repository Adapter (Git)
     ↓
External Repository (GitHub, GitLab, etc.)

Repository Interface

The engine works with the Repository interface which provides:

Package Revision Management:

• Draft creation and closure
• Listing with filtering
• Deletion

Package Management:

• Package listing
• Package creation and deletion

Repository Metadata:

• Version tracking
• Refresh operations
• Resource cleanup

This interface is implemented by repository adapters (Git adapter) but the engine never calls adapter code directly.

Draft Workflow

The engine uses a draft-based workflow for all modifications:

1. Create/Open Draft: Get mutable workspace

   • CreatePackageRevisionDraft: For new revisions
   • UpdatePackageRevision: For existing revisions

2. Modify Draft: Apply changes through task handler

   • Draft provides mutable access to package revision resources
   • Changes are isolated until committed

3. Close Draft: Commit changes

   • ClosePackageRevisionDraft: Persists to repository
   • Creates package revision
   • Generates Git commit or OCI layer

4. Rollback: On errors, draft is discarded

   • Ensures atomicity
   • No partial changes persisted

Credential and Reference Resolution

The engine is configured with resolvers for repository access:

Credential Resolver:

• Resolves authentication credentials for repositories
• Supports basic auth, bearer tokens, and GCP workload identity
• Passed to repository adapters through cache
• Engine doesn’t handle credentials directly

Reference Resolver:

• Resolves package revision references (upstream package revisions)
• Used during clone and upgrade operations
• Enables cross-repository package revision operations

User Info Provider:

• Provides authenticated user information
• Used for audit trails (PublishedBy field)
• Extracted from Kubernetes API request context

These resolvers are configured during engine initialization and passed to components that need them.

Change Notification

The Engine notifies watchers of package revision changes:

Watcher Manager Integration

Engine
     ↓
NotifyPackageRevisionChange(eventType, packageRevision)
     ↓
Watcher Manager
     ↓
API Server Watch Streams

Notification events:

• Added: New package revision created
• Modified: Package revision updated (metadata or lifecycle)
• Deleted: Package revision removed

When notifications are sent:

• After successful package revision creation
• After package revision updates (including lifecycle transitions)
• After metadata-only updates on published packages
• Not sent on failures or during draft operations

Watch Stream Support

The watcher manager enables Kubernetes watch API support:

1. Client subscribes: API server calls WatchPackageRevisions
2. Filter registration: Watcher manager stores filter and callback
3. Change notification: Engine calls NotifyPackageRevisionChange
4. Event delivery: Watcher manager delivers to matching watchers
5. Client receives: Watch event sent to client

Filter support:

• Repository name and namespace
• Package name and path
• Workspace name
• Lifecycle state
• Label selectors

Only watchers with matching filters receive notifications.

Watcher Lifecycle

Watchers are automatically cleaned up:

• When client context is cancelled (connection closed)
• When watcher callback returns false (stop watching)
• Watcher manager periodically removes finished watchers
• No manual cleanup required from engine

The engine simply notifies changes and the watcher manager handles delivery and cleanup.