Core AI Engine¶

• Main author - HEIA-FR & HEIG-VD
• Code
• Deployment configuration (backend) (frontend)
• Staging (backend) (frontend)
• Production (backend) (frontend)

Description¶

The Core AI Engine allows to create and manage pipelines of microservices.

The Core AI Engine is split in two main services:

• Core AI Engine Backend - The API to register services and create pipelines
• Core AI Engine Frontend - The user interface to interact with the backend

Another service is available, GlitchTip, to centralize the logging errors thrown by the Core AI Engine and its services.

Backend¶

The Core AI Engine Backend is built using Python 3.11, FastAPI, SQLModel and Domain-driven design.

The API documentation is automatically generated by FastAPI using the OpenAPI standard. A user-friendly interface provided by Swagger is available under the /docs route, where the endpoints of the services are described.

Diagrams¶

UML diagram (current version)¶

This diagram represents the current version of the Core AI Engine. A future and ideal version is available in the UML diagram (future and ideal version).

The models defined for each class is represented as follow:

classDiagram
    note for CoreModel "An abstract class to\norder by in the database"

    class CoreModel {
        <<abstract>>
        +id: UUID
        +created_at: datetime
        +updated_at: datetime
    }

    note for PipelineExecution "A pipeline execution\nis the current execution\nperformed by a given\npipeline"

    class PipelineExecution {
        +pipeline: Pipeline
        +current_pipeline_step: PipelineStep | None
        +tasks: List[Task]
        +files: List[FileKeyReference]
    }

    note for Task "A task is given\nto a service that\nperform an action\non the inputs and\nstore its results\nas outputs"

    class Task {
        +service: Service
        +pipeline_execution: PipelineExecution | None
        +data_in: List[str]
        +data_out: List[str]
        +status: Enum[TaskStatus]
    }

    class ExecutionUnitTag {
        +name: str
        +acronym: str
    }

    class FieldDescription {
        +name: str
        +type: str
    }

    class Service {
        +name: str
        +slug: str
        +summary: str
        +description: str
        +tags: List[ExecutionUnitTag]
        +data_in_fields: List[FieldDescription]
        +data_out_fields: List[FieldDescription]
        +status: ExecutionUnitStatus
        +docs_url: str | none
        +has_ai: bool
        +url: str
    }

    class Pipeline {
        +name: str
        +slug: str
        +summary: str
        +description: str
        +tags: List[ExecutionUnitTag]
        +data_in_fields: List[FieldDescription]
        +data_out_fields: List[FieldDescription]
        +status: ExecutionUnitStatus
        +steps: List[PipelineStep]
    }

    note for PipelineStep "A pipeline step is\none step in the\nwhole pipeline with\nits dependencies,\ninputs, etc."

    class PipelineStep {
        +identifier: str
        +needs: List[str]
        +condition: str | None
        +inputs: List[str]
        +pipeline: Pipeline
        +pipeline_executions: List[PipelineExecution]
        +service: Service
    }

    note for FileKeyReference "A FileKeyReference references an object in the S3 storage bucket"

    class FileKeyReference {
        +reference: str
        +file_key: str
    }

    class TaskStatus {
        <<enumeration>>

        str PENDING
        str FETCHING
        str PROCESSING
        str SAVING
        str FINISHED
        str ERROR
        str SCHEDULED
        str SKIPPED
        str ARCHIVED
        str UNAVAILABLE
    }

    class ExecutionUnitStatus {
        <<enumeration>>

        str AVAILABLE
        str UNAVAILABLE
        str DISABLED
    }

    CoreModel <|-- Task
    CoreModel <|-- PipelineStep
    CoreModel <|-- PipelineExecution
    CoreModel <|-- Pipeline
    CoreModel <|-- Service

    Pipeline "*" --> "*" ExecutionUnitTag: has
    Pipeline "*" --> "*" FieldDescription: has

    PipelineExecution "1" --> "*" FileKeyReference: has
    PipelineExecution "0..*" --> "1..1" Pipeline: refers to
    PipelineExecution "1..1" --> "0..*" PipelineStep: has

    PipelineStep "0..*" --> "1..1" Pipeline: refers to
    PipelineStep "0..*" --> "1..1" Service: refers to

    PipelineExecution "1..1" --> "0..*" Task: has
    Task "0..*" --> "1..1" Service: refers to

UML diagram (future and ideal version)¶

This diagram represents the future and ideal version of the Core AI Engine.

At the time being, the Core AI Engine is not fully implemented as described in the diagram, mostly due to SQLModel limitations with inherence.

The models defined for each class is represented as follow:

classDiagram
    note for CoreModel "An abstract class to\norder by in the database"

    class CoreModel {
        <<abstract>>
        +id: UUID
        +created_at: datetime
        +updated_at: datetime
    }

    note for ExecutionUnit "An execution unit is\nan abstract class to\nrun services or pipelines"

    class ExecutionUnit {
        <<abstract>>
        +name: str
        +slug: str
        +summary: str
        +description: str
        +tags: List[ExecutionUnitTag]
        +data_in_fields: List[FieldDescription]
        +data_out_fields: List[FieldDescription]
        +status: ExecutionUnitStatus
    }

    note for PipelineExecution "A pipeline execution\nis the current execution\nperformed by a given\npipeline"

    class PipelineExecution {
        +pipeline: Pipeline
        +current_pipeline_step: PipelineStep | None
        +tasks: List[Task]
        +files: List[FileKeyReference]
    }

    note for Task "A task is given\nto a service that\nperform an action\non the inputs and\nstore its results\nas outputs"

    class Task {
        +service: Service
        +pipeline_execution: PipelineExecution | None
        +data_in: List[str]
        +data_out: List[str]
        +status: Enum[TaskStatus]
    }

    class ExecutionUnitTag {
        +name: str
        +acronym: str
    }

    class FieldDescription {
        +name: str
        +type: str
    }

    class Service {
        +url: str
        +docs_url: str | None
        +has_ai: bool
    }

    class Pipeline {
        +steps: List[PipelineStep]
    }

    note for PipelineStep "A pipeline step is\none step in the\nwhole pipeline with\nits dependencies,\ninputs, etc."

    class PipelineStep {
        +identifier: str
        +needs: List[str]
        +condition: str | None
        +inputs: List[str]
    }

    note for FileKeyReference "A FileKeyReference references an object in the S3 storage bucket"

    class FileKeyReference {
        +reference: str
        +file_key: str
    }

    class TaskStatus {
        <<enumeration>>

        str PENDING
        str FETCHING
        str PROCESSING
        str SAVING
        str FINISHED
        str ERROR
        str SCHEDULED
        str SKIPPED
        str ARCHIVED
        str UNAVAILABLE
    }

    class ExecutionUnitStatus {
        <<enumeration>>

        str AVAILABLE
        str UNAVAILABLE
        str DISABLED
    }

    CoreModel <|-- Task
    CoreModel <|-- ExecutionUnit
    CoreModel <|-- PipelineStep
    CoreModel <|-- PipelineExecution

    ExecutionUnit <|-- Service
    ExecutionUnit <|-- Pipeline

    ExecutionUnit "*" --> "*" ExecutionUnitTag: has
    ExecutionUnit "*" --> "*" FieldDescription: has

    PipelineExecution "1" --> "*" FileKeyReference: has
    PipelineExecution "0..*" --> "1..1" Pipeline: has
    PipelineExecution "1..1" --> "0..*" PipelineStep: has

    PipelineStep "0..*" --> "1..1" Pipeline: belongs to
    PipelineStep "0..*" --> "1..1" ExecutionUnit: links to

    Task "0..*" --> "1..1" PipelineExecution: has
    Task "0..*" --> "1..1" Service: belongs to

Sequence¶

A normal sequence of events when launching a request to a service is represented as follows:

sequenceDiagram
    participant E as e - Core AI Engine
    participant S as s - Service
    participant C as c - Client
    participant O as o - Object storage
    S->>+E: POST(core-engine_url: str, service_json: ServiceCreate)
    E->>E: service: Service = Service.from_orm(service_json)
    E->>E: enable_service(service)
    E->>E: add_api_route(service.slug, handler)
    E-->>-S: return(200, service: ServiceRead)
    C->>+E: POST(/s.slug, data: UploadFile[])
    E->>+O: Store input files
    O-->>-E: Return file keys
    E->>E: task = create_task(data_in: file_keys[])
    E->>E: service_task = new ServiceTask(storage_url, task, callback_url)
    E->>+S: POST(s.url/compute, service_task)
    S-->>-E: return(200, {status: Task added to the queue})
    E-->>-C: return(200, task: TaskReadWithServiceAndPipeline)
    S->>+E: GET(service_task.storage_url/{key})
    E->>+O: Read input file
    O-->>-E: Return file
    E-->>-S: return(200, file)
    S->>S: result = process(data)
    S->>+E: POST(service_task.storage_url, result file)
    E->>+O: Store result file
    O-->>-E: Return file key
    E-->>-S: return(200, key)
    S->>S: task_update = jsonable_encoder(TaskUpdate({status: finished, task.data_out: data_out}))
    S->>+E: PATCH(service_task.callback_url, task_update)
    E-->-S: return(200, service: ServiceRead)
    C->>+E: GET(/tasks, task_id: str)
    E-->>-C: return(200, task: TaskReadWithServiceAndPipeline)
    C->>+E: GET(/storage/{key})
    E->>+O: Read result file
    O-->>-E: Return file
    E-->>-C: return(200, file)

The Core AI Engine constructs storage_url from its public HOST setting by appending /storage. Services use this URL for input downloads and result uploads. The Core AI Engine remains the only component in this flow that uses the configured S3 host, bucket, region and credentials.

Environment variables¶

All environment variables are described in the .env file.

The environment variables can be overwritten during the CI/CD pipeline described in the backend.yml GitHub workflow file.

Database migrations¶

When changing SQLModel models, create and review an Alembic migration:

1

uv run alembic revision --autogenerate -m "describe change"

Committed migrations are applied automatically on backend startup with:

1

uv run alembic upgrade head

Run the tests with Python¶

For each module a test file is available to check the correct behavior of the code. The tests are run using the pytest library with code coverage check. To run the tests, use the following command inside the src folder:

1
2

# Run the tests
pytest

Frontend¶

The Frontend is built using React and TypeScript.

Environment variables¶

All environment variables are described in the .env file.

The environment variables can be overwritten during the CI/CD pipeline described in the frontend.yml GitHub workflow file.

Build the application¶

In the frontend directory, build the Core AI Engine Frontend with the following commands.

1
2
3
4
5

# Install the dependencies
npm ci

# Build the Core AI Engine Frontend
npm run build

The output of the build is in the build directory.

Once a React application is built, the environment variables cannot be changed.

Build and run the Docker image¶

In order to build the Docker image, the application must be built beforehand. Then, the Docker image can be built with the following commands.

1
2
3
4
5

# Build the Docker image with a tag
docker build -t swiss-ai-center/core-engine-frontend .

# Run the Docker image
docker run -p 3000:80 swiss-ai-center/core-engine-frontend

The Core AI Engine Frontend is available on http://localhost:3000.

Q: Why don't we build the React application within the Docker image?

A: This setup allows us to speed up the build process of the Docker image: it does not need to download and install all dependencies every time the package.json file is updated. In a CI/CD set up, the node_modules can be cached in the build stage and the output can be passed to the publish stage.

Start the Core AI Engine locally¶

You have several options to start the Core AI Engine locally:

• Start the Core AI Engine locally with Docker Compose (recommended)
• Start the Core AI Engine locally with plain Python and Node.js
• Start the Core AI Engine locally with minikube and official Docker images
• Start the Core AI Engine locally with minikube and local Docker images

1
2
3
4
5

# Build the Docker image
docker compose build

# Start the Core AI Engine Backend
docker compose up

1
2
3
4
5

# Build the Docker image
docker compose build

# Start the Core AI Engine Frontend
docker compose up

1
2
3
4
5

# Start the dependencies
docker compose up minio db

# Generate the virtual environment
python3.11 -m venv .venv

1
2

# Activate the virtual environment
source .venv/bin/activate

1
2

# Activate the virtual environment
.\venv\Scripts\activate

1
2

# Sync dependencies
uv sync

1
2
3
4

# Install the requirements
pip install \
    --requirement requirements.txt \
    --requirement requirements-all.txt

1
2
3
4
5
6
7
8

# Optional: Edit the environment variables
vim .env

# Switch to the `src` directory
cd src

# Start the application
uvicorn --reload --port 8080 main:app

1
2
3
4
5
6
7
8

# Install the dependencies
npm ci

# Optional: Edit the environment variables
vim .env

# Start the Core AI Engine Frontend
npm run start

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Start MinIO
kubectl apply \
    -f kubernetes/minio.pvc.yml \
    -f kubernetes/minio.config-map.yml \
    -f kubernetes/minio.stateful.yml \
    -f kubernetes/minio.service.yml

# Start PostgreSQL
kubectl apply \
    -f kubernetes/postgres.pvc.yml \
    -f kubernetes/postgres.config-map.yml \
    -f kubernetes/postgres.stateful.yml \
    -f kubernetes/postgres.service.yml

# Start the core-engine
kubectl apply \
    -f kubernetes/core-engine-backend.config-map.yml \
    -f kubernetes/core-engine-backend.stateful.yml \
    -f kubernetes/core-engine-backend.service.yml

1
2

# Open a tunnel to the Kubernetes cluster
minikube tunnel --bind-address 127.0.0.1

1
2
3
4

# Start the Core AI Engine Frontend
kubectl apply \
    -f kubernetes/core-engine-frontend.stateful.yml \
    -f kubernetes/core-engine-frontend.service.yml

1
2

# Open a tunnel to the Kubernetes cluster
minikube tunnel --bind-address 127.0.0.1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Access the Minikube's Docker environment
eval $(minikube docker-env)

# Build the Docker image
docker build -t ghcr.io/swiss-ai-center/core-engine:latest .

# Exit the Minikube's Docker environment
eval $(minikube docker-env -u)

# Edit the `kubernetes/core-engine.stateful.yml` file to use the local image by uncommented the line `imagePullPolicy`
#
# From
#
#        # imagePullPolicy: Never
#
# To
#
#        imagePullPolicy: Never

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Start MinIO
kubectl apply \
    -f kubernetes/minio.pvc.yml \
    -f kubernetes/minio.config-map.yml \
    -f kubernetes/minio.stateful.yml \
    -f kubernetes/minio.service.yml

# Start PostgreSQL
kubectl apply \
    -f kubernetes/postgres.pvc.yml \
    -f kubernetes/postgres.config-map.yml \
    -f kubernetes/postgres.stateful.yml \
    -f kubernetes/postgres.service.yml

# Start the core-engine
kubectl apply \
    -f kubernetes/core-engine-backend.config-map.yml \
    -f kubernetes/core-engine-backend.stateful.yml \
    -f kubernetes/core-engine-backend.service.yml

1
2

# Open a tunnel to the Kubernetes cluster
minikube tunnel --bind-address 127.0.0.1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# Install Node dependencies
npm ci

# Optional: Edit the environment variables to change the Core AI Engine URL
vim .env

# Build the Core AI Engine Frontend
npm run build

# Access the Minikube's Docker environment
eval $(minikube docker-env)

# Build the Docker image
docker build -t ghcr.io/swiss-ai-center/core-engine-frontend:latest .

# Exit the Minikube's Docker environment
eval $(minikube docker-env -u)

# Edit the `kubernetes/frontend.stateful.yml` file to use the local image by uncommented the line `imagePullPolicy`
#
# From
#
#        # imagePullPolicy: Never
#
# To
#
#        imagePullPolicy: Never

1
2
3
4
5

# Start the Core AI Engine Frontend
kubectl apply \
    -f kubernetes/core-engine-frontend.config-map.yml \
    -f kubernetes/core-engine-frontend.stateful.yml \
    -f kubernetes/core-engine-frontend.service.yml

1
2

# Open a tunnel to the Kubernetes cluster
minikube tunnel --bind-address 127.0.0.1