ae-ano-detection¶

• Main author - HE-Arc
• Code
• Deployment configuration
• Staging
• Production

Description¶

This service uses an autoencoder model to detect an anomaly in a time series.

The service is built in two steps:

1. Model creation - The creation of the model from the data
2. Model serving - The serving of the built model

Model creation¶

The goal of this step is to prepare the data and train a new model. All further commands are ran in the model-creation directory.

Set up the environment¶

Set up the environment with the following commands.

1
2

# Generate the virtual environment
python3.11 -m venv .venv

1
2

# Activate the virtual environment
source .venv/bin/activate

1
2

# Activate the virtual environment
.\venv\Scripts\activate

1
2
3
4

# Install the requirements
pip install \
    --requirement requirements.txt \
    --requirement requirements-all.txt

Run the experiment¶

Run a new training using the following commands.

1
2
3
4
5
6
7
8
9

# Export the MinIO S3 credentials (ask them to other members of the team)
export AWS_ACCESS_KEY_ID=***
export AWS_SECRET_ACCESS_KEY=***

# Pull the required data for the experiment from MinIO
dvc pull

# Reproduce the ML experiment with DVC
dvc repro

The DVC pipeline is described in the dvc.yaml file.

Each stage describes the dependencies and the outputs of the stage. Every time a dependency of the experiment is updated, running dvc repro will run the stages of the pipeline that are affected and keep the results in cache to speed up future runs.

More information on their website: Get Started: Data Pipelines - dvc.org.

Push new data/results to MinIO¶

In order to push new results to MinIO, use the following commands (similar to Git). Note: DVC automatically adds files that are specified in the pipelines. In other words, there are no needs to explicitly add those files with dvc add. Don't forget to then add the DVC metadata files to Git as well.

1
2
3
4
5
6
7
8

# Get the data status
dvc status

# Add the required files to DVC
dvc add <the files you would add to DVC>

# Push the data to DVC
dvc push

Model serving¶

The goal of this step is to serve the model made in the previous step. All further commands are ran in the model-serving directory.

The API documentation is automatically generated by FastAPI using the OpenAPI standard. A user friendly interface provided by Swagger is available under the /docs route, where the endpoints of the service are described.

This simple service only has one route /compute that takes an image as input, which will be used to guess the number.

Retrieve the model¶

Run the following command to get the model created from the previous step.

1
2

# Copy the model from the creation directory
cp ../model-creation/model/ae_model.h5 ./model/ae_model.h5

Environment variables¶

Check the Core concepts > Service > Environment variables documentation for more details.

Start the service locally¶

Check the Core concepts > Service > Start the service locally documentation for more details.

Run the tests with Python¶

Check the Core concepts > Service > Run the tests with Python documentation for more details.