Efficient Argo Workflow Deletion and ECR Image Integration Script

Overview :-

#Argo workflows provide a robust framework for #orchestrating complex tasks and processes in #Kubernetes environments. However, as workflows accumulate, managing their lifecycle becomes paramount. The script discussed in this context offers a comprehensive solution for automatically deleting outdated or completed #ArgoWorkflows. This not only ensures a clutter-free environment but also frees up valuable resources and enhances overall system performance.

Pre-requestisites:-

• argo-cli should be installed on your system

Step-1 :- First write a #python script to delete the argo-workflows which is older then 2 days. Below is the sample #python script that delete’s the workflows.

Step-2 :- Create a directory named app in your home directory. with in the app directory create a file with .py extension and paste the below script into the file.

import subprocess
import json
import datetime

def delete_old_workflows():
    argo_namespace = "argo"
    current_date = datetime.datetime.now(datetime.timezone.utc)
    threshold = datetime.timedelta(days=2)

    # Get the list of Argo Workflow runs
    cmd = ["argo", "list", "-n", argo_namespace, "-o", "json"]
    try:
        output = subprocess.check_output(cmd)
        workflows = json.loads(output)
    except subprocess.CalledProcessError as e:
        print(f"Error getting Argo workflows: {e}")
        return

    for workflow in workflows:
        started_at = workflow["status"].get("startedAt", workflow["metadata"]["creationTimestamp"])
        started_at_datetime = datetime.datetime.fromisoformat(started_at.replace("Z", "+00:00"))
        age = current_date - started_at_datetime

        if age > threshold:
            workflow_name = workflow["metadata"]["name"]
            print(f"Deleting Argo workflow: {workflow_name}")
            try:
                subprocess.check_output(["argo", "delete", "-n", argo_namespace, workflow_name])
            except subprocess.CalledProcessError as e:
                print(f"Error deleting Argo workflow: {e}")

if __name__ == "__main__":
    delete_old_workflows()

Step-3 :- Now we need to create a docker file to build an image with above script. copy & paste the Below code into the docker file.

# Use the base image
FROM python:3.8

# Install argo CLI
RUN curl -sLO https://github.com/argoproj/argo/releases/download/v3.2.1/argo-linux-amd64.gz && \
    gunzip argo-linux-amd64.gz && \
    chmod +x argo-linux-amd64 && \
    mv argo-linux-amd64 /usr/local/bin/argo

# Set the working directory
WORKDIR /app

# Copy your script and other files
COPY script.py /app/

# Run the script
CMD ["python", "script.py"]

Step-4 :- Next open a command prompt or terminal window & locate to the app directory and build a docker image using above docker file.

Step-5 :- Login to your aws management console and navigate to #ECR service and create a private repository to push the docker image into #AmazonECR (elastic container registry). Use Below command to build,tag and push.

• Build your #Docker image using the following command.

docker build -t argo-workflows .

• Once building the Image is being completed Login to your ECR repository using below command

aws ecr get-login-password - region ca-central-1 | docker login - username AWS - password-stdin 123456789.dkr.ecr.ca-central-1.amazonaws.com

• After the Login is Succeded, tag your image so you can push the image to this repository:

docker tag argo-workflows:latest 123456789.dkr.ecr.ca-central-1.amazonaws.com/argo-workflows:latest

• Run the following command to push this image to your newly created #AWS repository:

docker push 123456789.dkr.ecr.ca-central-1.amazonaws.com/argo-workflows:latest

Step-6 :- After the image is being pushed create a yaml file in your app directory and copy the below code into that file to delete the workflows using a docker image.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: delete-workflows
  namespace: argo
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: delete-workflows
            template: delete-workflows
    - name: delete-workflows
      container:
        image: "12346789.dkr.ecr.ca-central-1.amazonaws.com/argo-workflows:latest"
        command: ["python", "script.py"]

Step-7 :- Now open a command prompt or terminal and configure your argo-workflows and run the below command or submit the workflow to delete the old argo-workflows.

# cmd to submit the workflows
argo submit workflow.yaml -n argo

• After running the above command u can see that the old workflows is being started running and that workflows will be deleting the old workflows. Once the running workflow got successed u can see that it has been deleted the argo-workflows which is greater than 2 days.

Find the source code Here :- “github.com/MahiraTechnology/Mahira-medium.git”

Conclusion :-

Streamlining #DevOps with #Comprehensive Workflow Deletion and ECR Image Integration

In the realm of #DevOps, automation is a cornerstone for achieving #efficiency and #reliability. The discussed script for Argo workflow deletion, coupled with ECR image #integration, encapsulates this principle perfectly. By seamlessly combining these two functionalities, #developers and #DevOps teams can:

1. Maintain a Clean Environment: The script automates the removal of redundant or completed Argo workflows, ensuring that only #relevant processes persist. This promotes a clutter-free environment and optimizes resource utilization.

2. Enhance Performance: The ability to delete outdated workflows on schedule prevents unnecessary resource consumption, leading to improved #system performance and responsiveness.