MLOps: Integrating ML with Devops works better.

In machine learning ,the one of the biggest problem for making the model is hyper parameters.Because Hyper parameters are the adjustable parameters which are responsible for the accuracy of the model. For human beings it is very difficult to manually change the values for this MLOps works better i.e it automatically fine tuning the model by changing the values.

In this article I am going to demonstrate my project.

Project Overview:

#ML-DevOps integration:

1. Create container image that’s has Python3 and Keras or numpy installed using dockerfile.

2. When we launch this image, it should automatically starts train the model in the container.

3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins 

4. Job1 : Pull the Github repo automatically when some developers push repo to Github.

5. Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to depoy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the cnn processing).

6. Job3 : Train your model and predict accuracy or metrics.

7. Job4 : if metrics accuracy is less than 80% , then tweak the machine learning model architecture.

8. Job5: Retrain the model or notify that the best model is being created

9. Create One extra job job6 for monitor : If container where app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left.

Creating a project:

Creating a Docker container using Dockerfile.

This file creates the docker container along with python and libraries installed for ML.

docker build -t sklearn:v1 /ws/
docker run -dit --name sklearn_code sklearn:v1

This image creates the container along with python and libraries needed for CNN.

For launching the containers

docker build -t keras:v1 /ws/
docker run -dit --name keras_code keras:v1

JOB_1:

Uploading the code to github.

Creating the post-commit hook which prints "Code is pushed into Github" after every successful push.

For adding Web hooks in Github we need a public IP, but in this project I used the ngrok for converting the private IP to public IP.

Creating the webhooks in Github. It triggers the jenkins automatically when there is any commit done.

After creating the webhook , just click "Create webhook" option below and make sure that Active is selected.

Now, It automatically pulls the code from github and copy into local system.

In the above step it create 1 folder in the local system and copy the pulled code into the folder.

JOB_2:

Here ,below is the code for detecting whether code is ML or CNN.It automatically detects and launch the respective container for that.

Here the it mails the owner that successfully container has been launched.

JOB_3:

Job3 will trigger or build after the job_2 .This code keep on watching the job2.

In this step it starts training the model.

JOB_4:

It checks the accuracy of the model, if accuarcy is below 95% then it add 1 more epoch and 1 more layer and retrain the model.

JOB_5:

This code runs the python program which the sends the mail for notifying the best model is created.

JOB_6:

This job keep on monitoring the container if the container stops it automatically launches the container again.

So,that's all guys , we have successfully integrated the ML with Devops and build a project that automatically train the model.

Thank you for reading..