TASK - Integrating ML DL with Jenkins

1- Pull the GitHub repository automatically when the codes and datasets are pushed on GitHub.

2- Identify using the code to start the respective container with preinstalled software to train your model and predict the accuracy.

3- If accuracy below 80%, tweak the ml model architecture and retrain. Notify if accuracy above 80%.

SOLUTION ::

We will start by creating containers using docker file - One for traditional machine learning and other for deep learning.

Machine Learning :

FROM centos:latest
RUN yum install python36 -y
RUN pip3 install numpy
RUN pip3 install pandas 
RUN pip3 install scikit-learn 
RUN pip3 install opencv-python 

CMD python3 /mnt/train.py

The model and dataset should be mounted on the /mnt directory and the code to be executed should be named train.py. The paths in the code should be altered as well.

Deep Learning:

FROM tensorflow/tensorflow
RUN pip3 install numpy
RUN pip3 install pandas 
RUN pip3 install keras
RUN pip3 install opencv-python 
RUN pip3 install scikit-learn 

CMD python3 /mnt/train.py

Job1 - Getting codes from GitHub.

The steps for this can be found in the following article.

The codes from GitHub will be stored in the workspace of Job1 and therefore this path will be mounted on /mnt.

And make sure to make a change in Job1, so the workspace is clean before every build.

Job2 - Training the model on a container. Send notification if accuracy above 80% else send to the tweak job and retrain the model.

** ML model is a onetime train condition here. DL model will only retrain.

I will be showing the setup for ML first and this will be enhanced with methods of DL in the later stage.

In the build, add Execute shell and add the following to it.

export ML=$(cat /var/lib/jenkins/workspace/mlops_job1/train.py | grep sklearn | wc -l)
export DL=$(cat /var/lib/jenkins/workspace/mlops_job1/train.py | grep keras | wc -l)
echo $ML
echo $DL
if [ $ML -gt 0 ] && [ $DL -eq 0 ]
then
sudo docker run -dit -v /var/lib/jenkins/workspace/mlops_job1:/mnt --name ml_container machinelearning:latest
export status=1
while [ 1 -eq 1 ]
do
   export status=$(sudo docker ps | grep ml_container | wc -l)
   if [ $status -eq 0 ]
   then
   sudo docker rm -f ml_container
   break
   fi
done
echo "ACCURACY"
cat /var/lib/jenkins/workspace/mlops_job1/accuracy.txt
cp /var/lib/jenkins/workspace/mlops_job1/accuracy.txt /var/lib/jenkins/workspace/mlops_mail/accuracy.txt
curl -u admin:redhat 192.168.99.102:8080/view/Ml%20+%20Jenkins/job/mlops_mail/build?token=mailing


elif [ $ML -ge 0 ] && [ $DL -gt 0 ]
then
sudo docker run -dit -v /var/lib/jenkins/workspace/mlops_job1:/mnt --name dl_container deeplearning:latest
export status=1
while [ 1 -eq 1 ]
do
   export status=$(sudo docker ps | grep dl_container | wc -l)
   if [ $status -eq 0 ]
   then
   sudo docker rm -f dl_container
   break
   fi
done
echo "ACCURACY"
cat /var/lib/jenkins/workspace/mlops_job1/accuracy.txt
python3 /var/lib/jenkins/workspace/mlops_job1/upgrade.py
cp /var/lib/jenkins/workspace/mlops_job1/accuracy.txt /var/lib/jenkins/workspace/mlops_mail/accuracy.txt
cp /var/lib/jenkins/workspace/mlops_job1/multiclassNN.h5 /var/lib/jenkins/workspace/mlops_mail/multiclassNN.h5

fi

This code checks for type of code and works on it accordingly. It also triggers the mail job as well.

python3 /var/lib/jenkins/workspace/mlops_job1/upgrade.py

This contains the code to tweak the train code which is also to be pushed to the repository. The update methods can be changed in here.

The codes used in this project can be found in this repository ::

Mail_Job- I installed the plugin and configured it already in the jenkins global settings. For reference, visit

Update the job accordingly. The file attached here has already been copied to the mail job workspace in job2.

** Monitoring Job has not been implemented in this solution.

Thanks for giving your time to my solution.