Integrating Machine Learning with DevOps

Machine Learning

Machine learning is an application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning focuses on the development of computer programs that can access data and use it learn for themselves.

DevOps

DevOps (development and operations) is an enterprise software development phrase used to mean a type of agile relationship between development and IT operations. The goal of DevOps is to change and improve the relationship by advocating better communication and collaboration between these two business units.

MLOps

The integration of machine learning is known as MLOps . MLOps (a compound of “machine learning” and “operations”) is a practice for collaboration and communication between data scientists and operations professionals to help manage production ML (or deep learning) life cycle .

Why MLOps ?

Manually constructing a machine learning model is a multi step process that requires domain knowledge, mathematical expertise, and computer science skills — which is a lot to ask of one company, let alone one data scientist. Not only that, there are countless opportunities for human error and bias, which degrades model accuracy and devalues the insights you might get from the model. Automated machine learning enables organizations to use the baked-in knowledge of data scientists without expending time and money to develop the capabilities themselves, simultaneously improving return on investment in data science initiatives and reducing the amount of time it takes to capture value.

Automated machine learning makes it possible for businesses in every industry — health care sector, marketing, retail, sports, manufacturing, and more — to leverage machine learning and AI technology — technology previously only available to organizations with vast resources at their disposal. By automating most of the modeling tasks necessary in order to develop and deploy machine learning models, automated machine learning enables business users to implement machine learning solutions with ease, thereby allowing an organization’s data scientists to focus on more complex problems.

Problem Statements:-

1. Create container image that has Python3 and Keras or numpy installed using dockerfile

2. When we launch this image, it should automatically starts train the model in the container.

3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins

Job 1 : Pull the Github repo automatically when some developers push repo to Github.

Job 2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the cnn processing).

Job 3 : Train your model and predict accuracy or metrics.

Job 4 : if metrics accuracy is less than 80% , then tweak the machine learning model architecture.

Job 5 : Retrain the model or notify that the best model is being created

Create One extra job, job 6 for monitor : If container where app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left.

Task description

1. Create container image that’s has Python3 and Keras or numpy installed using dockerfile.

job 1 : Pull the Github repository automatically when some developers push repository to Github.

job 2:- This job will be triggered only after the completion of first job.After the successful pulling of the files, Job 2 will launch the image container which will start training the machine learning model automatically and send the accuracy mail to the developer after training of the model.

job 3 : This job monitors the container which is running in job 2. If the container ,where the app is running fails due to any reason ,then automatically starts the container again from where the last trained model left & this job chain is created using build pipeline in Jenkins.

job 4 : This job is only for tweaking the ML model. It will check the accuracy an if accuracy is less than 85% it will trigger to job5 other wise if accuracy is greater than 85% then it will trigger to job6 .

job 5: Retrain the model or notify that the best model is being created and Send the mail if the expected accuracy is achieved.

job 6 : In this job , continuously check that whether our container is running or not and if not it would automatically restart the container.

AT LAST, Build-Pipeline View :

Thankyou Vimal Daga sir for making this possible and making us understand in a very easy way from starting with very basic concepts and then taking it a advance level.