Introduction to gRPC and Implementing gRPC in Django with ORM Access

Introduction to gRPC

What is gRPC:

gRPC stands for Google Remote Procedure Call. gRPC is an open-source RPC framework that is used to create scalable and quick APIs. It enables the development of networked systems and open communication between gRPC client and server applications. gRPC has been embraced by several top tech companies, including Google, IBM, Netflix, and more. The gRPC framework depends on cutting-edge tech stacks like HTTP/2, protocol buffers, and more for optimum API protection, high-performance remote procedure calls, and scalability.

What are RPCs?

RPC and REST - Representational State Transfer- have historically been two separate approaches to building APIs. Additionally, protocols like SOAP and GraphQL are also used for this purpose. Remote procedure calls let you write software as if it will run locally, although it might run on a different device.

They are the most conventionally used framework to design APIs. In contrast to a typical HTTP protocol call, an RPC employs a function call as the primary method of client-server interaction. RPC is a productive technique for creating APIs since the exchanges are simple, and the contents are lightweight. gRPC services also mimic this communication architecture. RPC employs IDL - Interface Definition Language to contract the data type and the methods that will be invoked. Adopted gRPC services from RPCs have become very popular in recent years. 

Overview of gRPC concepts

The utilization of cutting-edge technologies, which have high performance over JSON and XML and offer greater API integrity, is responsible for the creation and popularity of gRPC. Some of the gRPC concepts you should be aware of are:

Protocol buffers

Protocol buffers, also known as Protobuf, are serialization or deserialization standards that make it simple to define applications and automatically perform the code generation of client libraries. The most recent version, proto3, is simpler to use and offers the newest capabilities for gRPC.

.Proto files enable the gRPC services and communications among gRPC clients and server messages. The .proto file is loaded into memory at execution by the Protobuf compiler - protoc. This compiler builds gRPC client and gRPC server applications that employ the in-memory structure to serialize and deserialize the binary data. Each communication is sent and received between the user and remote service following code generation in gRPC. 

As data is translated into a binary form and encrypted signals are smaller, parsing with Protobuf uses less CPU power for gRPC. Therefore, even on computers with a weaker CPU, such as cell phones, messages are sent more quickly with gRPC.

#sample .proto file

service RouteGuide {
  rpc GetFeature(Point) returns (Feature);
  rpc RouteChat(stream RouteNote) returns (stream RouteNote);
}
message Point {
  int32 Latitude = 1;
  int32 Longitude = 2;
}
message Feature {
 string name = 1;
 Point location = 2;
}
message RouteNote {
  Point location = 1;
  string message = 2;
}        

HTTP/2

gRPC service is built on HTTP/2, the version of HTTP/1.1 that has fewer limitations. Although it works with the older HTTP protocol, HTTP/2 has several sophisticated features for gRPC. This includes a binary framing layer, which divides each HTTP/2 query and replies into smaller messages and frames them in binary format to improve message delivery. Additionally, gRPC supports multiple requests and responses from the client and gRPC server in bidirectional full-duplex streaming. HTTP/2 has a flow control method that allows for precise control of the RAM needed to buffer in-flight packets. It also offers header compression for gRPC services. Everything in HTTP/2 is encrypted before transmission, even headers, which provide high-performance remote procedure calls. gRPC provides both asynchronous and synchronous processing with HTTP/2, enabling the execution of various interactive and streamed RPC types. 

With the help of all these characteristics of HTTP/2, gRPC services can utilize fewer resources, which leads to faster response times among cloud-based applications and gRPC services and greater battery life for gRPC clients operating on mobile devices.

Streaming

A crucial idea that gRPC supports is streaming, which allows for the execution of several processes inside a single request. gRPC makes this feasible through HTTP/2's multiplexing feature, which allows several responses or requests to be sent or received simultaneously over one TCP - Transmission Control Protocol - connection. The primary streaming formats are server-streaming RPCs, client-streaming RPCs, and bidirectional-streaming RPCs.

Channels

As opposed to HTTP/2 streams, which permit numerous simultaneous streams on a single request connection, a channel with gRPC supports multiple continuous streams across multiple requests. They are employed to build a client stub and give a mechanism to link to a gRPC server on a specific IP and port.

Pros of gRPC

gRPC has several advantages over other API designing mechanisms. gRPC also improves upon the RPC structure. Here are the most prominent benefits of gRPC services:

• High performance remote procedure callsUsing Protobuf and HTTP/2, gRPC services provide up to 10 times the high performance and API protection of REST+JSON interaction. With the use of server pushes, multiplexing, and header compression, HTTP/2 provides high-performance rankings for gRPC services. While multiplexing does away with head-of-line delay, server push makes it possible for HTTP/2 to push material from the server to the client before it is required. The messages are compressed more effectively using HTTP/2, resulting in quicker loading with gRPC services.
• StreamingThe service description for streaming gRPC services already includes client- or server-end streaming principles. Building gRPC clients and streaming services are made significantly easier as a result.
• Code GenerationThe code generation for gRPC client and gRPC server programs is the key component of the gRPC web approach. For code generation from the .proto file, gRPC modules employ the .protoc compiler. The Protobuf format is controlled through code generation in gRPC, which is used to define both data formats and application endpoints. It can create client-side network stubs and server-side skeletons, which reduces the amount of time needed to design programs with a variety of services in gRPC services.
• InteroperabilityNumerous systems and programming languages, like Java, Ruby, Go, C#, and more, are supported by the gRPC resources and libraries. With these programming languages, developers can create performant apps while utilizing complete cross-platform compatibility with gRPC. This is thanks to the Protobuf binary wiring form and effective code generation for almost all systems.
• SecurityAPI security is ensured in gRPC using HTTP/2 over a TLS end-to-end encrypted session. gRPC promotes the adoption of SSL/TLS for data encryption and authentication between the server and the gRPC client.
• Productivity and UsabilitySince gRPC is a complete RPC alternative, it operates without a hitch on a wide range of systems and languages. gRPC also has great tooling, with a lot of the necessary boilerplate code being generated manually. Engineers can now concentrate more on core functionality because of the significant time savings with gRPC.

Cons of gRPC

Although we can hope that gRPC drawbacks will eventually get solved, they do pose some problems to its usage now. Some cons of gRPC you should be aware of are:

• Lack of maturityA technology's development can be a significant barrier to adoption. That is also clear while using gRPC. GraphQL, one of gRPC's peers, has more than 14k queries on StackOverflow, while gRPC only has a bit below 4k at the moment. The gRPC community lacks knowledge of best practices, solutions, and successes because there isn't much programmer assistance for HTTP/2, as well as protocol buffers outside of Google. However, as the gRPC community expands and draws in new developers, this will eventually evolve.
• Limited browser supportAs no current gRPC web browser can handle HTTP/2 frames, you can't effectively call a gRPC service from a browser since gRPC Web primarily depends on HTTP/2. As a result, you must utilize a proxy with gRPC, which comes with several drawbacks.
• Unreadable by humansUnlike XML and JSON, Protobuf files are not human-readable since data is compressed to a binary format. Developers must make use of additional tools, such as the server reflection protocol and gRPC command prompt to evaluate payloads, carry out troubleshooting, and create manual queries.
• Steep learning curveIt will take a while to become familiar with protocol buffers and discover methods for coping with HTTP/2 friction, in contrast to REST and GraphQL, both of which mostly employ JSON.

gRPC vs REST API

As gRPC heavily uses HTTP/2, it is impossible to call a gRPC service from a web browser directly. Modern browsers do not provide the control needed over web requests to support a gRPC client. Hence, a proxy layer and gRPC-web are required to perform conversions between HTTP/1.1 and HTTP/2.

So REST APIs are still well suited for client-server communication and gRPC is now mainly used for server-to-server communication between microservices

gRPC for microservices

One of the biggest advantages of microservices is the ability to use different technologies for each independent service i.e polyglot. Each microservices agrees on API to exchange data, data format, error patterns, load balancing, etc. Since gRPC allows for describing a contract in a binary format, it can be used efficiently for microservices communication independent of the languages.

n the above diagram, a microservices architecture is shown. The request from the client (web or mobile) reaches the API Gateway and then goes to the aggregator service.

The Shopping Aggregator microservice calls the other microservices internally and aggregates the data to send to the front end. As you can see, the communication between aggregators and other services happens through the gRPC protocol.

In a real-world scenario, microservices will talk to each other a lot to aggregate the data and send it to the client. If we use a normal REST API it will tend to be slower and the client will experience more latency. If we use gRPC between microservices, it will be much faster and have low latency for the client.

gRPC internals

Let us look at the steps needed for communication between 2 microservices using gRPC. The service that provides the service is the gRPC server and the other service which requires data from the provider is the gRPC consumer. A gRPC server can also be a consumer when it needs data from another microservice.

In this example, let us assume that the gRPC is implemented in the Go language and the consumer is implemented using Java. The network communication between the service and consumer takes place over HTTP/2 using the gRPC protocol.

There are 3 components as follows

1. Service Definition

2. gRPC Server

3. gRPC Client

1. Service Definition

The first step of building a gRPC service is to create the service interface definition with the methods that are exposed by that service along with input parameters and return types.

The service definition is specified in the ProductInfo.proto file, which is used by both the server and client sides to generate the code and this serves as the API contract between them.

These proto files are generated on the gRPC server and used in the client code irrespective of the language.

2. gRPC Server

With the service definition file, the source code can be generated using the protocol buffer compiler protoc. As mentioned in my previous article, protoc is supported by a lot of languages.

With the gRPC plug-in for protocol buffers, you can generate gRPC server-side as well as the regular protocol buffer code for populating, serializing, and retrieving your message types.

On the server side, the server implements that service definition and runs a gRPC server to handle client calls.

3. gRPC Client

On the client side, we generate the client-side stub using the service definition. The client stub provides the same methods as the server and it converts them to remote function invocation network calls that go to the server side. Since gRPC service definitions are language-agnostic, client stubs can be created in any language.

gRPC Use Cases

gRPC can be used for the following use cases

1. Microservices: gRPC is designed for low latency and high throughput communication. As discussed above, it works very well for microservices where efficiency and latency are critical.
2. Point-to-point real-time communication: gRPC has excellent support for bi-directional streaming. gRPC services can push messages in real-time without polling.
3. Polyglot environments: gRPC tooling supports all popular development languages, making gRPC a good choice for multi-language environments.
4. Low-power low-bandwidth networks: gRPC messages are serialized with Protobuf, a lightweight message format. A gRPC message is always smaller than an equivalent JSON message.
5. Inter-process communication (IPC): IPC transports such as Unix domain sockets and named pipes can be used with gRPC to communicate between apps on the same machine.

Implementing gRPC in Django with ORM Access

lets start from the basics

Step 1: Create a Virtual Environment

I assume you already have python running in your environment So, the next obvious step is to install Django. Before that, it will better if we create a virtual environment so that we can isolate this project with other projects of us and it also helps us in maintaining different versions of dependencies for different projects

Creating a virtual environment is very simple.

pip install virtualenv
virtualenv django

source django/bin/activate #for ubuntu

django/scripts/activate #for windows
        

The above code will install virtualenv, create a virtual environment called django and activate it which means we are inside the virtual environment now and our project which we will be creating will be isolated. If you want to get out of this environment just use the command deactivate

Step 2: Installing Django

The next step is to install Django

pip install django        

This will install Django. Now you are all set.

django-admin startproject gRPC_project        

The above code will create a Django Project with a name as gRPC_project

lets start a new app and create models and complete basic django setup

python3 manage.py startapp User        

lets create a sample model

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models


# Create your models here.

class User(models.Model):
    first_name = models.CharField(max_length=256)
    last_name = models.CharField(max_length=256)
    email = models.CharField(max_length=256, default='', db_index=True, unique=True)
    created_on = models.DateTimeField(auto_now_add=True)
    modified_at = models.DateTimeField(auto_now=True)

    def __unicode__(self):
        return u'{}-{}'.format(self.email, self.get_user_name())

    def to_json(self):
        return {
            'id': self.id,
            'name': self.get_user_name(),
            'email': self.email
        }

    def get_user_name(self):
        return self.first_name + ' ' + self.last_name        

make migrations and we are done with the basic setup

python3 manage.py makemigrations
python3 manage.py migrate        

lets install the dependencies

asgiref==3.3.4
Django==3.2.4
grpcio==1.38.0
grpcio-tools==1.38.0
protobuf==3.17.3
python-dotenv==0.17.1
pytz==2021.1
six==1.16.0
sqlparse==0.4.1        

this is the basic requirements.txt either copy this and install or run

pip install grpcio
pip install grpcio        

we are done with installing dependencies and seting up the basic configuration

now lets start writing .proto

this is a sample proto for user creation

syntax = "proto3";

package profileservice;

// ProfileService is a gRPC service that defines various RPC methods related to user profiles.
service ProfileService {
  // CreateProfile Method:
  // Creates a user profile based on the provided request.
  //
  // Request:
  //   - first_name: The first_name of the user.
  //     Field number: 1
  //   - last_name: The last_name of the user.
  //     Field number: 2
  //   - email: email of the user.
  //     Field number: 3
  //
  // Response:
  //   - status: The status of the create profile operation.
  //     Field number: 1
  //   - msg: A human-readable message providing additional information about the operation.
  //     Field number: 2
  //   - error: If an error occurred, this field contains a human-readable error message.
  //     Field number: 3
  rpc CreateProfile(CreateProfileRequest) returns (CreateProfileResponse) {};
}

message CreateProfileRequest {
  string first_name= 1;
  string last_name = 2;
  string profile_image_url = 3;
}

message CreateProfileResponse {
  int32 status = 1;
  string msg = 2;
  string error = 3;
}        

#sorry for adding too many comments hope you understand this

now correct the following code and run it

python -m grpc_tools.protoc -I../../protos --python_out=. --grpc_python_out=. ../../protos/profile.proto        

now in the specified location in the above code you can see two .py files are created that are the auto generated code from gRPC which enables to create connection between different servers . the same .proto file have to use in order to connect to the mentioned services

Now lets configure server.py . This is the basic setup of server.py which accept requests and returning the appropriate response

import grpc
import post_pb2
import post_pb2_grpc

class PostService(post_pb2_grpc.PostServiceServicer):
    def CreateProfile(self,request,context):
         respond = profile_pb2.CreateProfileResponse(
            'status' = success
        )
        return respond

def serve():
    server = grpc.server((futures.ThreadPoolExecutor(max_workers=10)))
    profile_pb2_grpc.add_ProfileServiceServicer_to_server(ProfileService(), server)
    server.add_insecure_port('[::]:50051')
    server.start()
    server.wait_for_termination()        

This setup is enough for creating a connection between two servers but this is not enough for accessing the ORM and do database operations because

Django and gRPC works in two different ways so this server file will not have any access to any of the Django files including models . So we have to make the file recognizable by django and we have to give access to django settings .

update the server.py like this

try:  #importing dependencies and system variables
    import os
    import sys
    from concurrent import futures
    import grpc
    from dotenv import load_dotenv
    import django

    load_dotenv()
    
    #ensure that the DJANGO_SETTINGS_MODULE is defined with a default value
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
    
    # Extend sys.path with the Django project path
    project_path = os.getenv('DJANGO_PROJECT_FULL_PATH')
    if project_path:
        sys.path.extend([project_path])
        if 'setup' in dir(django):
            django.setup()
    else:
        raise ImportError("DJANGO_PROJECT_FULL_PATH environment variable not set.")

except ModuleNotFoundError as e:
    print(f"Error: {e}")
    sys.exit(1)

try: #importing all the required django components (models,protos,handlers...)
    from profiles.models import User
    from profiles.protos import profile_pb2_grpc,profile_pb2

    
except ModuleNotFoundError as e:
    print(f"Error: {e}")
    sys.exit(1)


class ProfileService(profile_pb2_grpc.ProfileServiceServicer):
    # """
    # ProfileService handles all the possible gRPC connections mentioned in the 
    # profile.proto file . The methods of this class are responsible for that
    
    # """

    def CreateProfile(self, request, context):
         """CreateProfile Method:
        Creates a user profile based on the provided request.

        Request:
        - username: The username of the user.
        Field number: 1
        - user_id: The unique identifier for the user.
        Field number: 2
        - profile_image_url: The URL of the user's profile image.
        Field number: 3

        Response:
        - status: The status of the create profile operation.
        Field number: 1
        - msg: A human-readable message providing additional information about the operation.
        Field number: 2
        - error: If an error occurred, this field contains a human-readable error message.
        Field number: 3
        """
        try:
        # Get the first_name and last_name... from the request
        first_name = request.first_name
        last_name = request.last_name
        email = request
        User.objects.create(first_name = first_name,last_name =      last_name, email = email)
 response = profile_pb2.CreateProfileResponse(
            status = 200,
            msg = 'profile created',
        )
        
        
    except Exception as e:
        # If an exception occurs, handle it here
        print(f"Error: {e}")
        response = profile_pb2.CreateProfileResponse(
            status = 400,
            msg = 'Invalid Request not enough credentials',
            error = str(e),
        )
        #returning response to client
        return response
 
def serve():
    """Starts the gRPC server to handle PostService requests.

    Creates a gRPC server, adds the PostService implementation, and starts serving
    on the specified port.

    Usage:
        serve()

    Returns:
        None
    """
    # Create a gRPC server with a ThreadPoolExecutor
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))

    # Add the ProfileService implementation to the server
    profile_pb2_grpc.add_ProfileServiceServicer_to_server(ProfileService(), server)

    # Specify the port to listen on (insecure)
    server.add_insecure_port('[::]:50051')

    # Start the gRPC server
    print("server started running on port [::]:50051")
    
    server.start()
    
    #printing this on the console as a conformation for the successful start of the server
    print("server started running on port [::]:50051")

    # Wait for the server to be terminated (e.g., manually stopping the server)
    server.wait_for_termination()

if __name__ == '__main__':
    # If this script is the main entry point, start the gRPC server
    serve()        

in this we have imported every needed modules and configured everything now its time for testing it so lets create a client.py for accessing it

import grpc
from profiles.protos import profile_pb2_grpc,profile_pb2

#step 1 : Create Channel

channel = grpc.insecure_channel('localhost:50051')

#step 2 : Create client

client = profile_pb2_grpc.ProfileServiceStub(channel)

#step 3 : API call
request = profile_pb2.CreateProfileRequest(
     first_name = 'lets',
     last_name = 'do',
     email = 'this'
)
response = client.CreateProfile(request)
print(response.status,response.msg,response.error)
        

Hope this works XD

Conclusion

Although gRPC services have several benefits that make them attractive to businesses and developers, finally, the decision to use gRPC services over others like REST or SOAP depends on your application. While some software will have high-performance benefits with gRPC, others might be better suited to its alternatives. You should understand the drawbacks and advantages of gRPC services and decide whether it works for you.