Evolution of Web Application Architecture

It's been interesting to see how web applications have evolved over my career. Understanding how it's changed, and perhaps as importantly, why it's changed is important, especially when it comes to security.

Baremetal

I mostly have "grown up" in the cloud space but a good portion of my early career was actually with a startup that ran in a colo. While I was intended to be a full-stack developer, I was also volun-told for a whole host of extra duties such as swapping out the tape backup, being responsible for the cable room which housed our dev servers, and being the only person to stay in the office "just in case" when they all went to the colo one day.

During this time period, startups often had servers in a colocation (colo) which was a shared datacenter where you usually rented out some rack space. It was much cheaper than building out your own datacenter but was still fairly expensive.

Since everything ran on the physical server itself and the minimum size where the dollars made sense was often way more than a single web application needed, a single server might run many things. This was especially true if you had a decent amount of traffic but nothing that really had to scale. So you might have your HTTP server, the php code, the database, a bunch of cronjobs, and potentially more on a single server.

These servers were often directly on the public Internet. There rarely was much segmentation between the services, so vulnerabilities could get chained together. If someone put a piece of software with a web UI and the default username and password, it could be potentially be used to get sensitive info and more.

Additionally, it took a fair amount of time and money to create a new server since you had to have the physical hardware. So you really had to be careful with the servers.

Virtual Private Servers

At my next start up, we moved away from colos to the land of virtual private servers. We happened to use Slicehost which was amazing at the time. I remember being able to go into a chat and talking with the folks running it and getting help.

These weren't much different than the baremetal ones in terms of size and how we ran them. But we could start to do things like blue-green deployments, temporary machines for special jobs, etc.

EC2

At another startup, after our VPS provider had a few outages, we decided to move over to AWS and EC2. This is where things suddenly changed. The recommended adoption strategy was to stop creating these large machines that ran everything, and have separate machines for different functions, such as web server, database, memcache, and cron. We still kept our http server and our application code together, especially since we were running Perl, but I set to work separating out these intertwined pieces.

A major driver for this change was two-fold. First, virtual instances like ec2 were so much easier to set up. An API call could get you a brand new server, which meant you could scale your infrastructure to meet demand (although AWS autoscaling hadn't been invented quite yet).

In addition, it could often be cheaper to scale more limited purpose machines horizontally than it would be scale the all-in-one machines. If one of the services suddenly put you over the threshold for your current instance size and you had to scale, you'd likely have to run all the same services on the second machine. Whether you horizontally or vertically scaled, you were looking at doubling your cost.

As with the VPS option, EC2 works by using a hypervisor on servers to provide the virtual machines. By running just one type of service on each EC2 instead of everything, the separation provided by the hypervisor could increase security by segmenting different types of services.

Containerization

The smaller virtual machines that services like AWS' EC2 or OpenStack offered allowed for greater segmentation and separation. However, we ran into a lot of issues like how to upgrade shared systems libraries, all the resources needed to store and run full operating systems, etc. So the next jump was containerization.

Containers let us package together just what is needed to run a particular application and often we package it with a very slimmed down operation system like alpine. This dramatically reduces the attack surface for that application since it's hard to exploit a vulnerability that doesn't exist.

Additionally, containers allow for far more immutability, which further restricts attack surface. Since we are pre-baking most of the application into a container image, it's also much easier to bring a new version into existence, allowing us to cycle often, allowing for faster recovery and also reducing persistence.

With containers we have 3 levels of separation:

• Host - The hardware layer and the hypervisor.
• Virtual Machine - The guest OS that runs the container or other services
• Container - The application and its immediate dependencies

Kubernetes

To be fair, Kubernetes really needs its own article. The simple description of Kubernetes is that it's a way to orchestrate containers and fit as many as possible within the cluster footprint. But in reality it goes way beyond that and each cluster is essentially its own datacenter.

The big thing here is that we can run multiple applications on the same server again but they should be isolated from each other and they should only have within each container what they need to run, further reducing the interaction of vulnerabilities. System libraries at the guest OS level can be updated without impacting the applications. Applications can move around the cluster as needed and we can get good density of applications for the available resources.

What's happen is that in many ways we've made the patterns for deploying securely simpler but at the cost of a lot more layers, some of which are more complex than others.