So, How Do Computer Graphics Work?
One of the open threads left in our article on the fast inverse square root algorithm was how exactly all those advanced 3D graphics worked. And while I don’t want to get too technical or mathematical, I do think it’s worth returning to this topic to get the 101 on computer graphics.
Basics of 3D Graphics
Your screen is made up of a bunch of really small light cells. The way in which these light cells differ depends on what exact screen you’re using, but they all have the same purpose: when your computer sends a message saying a certain cell should be a certain color, the cell changes to that color. There are millions of these cells, and they are sometimes changing hundreds of times a second.
Now, how this approach works when it comes to 2D graphics is pretty easy to understand. Right now, my Google doc is a white background with black text (I left my dark theme at home). The code Google made tells the browser to make it this color, the browser tells the OS it needs to be this color, and the OS tells the computer to change my monitor to that color. Once again, at a high-level, this isn’t anything too crazy.
But 3D is weird. Our monitors obviously cannot handle 3D – even VR headsets, which are made to look 3D, are mostly just relying on perspective tricks using the lenses. So why does it not feel like a 2D image when we play video games?
A few changes to the traditional formula are done here. The first is that we create a 3D Model – don’t think of this in the colloquial “3d model” term like a .obj or .dae file, but as in literally a mathematical map of 3D space. We use x,y,z coordinates to illustrate points and faces of entities within our new 3D world. But simply saying where these things would be if they were 3D doesn’t really do anything – if we tried to render it, it would still look like a plain 2D image.
This is where those “advanced mathematical equations” come into play.
Basically, we want to make our 2D scene as realistically 3D as possible. So, what better way to do this than to use real world equations? We can get information on how light reflects on objects, how our eyes perceive depth, or how physics works – and then we can apply it to the 3D Model in order to give us the illusion that it’s all happening in 3D space.
Primarily, there are four major categories in which we use to define realism in a 3D Model: camera transformation, shading, shadowing, and depth buffers. As is usually the case, the best way to explain this is through an example.
How to Make a 3D Model
Traditionally, there’s two ways of making a 3D model. If you’ve ever tried to do it before, it’s likely you were using software such as Blender or Unreal. These are 3D engines – they do all the advanced math for you on the backend, so all you have to do is press a couple of checkboxes, alter a few variables, and change things around in an already existing 3d space. But there’s also 3D APIs – these are programming languages that allow you interface directly with a GPU, a specialized part of a computer specifically created to handle this advanced mathematics. Blender and Unreal are actually built using these APIs, and even if you haven’t used them directly before you might be surprised to find they’re already installed onto your computer!
Recommended by LinkedIn
There are many of these 3D APIs available – OpenGL, WebGL, Vulkan, and DirectX are among the biggest. Fortunately for you, we’re going to keep it general and so you won’t need to learn one of these APIs or boot up Visual Studio in order to understand what we’re talking about.
The first thing you do in any 3D API is create a window. This is actually unrelated to the process of making our 3D Model, but is such a fundamental part of the process that it’s impossible to start with anything else. This window basically just allows the OS a viewpoint into this 3D world you’re creating – much like how a browser window allows the OS to view a connection to the internet, etc.
In the next step, we create an entity (such as a cube) in our 3D Model by describing vertices (specific x/y/z points) and indices which define faces (the blocking that encloses points, thus making them a 3D object). As I’ve mentioned before this still doesn’t make the scene look 3D, but it helps our mathematical model understand where certain “blocking” points are that would require light reflection or depth perception.
After we create these vertices/indices, we ship them off to the GPU’s memory. As I’ve said, the GPU does the heavy lifting in the wide majority of 3D calculations, so we’re keeping the GPU informed of where things are in our 3D space so it can make speedy calculations when we tell it to place something like a light or a camera.
Speaking of light, our next step is the shader. A shader is arguably the most important piece of the computer graphics puzzle – at a very basic definition, it defines how texture changes on an object when light shines on it. But there’s a lot of ways we can define texture: color, roughness, shininess, transparency, and many more that are outside the scope of this article!
Typically, “shading” and “shadowing” are handled by the shader, whereas “camera transformation” and “depth buffering” are handled by the API. They are pretty easy to package together: shading and shadowing are based on light interactions (the way an object shines and the shadow it leaves behind) and transformation/buffering are based on camera interactions (if a camera is at point A, what do we see? How are objects layered on top of each other, and are certain ones easier to see than others?). So, that segues us very nicely into the final chapter of this article…
Adding Spice with Cameras and Lighting
Now, hold on a minute – we added the ability for our 3D Model to handle lighting, but we didn’t actually add any lights! Beyond that, how does our 3D Model know which way to face the scene if there’s no camera?
Fortunately, this is pretty easy to do in the API. Both consist of three components: a point in x/y/z space (its origin), a vector (the direction it’s facing) and a ray (how far it interacts with other objects in the x/y/z space). If you read our FISR article, this should all be sounding pretty familiar!
When it comes to the camera (also sometimes called the “viewpoint”), the complexity usually ends there. But for lighting – arguably the most important entity that sets up a believable 3D scene – there’s a few more things we can tweak. Like shading, this could be its own article – but I do want to mention one category in particular.
If we want to calculate light, there’s two ways we can do it: ambient lighting, and ray-traced lighting. Ambient lighting is the more basic calculation – it is usually just an equal spread of light, either around the entire origin (a “point” light) or in the direction of the vector (a “spot” light). However, ray-tracing – a relatively new invention – uses real-world light physics to determine how far any given ray of light would go, and how it would look, based on its journey from the origin. Engines like Blender and Unreal now have native support for ray-tracing, but it still requires a pretty beefy GPU to compute!
Anyway, that’s all for our Computer Graphics 101. We still have a lot we can cover – such as shading and lighting – so I can see this topic becoming a playlist on here. Let me know in the comments if there’s anything you’d like to see in particular!