A quick introduction
Nintendo’s goal was to give players the best graphics possible, for this it will partner with one of the biggest players in computer graphics to produce the ultimate graphics chip.
The result was a nice-looking console for the family… and a 500-page manual for the developer.
Don’t worry, I promise you this article will not be that long… Enjoy!
The main processor is a NEC VR4300 that runs at 93.75 MHz, it’s a binary-compatible version of Silicon Graphics’ MIPS R4300i that features:
- A MIPS III ISA: Succeeds the MIPS II featuring new 64-bit instructions. 64-bit words are called doublewords.
- An internal 64-bit bus while connected to an external 32-bit data bus.
- 5-stage pipeline: Up to five instructions can be executed simultaneously (a detailed explanation can be found in a previous article).
- 24 KB L1 cache: Divided in 16 KB for instructions and 8 KB for data.
An internal 64-bit FPU is also included in this package, the CPU identifies it as a co-processor (COP1) although the unit is fitted next to the ALU and it’s only accessed through the ALU pipeline, meaning there’s no co-processing per se.
Simplified memory access
The way RAM is assembled follows the unified-memory architecture or ‘UMA’ where all available RAM is centralised in one place only and any component that requires RAM will access this shared location. The component arbitrating its access is, in this case, the GPU.
The reason for choosing this design comes to the fact that it saves a considerable amount of production costs while, on the other side, it increments access contention if not managed properly.
No DMA controller?
Due to the unified memory architecture, the CPU no longer has direct access to RAM, so the GPU will be providing DMA functionality as well.
Apart from the UMA, the structure of RAM is a little bit complicated, so I’ll try to keep it simple, here it goes…
The system physically contains 4.5 MB of RAM, however it’s connected using a 9-bit data bus where the 9th bit is reserved for the GPU (more details later). As a consequence, every component except the GPU will only find up to 4 MB.
The type of RAM fitted in the board is called Rambus DRAM or ‘RDRAM’ for short, this was just another design that competed against SDRAM on becoming the next standard. RDRAM is connected in serial (where transfers are done one bit at a time) while SDRAM uses a parallel connection (transfers multiple bits at a time).
RDRAM’s latency is directly proportional to the number of banks installed so in this case, with the amount of RAM this system has, the resulting latency is significant.
By contrast, the amount of available RAM on this console can be expanded by installing the Expansion Pak accessory: A fancy-looking small box that includes 4.5 MB. Curiously enough, the RAM bus must be terminated, so the console always shipped with a terminator (called Jumper Pak) fitted in the place of the Expansion Pak. Now, you may ask, what would happen if you switch on the console without any Pak installed? Literally nothing, you get a blank screen!
The core of the graphics reside on a huge chip designed by Silicon Graphics called Reality Co-Processor running at 62.5 MHz. This package contains a lot of circuitry so don’t worry if you find it difficult to follow, the graphics sub-system has a very complex architecture!
The main philosophy of this design is that the GPU is not meant to be a ‘simple’ rasteriser like the competitor’s. Instead, it should also be capable of accelerating geometry calculations (offloading the CPU), and for that, more circuitry will be needed.
Having said that, this chip is divided into three main modules, two of them are used for graphics processing:
Reality Signal Processor
Also known as RSP, it’s just another CPU package composed of:
- The Scalar Unit: A MIPS R400-based CPU which implements a subset of the R400 instruction set.
- The Vector Unit: A co-processor that performs vector operations with 32 128-bit registers. Each register is sliced in eight parts to operate eight 16-bit vectors at once (just like SIMD instructions on conventional CPUs).
- The System Control: Another co-processor that provides DMA functionality and controls its neighbour module, the RDP (more about it later on).
In order to operate this module, the CPU stores in RAM a series of commands called Display list along with the data that will be manipulated, then the RSP reads the list and applies the required operations on it. The available features include geometry transformations (such as perspective projection), clipping and lighting.
This seems straightforward, but how does it perform these operations? Well, here’s the interesting part: Unlike its competitors (PS1 and Saturn), the geometry engine is not hard-wired. Instead, the RSP contains some memory (4 KB for instructions and 4 KB for data) to store microcode, a small program, with no more than 1000 instructions, that implements the graphics pipeline. In other words, it directs the Scalar Unit on how it should operate our graphics data. The microcode is fed by the CPU during runtime.
Nintendo provided different microcodes to choose from and, similarly to the SNES’ background modes, each one balances the resources differently.
Reality Display Processor
After the RSP finished processing our polygon data, it will start sending rasterisation commands to the next module, the RDP, to draw the frame. These commands are either sent using a dedicated bus called XBUS or through main RAM.
The RDP is another processor (this time with fixed functionality) that includes multiple engines used to map textures onto our polygons, mix colours and build the new frame.
It can process either triangles or rectangles as primitives, the latter is useful for drawing sprites. The RDP’s rasterisation pipeline contains the following blocks:
- A Rasteriser: Allocates the initial bitmap that will serve as frame-buffer.
- A Texture Unit: Processes textures using 4 KB of dedicated memory (called ‘TMEM’) allowing up to eight tiles to be used for texturing. It can perform the following operations on them:
- Bilinear filtering: Maps the selected 2D texture over the 3D shape and smooths it to avoid pixelated areas.
- A ‘complete’ filter would require four points to carry out the interpolation, however, this console only uses three (triangular interpolation) resulting in some anomalies. Thus, certain textures will have to be ‘adapted’ beforehand.
- Mip-Mapping: Automatically selects a scaled-down version of the texture depending on its level of detail. This avoids computing large textures that would be seen far away from the camera and prevents aliasing.
- If enabled, the N64 maps textures using trilinear filtering instead. This new algorithm will also interpolate between the mipmaps to soften sudden changes in the level of detail.
- Perspective correction: Chosen algorithm for mapping textures onto triangles. Unlike other algorithms, this one takes into account the depth value of each primitive, achieving better results.
- Bilinear filtering: Maps the selected 2D texture over the 3D shape and smooths it to avoid pixelated areas.
- A Colour Combiner: Mixes and interpolates multiples layers of colours (for instance, to apply shaders).
- A Blender: Mixes pixels against the current frame-buffer to apply translucency, anti-aliasing, fog, dithering. It also performs z-buffering (more about it later on).
- A Memory interface: Used by the previous blocks to read and write the current frame-buffer in RAM and/or fill the TMEM.
The RDP provides four modes of functioning, each mode combines these blocks differently in order to optimise specific operations.
Since this module is constantly updating the frame-buffer, it handles RAM very differently: Remember the unusual 9-bit ‘byte’? The ninth bit is used for frame-buffer-related calculations (like z-buffering and antialiasing) and can only be accessed through the Memory interface.
The resulting frame-buffer must be sent to the Video Encoder in order to display it on screen (DMA and the Video Interface component are essential to accomplish this).
The theoretical maximum capabilities are 24-bit colour depth (16.8 million colours) and 640×480 resolution (or 720×576 in the PAL region). I mention it as ‘theoretical’ since using the maximum capabilities can be resource-hungry, so programmers will tend to use lower stats to free up enough resources for other services.
Let’s put all the previous explanations into perspective, for that I’ll borrow Nintendo’s Super Mario 64 to show, in a nutshell, how a frame is composed:
Models and Lighting
In order to save polygons, some characters are modelled using sprites (quads)
To start with, our 3D models are located in the cartridge ROM, but in order to keep a steady bandwidth, we need to copy them to RAM first.
Then it’s time to build a scene using our models, the CPU could do it by itself but it may take ages, so the task is delegated to the RCP. The CPU will instead send orders to the RCP, this is done by carrying out these tasks:
- Compose the Display List that contains the operations to be carried out by the RSP and store it in RAM.
- Point the RSP where the display lists are.
- Send microcode to the RSP to kickstart the Scalar Unit.
Afterwards, the RSP will start performing the first batch of tasks and the result will be sent to the RDP in the form of rasterisation commands.
Textures and Rasterisation
So far we managed to process our data and apply some effects on it, but we still need to:
- Apply textures and other effects.
- Draw a frame-buffer, so we can display it on a screen.
As you may guess, these tasks will be performed by the RDP. The CPU will feed the data (such as textures) by placing it on RAM, this module has a fixed pipeline but we can select the optimal mode of operation based on the current task to improve the frame-rate.
Once the RDP finishes processing the data, it will write the final bitmap to the frame-buffer area in RAM. Afterwards, the CPU must transfer the new frame to the Video Interface (preferably using the DMA) which will in turn sent it to the Video Encoder for display.
Here are some examples of previous 2D characters for the Super Nintendo that have been redesigned for the new 3D era, they are interactive so I encourage you to check them out!
Modern visible surface determination
If you’ve read about the previous consoles, you came across the never-ending problem regarding visibility of surfaces and by now may think polygon sorting is the only way out of this. Well, for the first time in this series, the RDP features a hardware-based approach called Z-buffering. In a nutshell, the RDP allocates an extra buffer called Z-Buffer in memory. This has the same dimensions of a frame-buffer, but instead of storing RGB values, each entry contains the depth (Z-value) of the nearest pixel with respect to the camera.
After the RDP rasterises the vectors, the z-value of the new pixel is compared against the respective value in Z-buffer. If the new pixel contains a smaller z-value, it means the new pixel is positioned in front of the previous one, so it’s applied onto the frame-buffer and the z-buffer is also updated. Otherwise, the pixel is discarded.
Overall, this is a huge welcomed addition: Programmers do not need to worry anymore about implementing software-based polygon sorting methods which drain a lot of CPU resources. However, Z-buffer does not save you from feeding unnecessary geometry (discarded or overdrawn, both consuming resources). For this, game engines may choose to include an occlusion culling algorithm to discard unseen geometry as early as possible.
Secrets and limitations
SGI clearly invested a lot of technology into this system. Nonetheless, this was a console meant for the household and as such, it had to keep its cost down. Some hard decisions resulted in difficult challenges for programmers:
Due to the huge number of components and operations in the graphics pipeline, the RCP ended up being very susceptible to stalls: An undesirable situation where sub-components keep idling for considerable periods because the required data is delayed at the back of the pipeline.
This will always result in performance degradation and is up to the programmer to avoid them. Although to make things easier, some CPUs such as the Scalar Unit implement a feature called Bypassing which enables to execute similar instructions at a faster rate by bypassing some execution stages that can be skipped.
For example, if we have to compute sequential
ADD instructions there’s no need to write the result back to a register and then read it back every time each
ADD is finished, we can instead keep using the same register for all additions and do the write-back once the last
ADD is completed.
Inside the RDP there are 4 KB Texture memory available to be used as Texture Cache, its main goal is to avoid stalling read cycles from RAM. Unfortunately, in practice 4 KB happened to be insufficient for high-resolution textures. Furthermore, if mipmapping is used, the available amount of cache is then reduced to half.
As a result, some games used solid colours with Gouraud shading (like Super Mario 64) and others relied on pre-computed textures (for example, where multiple layers had to be mixed).
Before we go into the details, let’s define the two endpoints of the audio sub-system:
- Our starting point is the cartridge ROM, it contains data that only the CPU can interpret.
- The ending point is the Digital-to-Analog converter or ‘DAC’, which only understands waveform data.
Now, how do we connect both ends? Consoles normally include a dedicated audio chip that does the work for us. Unfortunately, the Nintendo 64 doesn’t have such dedicated chip, so this task is distributed across these components:
- The main CPU: Transfers the audio data from the game’s ROM to RAM, then it initialises Audio Lists to be used by the RSP.
- The RSP: With the use of even more microcode, it interprets the audio lists previously stored in RAM and performs the required operations to the audio data which, for example, can include:
- Uncompressing ADPCM samples and applying effects.
- Sequencing and mixing MIDI data using audio banks stored in RAM as well.
The resulting data is, as expected, waveform data. This is then sent to the Audio Interface or ‘AI’ block which will then transfer it to the digital-to-analogue converter. The resulting waveform contains two channels (since our system is stereo) with 16-bit resolution each.
Time to checkout the soundtracks made for the N64. There are too many (good ones) to mention in this article, so here are some that caught my attention:
The music of this game is tied to its daunted atmosphere
This game has nice and unique house-based soundtrack
Secrets and limitations
Because of this design, the constraints will depend on the implementation:
- Sampling rate can be up to 44.1Hz, but using the top rate will steal lots of CPU cycles.
- There’s no strict limit in the number of channels, it all depends how much the RSP is capable of mixing (often around 16-24 channels if processing ADPCM or ~100 if PCM).
- Memory is another concern, while competitors relied on larger mediums (i.e. CD-ROM) and dedicated audio memory, Nintendo 64 cartridges hold much less data (let alone music data) and have to share its main memory with other components. For those reasons, players may notice that N64 ports contain lesser quality music or repeated scores.
- A method for overcoming this limitation consisted in implementing a software sequencer that could ‘construct’ the samples at runtime using a pre-populated set of sounds (similar to MIDI music).
Similar to the PS1 and Saturn, N64 games are written for bare-metal. However, there are no BIOS routines available to simplify some operations. As a substitute, games embed small OS that provides a fair amount of abstraction to efficiently handle the CPU, GPU and I/O.
This is not the conventional desktop OS that we may imagine at first, it’s just a micro-kernel with the smallest footprint possible that provides the following functionality:
- Multi-Threading (bear in mind the CPU is single-core).
- Scheduling and Pre-emption.
- Simplified register and I/O access.
The kernel is automatically embedded by using Nintendo’s libraries, additionally, if programmers decide not to include one of the libraries, the respective portion of the kernel is skipped to avoid cartridge space being wasted.
As you know by now, I/O is not directly connected to the CPU, so the RCP’s third module (which I haven’t mentioned until now) serves as an I/O interface, it basically communicates with the CPU, controllers, game cartridge and Audio/Video DACs.
Nintendo held on to the cartridge medium for storage and as a consequence, games enjoyed higher bandwidths (between 5-50 MB/s depending on the ROM’s speed) while being more expensive to produce. The biggest cartridge found in the market has 64 MB.
Inside cartridges manufacturers may include extra memory (in the form of EEPROM, flash or SRAM with a battery) to hold saves, however this is not a strong requirement any more since certain accessories could be used to store saves as well.
The Nintendo 64 controller included a connector used to plug in accessories, some of them are:
- The Controller Pak: Another medium (similar to Sony’s Memory Card) used to store save data and use it on other consoles.
- The Rumble Pak: Contains a small motor in order to provide haptic feedback, useful for immersing the player on certain games.
All accessories connected to the controller are managed by the Peripheral Interface.
Apart from that, this console included a special connector at the bottom of its motherboard which was meant to be used by the yet-unreleased Disk drive, some sort of an ‘extra floor’ that contained a proprietary disk reader, the drive was only released on Japan nonetheless and eventually cancelled for the rest of the world.
Source Development Kit
In general, development was mainly done in C, assembly was also used to achieve better performance. While this system contained a 64-bit instruction set, 64-bit instructions were rarely used since in practice, 32-bit instructions happened to be faster to execute and required half the storage.
Libraries contained several layers of abstractions in order to command the RCP, for example, structs like the Graphics Binary Interface or ‘GBI’ were designed to assemble the necessary Display lists more easily, the same applied for audio functions (its struct was called Audio Binary Interface or ‘ABI’).
In terms of microcode development, Nintendo already provided a set of microcode programs to choose from, however if developers wanted to customise it, that would indeed be a challenging tasks: The Scalar Unit instruction set wasn’t initially documented (at the request of Nintendo, of course), later on the company changed its position and SGI finally released some documentation for microcode programming.
Hardware used for development included workstations supplied by SGI, like the Indy machine which came with an extra daughterboard called U64 that contains the hardware and I/O of the retail console. Tools were supplied for Windows computers as well.
Other third-party tools consisted in cartridges featuring a long ribbon cable that connected to the workstation. This cartridge fitted in a retail Nintendo 64 but included internal circuitry to redirect the read requests from the console to the workstation’s RAM. The deployment/debugging process was carried out by transferring a copy of the game to RAM and then, when the console was switched on, it would start reading from there.
Anti-piracy / Region Lock
The anti-piracy system is a continuation of the SNES’ CIC. As you know, bootleg detection and region locking are possible thanks to the CIC chip (which must be present in every authorised game cartridge), the Nintendo 64 improved this system by requiring different games to have a specific variant of the CIC chips in order to make sure the cartridge was not a counterfeit or contained a CIC clone, the Peripheral Interface or ‘PIF’ would do checksum checks at the start and during gameplay to supervise current CIC installed on the cartridge.
If by any reason the PIF considers the current cartridge is not valid, it will then induce the console in a permanent freeze.
Region-locking was done by slightly altering the shape of the cartridge between different regions so the user can’t physically insert the game on an N64 from a different region.
Overall, there was not too much concern regarding piracy thanks to the use of cartridge medium, although game prices were three times higher than CD-based ones.
As silly as it may seem, Nintendo left one door opened: The Disk Drive port.
A few companies reversed engineered the interface in order to develop their own hardware, and some of the resulting products became a concern for piracy.
I guess the one worth mentioning is the Doctor v64, this device has the same shape as the Disk Drive port but included a CD-ROM drive that’s used to clone the contents of the cartridge to a CD, the opposite (reading Roms from a CD) is also possible.
When I was a kid I used to play some N64 games on a Pentium II machine using an emulator, it wasn’t that bad but then I wondered now how the freck was it able to happily emulate a complex 64-bit machine since, among other things, my PC barely had enough RAM to keep the integrated video alive.
The truth is, while reproducing the architecture of this console can be complex, things like microcode will give a hint of what the console is trying to do, and since emulators don’t have to be cycle-accurate, they can apply enough optimisations to provide more performance in exchange for real emulation.
Another example are the 64-bit instructions, since games barely used them, emulation speed would hardly be hit when running on a 32-bit host machine.
That’s all folks
I have to say, this article may be the longest one I’ve ever written, but hopefully you found it a nice read!
I’ll probably take the following days to tide up some things on the website instead of starting to write the next article.
Until next time!
Sources / Keep Reading
Graphics / Audio
- Motherboard and console: Evan Amos Gallery
- Diagrams and game screenshots: Me
This article is part of the Architecture of Consoles series.
If you found it interesting please consider donating, your contribution will be used to get more tools and resources that will help to improve the quality of current articles and upcoming ones.
A list of desirable tools and latest acquisitions for this article are tracked in here:
## Interesting hardware to get (ordered by priority) - Nintendo 64 console and a controller (£40 - £60) - A N64 game (£5 - £15) - Any Dev kit (only if found at a reasonable price) - N64 DD (only if found at a reasonable price) - N64RGB board (£25) - RAD2X (£47.99)
Always nice to keep a record of changes.
## 2020-05-08 - Added Mario 64 screenshot with original resolution ## 2020-04-22 - Improved 9-bit explanation. Thanks /u/mallardtheduck ## 2020-04-14 - Small changes to perspective correction info ## 2020-04-11 - Dedicated more text to the visibility approach ## 2020-02-26 - Expanded texture unit section. ## 2020-01-11 - Added repertoire ## 2019-10-29 - Added some 3d models to fiddle with ## 2019-09-17 - Added a quick introduction - Corrected some explanations ## 2019-09-12 - Released to the public, yay