Exploring security vulnerabilities by exploiting buffer overflow using the MIPS ISA

By exploiting a well known security vulnerability in many C library implementations, it is possible for an unprivileged user to gain unrestricted system privileges. With an understanding of how the process execution stack is allocated and managed during process execution, a user can override the return address of a C library routine and thereby resume execution at a different address where a set of malicious functions can be invoked [1]. This is known as the buffer overflow exploit. With buffer overflow as the underlying theme, an example will be described using C and the MIPS assembly language that simultaneously exposes students to issues in computer security, operating systems concepts such as memory management and function invocation/return, and the MIPS instruction set architecture.