Grand Theft Auto V (GTA 5) uses the RAGE engine, short for Rockstar Advanced Game Engine. This engine was developed by Rockstar San Diego and has been used in many of Rockstar Games' popular titles, including Red Dead Redemption and Max Payne 3.
The RAGE engine supports a wide range of features required for open-world games, like advanced artificial intelligence, high-quality graphics, physics simulation, and detailed environments.
However, it's important to note that GTA 5 doesn't use just the RAGE engine in its pure form. The game also involves plenty of proprietary technology developed by Rockstar North specifically for GTA 5, such as Euphoria dynamic animation engine and Bullet Physics Library to handle behaviours of both characters and vehicles. RAGE is not publicly available for other developers to use.