Fallout 3 uses the Gamebryo engine, a flexible and robust game development middleware from Numerical Design Limited (NDL), which was later acquired by Emergent Game Technologies.
The Gamebryo engine has been used for many different games across various genres due to its adaptability and flexibility. However, it's worth noting that Fallout 3's version of Gamebryo was heavily modified by Bethesda Softworks, the developer of Fallout 3, to suit the open-world nature of the game.
In addition, Bethesda also developed the Radiant AI system, which is integrated into their Gamebryo-based engine. This system offers advanced AI behaviours, enabling non-player characters in the game to make more realistic decisions based on the world around them.
Please note that while discussing game engines can offer insights into the development process, creating a game like Fallout 3 involves much more than just choosing an engine. Art assets, sound design, story writing, and other factors are all significant components of game development.