I've been following emulation for many years and tried keeping up with certain emulators to see how they progressed, so I was colored with surprise when Phil Spencer, the head of the Xbox division, started dropping knowledge-bombs about the technical side of emulation and how Microsoft managed to get the Xbox One to properly emulate the Xbox 360.
Gamespot has a very interesting article captured from Giant Bomb's podcast where Spencer explained exactly how the engineering team at Microsoft were able to take the PowerPC architecture of the Xbox 360 and squeeze it into the x86 architecture of the Xbox One.
The approach that we've taken is to actually emulate the full Xbox 360 hardware layer. So the [operating system] for the 360 is actually running when you run the game, [...] If you watch the game's boot you'll see the Xbox 360 boot animation come up. From a performance standpoint it allows [emulation] to work. We're able to get frame by frame performance equivalents.
Spencer explains that the system things you're playing on Xbox Live because it's emulating the Xbox 360's core operating system. They're doing it via a virtual machine, which is basically running a 1:1 software replica within the confines of another operating system. People do this all the time for old DOS programs, or earlier versions of Windows. Heck, you can even use virtual machines to host LAN parties on your computer.
The head of the Xbox division stated...
The 360 games think they're running on the 360 OS, which they are. And the 360 OS thinks its running on the hardware, which it's not, it's running on an emulated VM. On the other side, the Xbox One thinks it's a game. That's why things like streaming, game DVR, and screenshots all work, because it thinks there's just one big game called 360.
So all of the benefits offered by the Xbox One's three-tiered operating system are available even while playing Xbox 360 games. It's also connecting through the Xbox 360's Live network infrastructure, so that means that for as long as the backwards compatibility is profitable for Microsoft they're likely going to keep the lights on for Xbox Live on the Xbox 360.
Another big question many gamers have is “how do they add new games to the library?” well, it's done through a wrapper on a per-game basis. Back in the old days of the N64 emulation they used to do the same with different Glide wrappers for different games – ones that used special visual or graphical techniques that required a specific wrapper to emulate, especially if you didn't have a 3DFX card or you were using those old Diamond AGP units. A quick example would be Perfect Dark's transparent alpha mapping – it required a special wrapper back then to make that feature usable on some cards. The same thing applied with certain transparent or reflective features for games like Legend of Zelda and Jet Force Gemini.
You download a kind of manifest of wrapper for the 360 game, so we can say 'hey, this is actually Banjo, or this is Mass Effect. The emulator runs exactly the same for all the games.
Unfortunately they haven't quite cracked the code for multi-disc games but they're working on it. So obviously Blue Dragon and Lost Odyssey and L.A. Noire will be a ways off from being compatible with the Xbox One's backwards compatibility.
Given Microsoft's focus on making this dream into a reality, so far it sounds like they're on the right path and some gamers are already stocking up on titles as they prep for the feature to roll out this holiday season.