Has anyone heard of MRAM? Or other memories like it?
MRAM... non volatile, reasonably fast (35 ns), actually available now by Everspin (so far up to 2 MB products). While not big enough in capacity to store everything you ever wanted in there... perhaps it could get use in a smaller system, having the programs stored and run on the same memory chip.
Now how would that affect programming? A memory leak could seriously corrupt other programs because they are being stored in that memory, the RAM / Hard Drive separation of every other system makes it so a memory leak is nondestructive (giving the green light for programmers to program like crap). How does one cope?
Also it seems that if your processor speed isn't too fast, you could make this a memory to memory CPU architecture, without crazy register schemes. Would a stack architecture then be good for this?