I was going to say use Gentoo, but if you think Debian requires
Tweaking, you're not going to enjoy Gentoo.  To get a custom, tweaked,
faster system with only the packages you want, you can do worse than to
use Gentoo.  The trade off is that you'll constantly be tweaking and
compiling packages to stay up to date.  Not a great environment to have
for production, but fun for a desktop.

Not really sure what your focus is, though.  Are you running on older
equipment?  Can you upgrade the RAM in your machine? Are you interested
in faster computation of mathematical algorithms?  Using R, Matlab, or
Mathmatica?  Or are you simply not in to graphically intense desktops.

Tips for a faster Ubuntu experience: Compile and use the proprietary
display drivers.  (ATI/Nvidia, etc.)  Turn off Visual Effects in the
Appearance Preferences.  Or simply use a different desktop.  I use Wmii
for a no-frills yet very functional experience.  I only load
applications I want, but then again, that's how I've always done it with
Debian and Ubuntu.  I can't say I've seen this power-hungry monster
you're describing for Ubnutu.  Don't want a full desktop, install with
the alternative CD instead.

(I'm also curious why you think Debian takes a long time to install.)

Anyway.  Good luck!

Chad