> I want to build my own embedded Linux-based OS out a different distro. I know
> one can use Buildroot, etc, but I would like some ideas on architecting such
> a solution rather than picking one.

So a good start is having the requirements and available system

I mainly use debian because i'm a kernel developer, and it gives me
all the tools i need for development on ARM based system. I normally
cross compile the kernel on a big x86 box, but debian gives me the
base system on the target, i can apt-get any special tools i need, and
if i need to do some user space development, a toolchain can quickly
be installed and used, although building anything serious on the
target is slow.

For your user case, how much flexibility do you need? Often embedded
systems have one purpose, so you don't need the flexibility Debian
gives you. buildroot will give you something much more tailored to
your specific use case. Same is true for yocto, etc.

Is security an issue? Embedded systems are often shipped and forgotten
about. But if you want to keep it secure, you are going to have to
rebuild it every few weeks/months. So you need a distribution with
security updates, and a method you can either update individual
packages, or rebuild the whole image and install it.

You say pinball like machine. I'm assuming you mean on screen display,
rather than controlling solenoids, having switch inputs for paddles,
etc? What graphics toolkit do you want to use? QT is often used in
embedded systems, but there are other choices, and ideally you want a
distro which has what you want.

Does the ARM platform you have chosen need binary blobs to make some
of it work? Does it have an open source GPU driver? It can be hard
getting vendor binary drivers working on anything other than what the
vendor supports. So you might want to see what the ARM SoC vendor
supports in terms of distributions.