That's where you go wrong in my opinion. Doing it together would take double the time or more.

To make this thing happen, one needs to focus on the task. Faffing about with several architectures and plenty of people testing is just not the right approach. The drag from supporting different architectures is not primarilly related to the compilation or testing time as you suggested. It's just a mess to maintain the build system and all dependencies for two different architectures when it comes to a monster like Libre Office.

Seriously, 1 -2 weeks of dedicated work by one guy to make the thing actually run wihout any native GUI on one single machine is what is needed initially. Later on, one can discuss if it makes sense to share code or not.