Re: [gentoo-soc] My GSoC Gentoo 2018 Proposal for "Portage powered Android"

[Benda Xu <heroxbd@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 13393 bytes --]

Hi Pengcheng,

Very well written proposal indeed!  I enjoyed reading it.

Pengcheng Xu <i@jsteward.moe> writes:

> After getting in touch with the mentor of the idea, Benda Xu (heroxbd)
> and exchanging ideas, I’ve made a draft on my ideas and a rough plan
> about what I’m going to do, forming a proposal.  It’s available as PDF
> in the attachment. Feedbacks wanted!

> Gentoo GSoC 2018 Proposal ​
> [draft]
> Portage powered Android

Just a random thought, you might want to revise the project name to make
it more catchy.

> Idea overview:

> ...from Google Summer of Code/2018/Ideas/Portage powered Android (by
> heroxbd): The Android custom ROM development has been based on
> cross-compilation and flashing whole partitions. This paradigm has
> been serving well for the embedded system developments. But as the
> performance of personal mobile devices boost, it becomes feasible and
> desirable to introduce package management like personal
> computers. Package management will make software installation and
> update reliable, reproducible, incremental and convenient.

> This project aims to introduce Gentoo's prestiges package manager,
> portage, to manage Android software stack. We are going to use the
> Gentoo on Android as a starting point.  Starting from the GNU userland
> provided, we are going to work with the LineageOS build system based
> on the Android Open Source Project, to write ebuilds for the
> individual components. Starting from the linux kernel first, the
> Android will be reproduced from bottom up.

It will become better if you could reduce the quote into 1-2 sentences
and rewrite the overview in your own words.

Also, if you need to get familar with AOSP build system for this
project, better get preapred now.

> A more detailed look:

> 1. The aim of this project, from my perspective of view, is to
> completely rebuild Android from bottom up, starting with getting hand
> on the raw, untouched build system of Android (or more specifically,
> its Lineage OS distribution) and produce a working system, so that we
> can verify that everything is working (unlikely), or fix up whatever
> problems (very likely) we may run into when building up the Lineage
> (large part of it AOSP) tree. Most importantly, this can get us
> familiar with how the existing Android build system works,

us == traditional GNU/Linux distribution community?  Because getting
familiar with the AOSP is not an interest for the AOSP community.

> while in the meantime getting familiar with the various tools to work
> with Android, from cross-compile toolchains, deploying tools to
> maintenance tools on the device (e.g. bootloader & recovery).

> 2. Then, to gain control of the system and unleash the full power of
> our GNU tools, I believe it necessary to place ourselves a layer ​below
> ​the existing Android system, so that we can easily poke around in the
> Android system in order to update it the way we want (the main purpose
> of this project). I’ve come up with two approaches (call it Approach a
> and b). Both of which require tampering the initramfs (maybe more),
> but does not involve touching the whole bunch of mess inside Android:

> a. From the entry point in the initramfs, we launch a small loader to
> fork our real init into background, 

Is it possible just to have our real init call the Android init?  Then
no background forking is needed.

> and then fires up the Android init. Our launched init waits for
> Android init finishing initializing the devices, then we can continue
> our job, with the rest of the Android system booting up in the
> meantime. (This idea originally came from this thread ​[1]​;

s/this thread/this page/ ?

> the difference is that we’ll be trying to launch a full-blown init
> system, so that a great amount of packages in the existing Portage
> tree can utilize their service files (whether systemd units or OpenRC
> scripts) without modifying.

s/modifying/modification/ .

> b. Launch the system with our own (real) init, initializing the core,
> necessary devices with udev (as what a normal Linux system will do),
> then launch the Android (sub)system inside a container. This is
> actually the technique used in the Android subsystem in Chrome OS ​[2]​,
> with the help of various namespace facilities provided by the Linux
> kernel. 

Also ARC++, 

  https://en.wikipedia.org/wiki/Google_App_Runtime_for_Chrome
  https://lwn.net/Articles/701964/

I am not sure if ARC++ needs to execute Android init.  Have you checked
that?

> This approach looks more promising and with far less hacky-smells, yet
> it definitely requires more investigation and work.

If a central idea needs investigation, better do it now.

> 3. After we have acquired the level of control and, meanwhile, have
> the Android environment to deal with running somehow separated from
> our existing Linux system (it’s a complete Linux system at this
> point)

GNU system?  GNU/Linux system?  This terminology can be comfusing
because Android itself is also on top of Linux.

> , we can start to move things inside Android territory into the
> management of Portage.

> I’ve splitted this process into the following stages:

> a. Get the Android kernel source sorted out and packaged as some sort
> of kernel source package. The final form should be something like
> sys-kernel/android-sources with ​USE_EXPAND like ​PLATFORM​, which
> denotes the variant of the kernel source according to different
> devices (like ​PLATFORM=”hammerhead”​); or something like
> android-kernel/hammerhead-sources​. This enables us to manage the
> source and build process of the kernel more easily, as well as
> tweaking options and applying patches. (maybe merging mainline kernel
> as well, but that’s another story)

Nice!

> b. Try building some parts of the Android system (or more precisely,
> some components) separately, and get a more detailed understanding of
> the Android build system (namely ​Android.mk and friends). This process
> is necessary, as the original Android build system is somehow a tied
> piece of machinery, which is designed to work as a whole. Considerable
> amount of work is needed to take this system apart, along with (maybe
> quite a lot of) investigating, experimenting and hacking.

If decoupling is difficult, building the whole AOSP from a single ebuild
is also acceptable.  I can split them up later.

> c. After we have got a complete understanding of the Android build
> system, as well as having been able to carve a part of it out without
> much difficulties, we should start composing something like
> ​android.eclass​, so that we can automate the process in handling the
> Android components with Portage. The advantage of this stage is that
> once this eclass is built, we can automate the process of building any
> possible new or changed Android components, as well as reducing the
> hacking work needed for each individual component of the Android
> system. (Needing to hack every piece sounds like an impossible task.)

Indeed.

> d. The only job left for now would be to adopt every part of the
> Android system to using the freshly-made eclass. This is also an
> intensive testing process for the eclass, as surely many bugs and
> uncaught corner cases will pop out here and there. This can enable us
> to refine the eclass further. Another important thing at this part is
> to make sure some sort of isolation in different versions of the core
> system, as there’s a compatibility guarantee for Android applications
> called API level. We need to make sure that we leverage Portage
> techniques such as profiles to isolate different API levels apart, so
> as not to mix the system up across different versions, thus breaking
> the compatibility guarantee. Meanwhile, implementing this level of
> isolation with profiles allows us to switch the profile and then roll
> the system up to the new API level (no annoying blockers as that’s
> somehow unrelated to the “base system” that’s used to compile the
> Android land).

Exactly.

> 4. After 3, the main Android system would be sitting inside the
> container / chroot safe and sound

If the Android is in /system, is a container still necessary?

> , with all parts of it being managed by Portage. This stage 4, as in
> my plan, would be to clean up the work and sort out a way to
> conveniently distribute the work.  Required things would be:

> a. Blog (intensively?) about the work done, so as to enable
> reproducing the entire process from scratch (in case this project is
> later abandoned and forgotten in the dust, yet someone (?) from the
> future wants to pick it up) -I’ve had quite unpleasant memories from
> experience with Gentoo/FreeBSD, trying to guess how things worked from
> void. Related documentation should be summed up and put into Gentoo
> Wiki, perhaps.

I prefer blogging happening along the work, instead of a series of final
summaries.  That will give you early feedbacks.

> b. Develop a way to distribute the initial setup process, as we can’t
> expect every user to start with the bootstrap process. Alternatively,
> offer a way to convert existing systems into the system we’re trying
> to build in discussion. The first method is preferred, though, as we
> reserve right to change the structure of the system instead of using
> what’s determined by the existing Android system, such as partition
> schemes, filesystem choices and filesystem root layouts.

As the first step, please make sure a Gentoo-Handbook-like documentation
is available.  Then we can talk about user-friendliness to the users
outside Gentoo.

> c. Set up build hosts for different devices, and (optionally) handle
> the production of binary packages. Though this may sound contradicting
> with the Gentoo philosophy, but building the Android system can be
> tiring and resource-consuming, while requiring users to set up
> cross-compile hosts sounds cumbersome. Or, at least, encourage the use
> of a building hosts locally or at small scales.

Same as the above.

> Expected outcomes ​
> (tl;dr)​
> :

> 1. What we get: 

> a. A working (perhaps no framebuffer support though; I believe that
> belongs to another project as it would be distracting and too much
> work) 

Avoid writing long sentence in paranthesis.  If you want to cover this
point, write it out in the paragraph.


IMHO, for me I am more interested in a working framebuffer than e.g., a
binary package repository.  But you have the final word to prioritize
the tasks.

Alternatively to a traditional framebuffer, frecon might be insightful:

  https://chromium.googlesource.com/chromiumos/platform/frecon/+/HEAD/DESIGN-DOC.md

> Linux distribution on existing Android devices;

> b. A working Android environment hosted inside or parallel with the
> Linux environment, with Portage managing it, taking the place of the
> original “System Upgrade” process;

Nice!

> c. A Portage overlay and ​android.eclass available to other general
> purpose Gentoo Linux systems

We can reuse proj/android.git from the beginning.

  https://gitweb.gentoo.org/proj/android.git/

> (though further adaption from projects like Android-x86, but that’s
> out of the scope of this project)

No, I don't think Android-x86 is relevant here.  It a dead-end and we
don't need to mention about it.

> 2. What we can do with this:

> a. Utilize tools that we’re familiar with to do things on the mobile,
> from more general toolchains available directly on the device, to
> various services we may want to run; basically we have a portable
> micro-server

Absolutely!

s/$/./

> b. Smoothen the update process of system without having to load up a
> full system image every time (this is especially important for Lineage
> OS nightly users: you need a full rom package to flash even if only a
> few bits of the system (usually just a few new commits) got changed)

Indeed.  But try to get rid of the long nested paranthesis.

> c. Easier adoption to techniques we’re already familiar with to
> perform tasks

> About me: I’m Pengcheng Xu, a student currently studying in the School
> of Electronics Engineering and Computer Science at Peking University,
> China. Having used Gentoo for over 5 years and still currently using
> Gentoo as my server system (it’s a SPARC I got from Brendan Horan
> <​brendan@horan.hk​>​on the ​gentoo-dev​mailing list) as well as Gentoo
> Prefix on macOS and Ubuntu Linux, I have proposed quite many bugs and
> some patches as well

Put a list of bugs and patches you worked with in an appendix
explicitly.  If you have patches at hand, submit them soon.  If you have
pending fixes for Gentoo/FreeBSD, publish them via the bugzilla.  This
will make you more appealing to this program.


Finally, please check out the student's guide to refine this proposal:

  https://google.github.io/gsocguides/student/

> [1]: ​http://whiteboard.ping.se/Android/Debian
> [2]:
> https://events.static.linuxfound.org/sites/events/files/slides/
> ContainersPresoLCE.pdf

Yours,
Benda

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]