public inbox for gentoo-soc@lists.gentoo.org
 help / color / mirror / Atom feed
* [gentoo-soc] Weekly Report: Portage Powered Android
@ 2020-07-13 23:09 Gunwant Jain
  2020-07-14  0:17 ` Benda Xu
  2020-07-14  4:05 ` Pengcheng Xu
  0 siblings, 2 replies; 49+ messages in thread
From: Gunwant Jain @ 2020-07-13 23:09 UTC (permalink / raw
  To: gentoo-soc

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

Hello Everyone,

Like I mentioned in my previous off-timed weekly report, I had been
working on studying Android's Build Process this week.

I have published a blog [1] regarding the same (most of it is just
confirmed workflows). I also laid a plan on how to approach building
Android on aarch64 host.

The next two weeks will be focused on that. I will be setting up an ARM
based VPS. Once we are through with natively compiling Android, I plan
to separate LLVM/Clang toolchain along with Bionic from the AOSP
codebase, Bionic here being the tricky part.

Regards,
Gunwant

[1] https://wantguns.gitlab.io/blog/android/build_android/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread
* [gentoo-soc] Weekly Report: Portage Powered Android
@ 2020-08-17 18:05 Gunwant Jain
  2020-08-17 23:15 ` Benda Xu
  0 siblings, 1 reply; 49+ messages in thread
From: Gunwant Jain @ 2020-08-17 18:05 UTC (permalink / raw
  To: gentoo-soc

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

Hello everyone,

As I planned in my previous report, this week I was working on the
llvm_android build script. I can finally report that it works and we can
build AOSP's llvm for AArch64 Host now[1]. 

I wasn't able to handle the prebuilt GCC dependency libraries the
aforementioned LLVM toolchain requires. This would take longer time to
deal with because AOSP's build scripts[2] are all over the place. 
And so I prioritize building GCC below the main goal of compiling 
Android on AArch64 for the time being.

Coming to which, I have started testing building Lineage on AArch64
today. I believe this should take another week of efforts, if not more.

Regards,
Gunwant

[1] https://github.com/WantGuns/toolchain_llvm_android
[2] https://android.googlesource.com/toolchain/gcc/+/refs/heads/master

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread
* [gentoo-soc] Weekly Report: Portage Powered Android
@ 2020-08-12  1:41 Gunwant Jain
  2020-08-12  7:34 ` EBo
  0 siblings, 1 reply; 49+ messages in thread
From: Gunwant Jain @ 2020-08-12  1:41 UTC (permalink / raw
  To: gentoo-soc

[-- Attachment #1: Type: text/plain, Size: 1409 bytes --]

Hi all,

This and the previous week, I have been working on building AOSP's LLVM
toolchain for aarch64. tldr is that I have succeeded in doing so, but I
have yet to commit my progress at the (yet to be) SharkBait manifest[1].

Really there are just a couple of things to work on before we are ready
to build Android on arm64:

1. The build process is a bit immature. I will commit the polished
changes in a day or two at the LLVM source[2].

2. While trying to minimize dependencies that AOSP bundles for building
LLVM, I noticed that building LLVM requires using libraries from a
prebuilt gcc[3]. This is unwanted because we aim not to use any of the
prebuilt dependencies that AOSP ships. 
Therefore this week I am working upon building AOSP's gcc[4] for
aarch64.

Once the GCC dependency is sorted, I will start testing building Lineage
on aarch64. I have set a personal goal to atleast start these tests by
this week, since we do not have much time left.

Finally, I would like to thank my mentor, Heroxbd for providing the
arm64 server, due to which I could pick up the pace and build AOSP's
LLVM on aarch64.


Regards,
Gunwant

[1] https://github.com/WantGuns/manifest/tree/lineage-17.1
[2] https://github.com/WantGuns/toolchain_llvm_android
[3] https://github.com/WantGuns/manifest/blob/llvm-r353983/default.xml#L39
[4] https://android.googlesource.com/toolchain/gcc

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 484 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread
* [gentoo-soc] Weekly Report: Portage Powered Android
@ 2020-07-30 15:00 Gunwant Jain
  0 siblings, 0 replies; 49+ messages in thread
From: Gunwant Jain @ 2020-07-30 15:00 UTC (permalink / raw
  To: gentoo-soc

[-- Attachment #1: Type: text/plain, Size: 1667 bytes --]

Hello everyone,

I am sending my weekly report this late because I figured that I had
already discussed my findings this week on the mailing list. But my
mentor asked me to write it nevertheless.

My task was to build AOSP's LLVM toolchain, but for AArch64 host. 
Patching the build scripts for that was the easy part. But running the
script over and over became tedious at some point. Many times, my sshfs
caused problems with the CMakeCache, essentially forcing me to clean
build. Other times, the OOM killer got me. 

Nevertheless I can report that llvm's stage1 builds for the AArch64 host
and CMake is accepting the stage1 compilers to build the final llvm
stage2. I have yet to fully build llvm stage2, but I am sure that my
Mentor's kind offering of an arm64 server would check off that task.

I have already explained llvm stage1 and stage2, in some depth, in the 
previous weekly report's reply chain. I'll put a brief explanation here
anyways.
Bascially, AOSP packages its own LLVM distribution. The idiomatic way to
build LLVM in this case would be to build a "stage 1" compiler with your
host toolchain and then build the "stage 2" compiler using the "stage 1"
compiler. In AOSP's case, that stage2 compiler is used :   
    - in the AOSP tree as the prebuilt toolchain, used to build AOSP.
    - as the stage1 compiler to build a version bumped LLVM toolchain
      (later used as the prebuilt toolchain in the AOSP tree).

Therefore my next task would be to use that llvm toolchain (built for
AArch64 host) in the AOSP tree and build Android on arm64 unlike the
conventional cross-compiling method.

Regards,
Gunwant


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread
* [gentoo-soc] Weekly Report: Portage Powered Android
@ 2020-07-20  7:40 Gunwant Jain
  2020-07-24 14:08 ` Benda Xu
  0 siblings, 1 reply; 49+ messages in thread
From: Gunwant Jain @ 2020-07-20  7:40 UTC (permalink / raw
  To: gentoo-soc

[-- Attachment #1: Type: text/plain, Size: 1571 bytes --]

Hello Everyone,
This week, I started managing the structure for the modified LineageOS
fork we would keep. Namely the modified `repo-manifest` [1] and
`llvm_android` [2], for now. No patches have been applied to them yet
for the reasons discussed below.

My plan for this week was to start with patching the `llvm_android`
build scripts. Android uses a prebuilt Clang (bundled with the repo)
among other stuff to build the toolchain. So I had to emerge Clang on my
phone.
Emerging Clang turned out not to be a breeze when I have a phone with 3
gigs of RAM. So for about the longest time, I was busy trying to make up
for it. I tried setting up distcc, increased swap/zram, but still the
OOM killer got me.
Later last night(/day), I came across the tips for building clang/gcc in
an embedded Android environment at [3]. So I followed that and as of
now, me writing the mail, OOM has not got me, Clang is emerging.

While I considering the patching part easy, I guess the major bottleneck
in the process of building Android natively would be the compilation.
Essentially I will have to trade time for memory. I will also give
another shot to distcc while I am at this.

Regardless of problems I faced, I made "TODO" comments in the source and
wrote notes about where to patch the build scripts locally. I avidly
wait for Clang to compile and then start testing the patched build
script.

Regards,
Gunwant

[1] https://github.com/WantGuns/manifest
[2] https://github.com/WantGuns/toolchain_llvm_android
[3] https://wiki.gentoo.org/wiki/Project:Android/tarball#Out_of_Memory

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread
* [gentoo-soc] Weekly Report: Portage Powered Android
@ 2020-07-08 11:39 Gunwant Jain
  2020-07-08 13:53 ` Pengcheng Xu
  2020-07-09  1:04 ` Benda Xu
  0 siblings, 2 replies; 49+ messages in thread
From: Gunwant Jain @ 2020-07-08 11:39 UTC (permalink / raw
  To: gentoo-soc; +Cc: heroxbd

[-- Attachment #1: Type: text/plain, Size: 1391 bytes --]

Hey Everyone !

I was about to give a coherent report for what I have been doing for 2
weeks next week, but my mentor suggested to explain my findings briefly.

I found out that AOSP has actually been phasing out of using GCC in
favour to Clang/LLVM based toolchains. And so, from the last time I 
gave in my weekly report, I have been checking up on llvm_android [1],
the source of the toolchain which AOSP uses to build itself.

I built the toolchain for x86 host using the original scripts AOSP
provides and found out that they in terms of Bionic (Android's libc),
they only copy Bionic's headers, compiles Clang for different targets
using a prebuilt toolchain bundled with the entire package. 

Later, when we build Android, the prebuilt toolchain compiles Bionic and
uses it as intermediatories to compile the other bulk of Android
(frameworks, etc.).

My main objective I think would be to introduce a aarch64 host config 
for llvm_android, and integrate the toolchain in the AOSP build process.
Take this with a grain of salt, as I believe I should put more research
into this first, which is why I was taking another week.

I would probably write a detailed blog documenting AOSP's build process
by the end of this working week, covering everything we need to proceed.


Regards,
Gunwant

[1] https://android.googlesource.com/toolchain/llvm_android/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread
* [gentoo-soc] Weekly Report: Portage Powered Android
@ 2020-06-29 20:12 Gunwant Jain
  2020-06-29 21:27 ` EBo
                   ` (2 more replies)
  0 siblings, 3 replies; 49+ messages in thread
From: Gunwant Jain @ 2020-06-29 20:12 UTC (permalink / raw
  To: gentoo-soc

[-- Attachment #1: Type: text/plain, Size: 959 bytes --]

Hello Everyone,

This week, I had relatively less work done.
I spent most of my time perusing the LFS guide and osdev.org learning
about building toolchains. Then I went onto building the
aarch64-linux-android toolchain as described in [1]. I updated the
kernel sources for my phone [2]. And in general tidied up my kernel
repository. 
But while building the toolchain, I was not able to compile `gcc` as 
I was riddled with errors along the way. Most of them are related to the
newer `gcc` (version 9.x) finding incompatibility in the sources of the
`gcc` I am trying to build (version 4.9).
I will continue to build the toolchain and hopefully move onto
separating `Bionic` from the other side of Android mid-week.

Thanks,
Gunwant

[1] https://wiki.gentoo.org/wiki/Android/SharkBait/Building_a_toolchain_for_aarch64-linux-android
[2] https://github.com/WantGuns/overlay/blob/master/sys-kernel/linux-headers/linux-headers-4.4.222.ebuild

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread
* [gentoo-soc] Weekly Report: Portage Powered Android
@ 2020-06-22 19:15 Gunwant Jain
  2020-06-23  4:45 ` Pengcheng Xu
  2020-07-01  7:14 ` Benda Xu
  0 siblings, 2 replies; 49+ messages in thread
From: Gunwant Jain @ 2020-06-22 19:15 UTC (permalink / raw
  To: gentoo-soc; +Cc: heroxbd

[-- Attachment #1: Type: text/plain, Size: 1751 bytes --]

Hello Everyone,

I was supposed to integrate `bootstrap-init` and `sar-preinit` into
SharkBait this week. I couldn't quite do that.
`bootstrap-init` was initially cross-compiled for aarch64 using NDK. So
the first job was to get rid of NDK and use a different "build-system". 
I opted for CMake and ported the project to use it [1]. But the binary
compiled by CMake on my phone (arm64 Gentoo), would not boot Android.
The build rules can be inspected on [1].

I believe this is because of the fact that I am not using a Bionic based
toolchain. So, I started working on building such a toolchain mid-week.
And this is where my current task starts coinciding with my next 
scheduled task of providing a aarch64-linux-android toolchain where we 
build Bionic as well. The crux of this task was already performed by 
KireinaHoro in his GSoC [2]. Since my next scheduled task is required to 
be finished for my current task, I propose to merge `bootstrap-init` 
along with `sar-preinit` together later in the SharkBait workflow. 

And to cover the installing process for users right now, I will write a
blog and post it on [3] within 24 hours of writing this mail. The blog
will provide a guide on assembling the components required for
System-As-Root based Android devices and installing SharkBaitOS. Keep in 
mind that this process will indeed be simplified for the user later. 
And I propose to edit SharkBait's official wiki later in my GSoC tenure
explaining the simplified procedure to end-users and porters.

Thanks,
Gunwant

[1] https://gitlab.com/WantGuns/bootstrap-init/-/tree/cmake
[2] https://wiki.gentoo.org/wiki/Android/SharkBait/Building_a_toolchain_for_aarch64-linux-android
[3] https://wantguns.gitlab.io/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread
* [gentoo-soc] Weekly Report: Portage Powered Android
@ 2020-06-15  8:03 Gunwant Jain
  2020-07-01  7:10 ` Benda Xu
  0 siblings, 1 reply; 49+ messages in thread
From: Gunwant Jain @ 2020-06-15  8:03 UTC (permalink / raw
  To: gentoo-soc

[-- Attachment #1: Type: text/plain, Size: 744 bytes --]

Hello everyone,

Most of my work in this week revolved around bringing a `bootstrap-init`
[1] for, as the name suggests, sets up the environment for 
System-as-Root Android inside the LXC container and finally exec's the
Android init.
What this essentially means is that now we can use SharkBait on SAR
devices (Android 9+). 
The process of setting SharkBait up for such devices is a bit different
and I am currently in discussion with my mentors for possible design
flows in order to make it easier for Porters and Users.

I will be integrating all my work in SharkBait and quickly transition
into working on Android's build systems mid-week, in this week.

Thanks,
Gunwant


[1] https://gitlab.com/WantGuns/bootstrap-init

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread
* [gentoo-soc] Weekly Report: Portage Powered Android
@ 2020-06-09 11:39 Gunwant Jain
  2020-07-01  7:08 ` Benda Xu
  0 siblings, 1 reply; 49+ messages in thread
From: Gunwant Jain @ 2020-06-09 11:39 UTC (permalink / raw
  To: gentoo-soc

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]

Hello, 
This week, I had quite some progress on my GSoC project.
The task scheduled was to provide a `preinit` which could boot Gentoo in System-As-Root enabled phones.
I completed that for good using the `magiskboot` tool, by hexpatching the kernel for such devices [1].
I faced some issues after patching the kernel as well, resolving them without the access of a serial console was inconvenient but achieved nevertheless. I have written extensive blogs, explaining the entire journey of doing it [2]. Overall, the process of using preinit is now streamlined and simple.

Apart from this, I have written a blog on the boot-mechanisms different versions of Android have. My blogs would be the primary source of watching my activity on GSoC, apart from the irc of-course. 

The next task scheduled for week 1-2 was to boot Android in a container in the Gentoo land. I am pleased to inform that we have achieved that goal as well. It is in barebones presently, but everything (RIL, camera, etc.) works inside the Android container. 
For the upcoming week, I will be writing a new `init` for bringing support to almost every boot-mechanism Google has experimented with. This would also help the project in being very scalable because we would have a single point of maintainence.

Regards,
Gunwant Jain


[1] https://gitlab.com/WantGuns/sar-preinit 
[2] https://wantguns.gitlab.io

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread
end of thread, other threads:[~2020-08-20  6:21 UTC | newest]

Thread overview: 49+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-07-13 23:09 [gentoo-soc] Weekly Report: Portage Powered Android Gunwant Jain
2020-07-14  0:17 ` Benda Xu
2020-07-14  5:01   ` Gunwant Jain
2020-07-15  1:04     ` Benda Xu
2020-07-15  1:09       ` UnderSampled
2020-07-15  4:55         ` Gunwant Jain
2020-07-15  4:45       ` Pengcheng Xu
2020-07-15  4:58         ` Gunwant Jain
2020-07-15  4:52       ` Gunwant Jain
2020-07-14  4:05 ` Pengcheng Xu
2020-07-14  5:05   ` Gunwant Jain
  -- strict thread matches above, loose matches on Subject: below --
2020-08-17 18:05 Gunwant Jain
2020-08-17 23:15 ` Benda Xu
2020-08-20  6:21   ` Gunwant Jain
2020-08-12  1:41 Gunwant Jain
2020-08-12  7:34 ` EBo
2020-08-14  7:26   ` Gunwant Jain
2020-07-30 15:00 Gunwant Jain
2020-07-20  7:40 Gunwant Jain
2020-07-24 14:08 ` Benda Xu
2020-07-24 15:02   ` Gunwant Jain
2020-07-24 17:07     ` UnderSampled
2020-07-24 18:12       ` Gunwant Jain
2020-07-25  3:00     ` Benda Xu
2020-07-25  8:02       ` Gunwant Jain
2020-07-25  8:21         ` Gunwant Jain
2020-07-08 11:39 Gunwant Jain
2020-07-08 13:53 ` Pengcheng Xu
2020-07-09  1:04 ` Benda Xu
2020-06-29 20:12 Gunwant Jain
2020-06-29 21:27 ` EBo
2020-06-30 21:42   ` Gunwant Jain
2020-07-01  2:37     ` EBo
2020-07-01  3:15       ` Gunwant Jain
2020-07-01 12:40     ` Lucas Ramage
2020-07-01  7:17 ` Benda Xu
2020-07-01 16:12   ` Gunwant Jain
2020-07-01 14:07 ` Pengcheng Xu
2020-07-01 16:53   ` Gunwant Jain
2020-07-02  3:50     ` UnderSampled
2020-07-02 12:14       ` Gunwant Jain
2020-06-22 19:15 Gunwant Jain
2020-06-23  4:45 ` Pengcheng Xu
2020-06-23 11:44   ` Gunwant Jain
2020-07-01  7:14 ` Benda Xu
2020-06-15  8:03 Gunwant Jain
2020-07-01  7:10 ` Benda Xu
2020-06-09 11:39 Gunwant Jain
2020-07-01  7:08 ` Benda Xu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox