[gentoo-soc] Proposal: kernelconfig

[gentoo-soc] Proposal: kernelconfig

[André Erdmann]

Hi,

I've submitted a proposal for kernelconfig today,
an attempt at creating kernel configurations automatically,
with hardware detection and package management integration,
included in this mail. Feel free to comment ;-)

Kind Regards,
André Erdmann


=== proposal starts here ===



Abstract
========

Automated creation of kernel configuration files
from curated sources, detected hardware, installed packages and user input.



Objective
=========

Kernel configuration can be a tiresome procedure.
One starts with a basic config file,
either generated, downloaded or taken from previous runs,
and has to know - or find out - which options need to be set
for a particular piece of hard- or software.

This project aims at automating kernel configuration in both areas.
For hardware, this involves detecting devices, possibly on a different machine,
and mapping them to kconfig options.
And for software, this means setting a number of options in order to fit
a specific role determined by both installed packages and user preferences.

Targeting at beginners and more experienced users,
it is necessary to leave choices up to users while filling the gaps
via automatisms.
For example, "package qemu with kvm and without networking" is more specific
than "virtualization", but both may or may not result in the same configuration,
depending on architecture/hardware, installed packages and other user choices.

To realize this, a more flexible configuration mini-language will be introduced,
based on and compatible with the existing facilities.

At last, it should be possible to detect whether the resulting configuration
makes sense and whether it is conflict-free.



Ideas
=====

This section gives a more in-depth view of what I have planned.


Basic Design
------------

Users define a configuration basis and an output state, and kernelconfig
computes how to get there.

The configuration basis may stem from a command ("make someconfig"),
a curated source, or any other file.

The output state is what users declare in the settings file or via command
line parameters. It consists of direct instructions where config options
are given by name, e.g. enable CONFIG_E1000E,
and indirect instructions where a symbolic name
such as "package shorewall" needs to be resolved to actual config
options and their acceptable values.

In either case, the instructions are incomplete,
because the explicitly stated config options depend on other options,
which in turn depend on even more options.
Requiring users to write full configuration fragments in the settings file
is highly impractical, so one of kernelconfig's essential tasks is
detecting kconfig symbol dependencies
and resolving the dependency graph (if possible).

Another issue is that human input is prone to errors
like  assigning a string value to a tristate-type kconfig option
or making two different decisions for a kconfig choice,
for example setting both 'cfq' and 'deadline' as the default I/O scheduler.
kernelconfig will detect and handle such errors.


Configuration Basis
-------------------

The base configuration file does not necessarily exist on the machine
running kernelconfig, it has to be obtained somehow.
The "somehow" may be expressed in two ways:

* a script/executable responsible for getting the configuration basis

* a source definition ini-like file that describes how to get the
  configuration basis, but leaves the actual work up to kernelconfig

The preferred variant for curated sources will be the source definition
file, because it allows kernelconfig to verify user input, for example
whether the architecture is supported by a particular curated source
or whether a parameter is known and valid.
A combination of both source definition file and script can be used
for more complex sources.


Macros
------

Macros are instructions that modify the kernel configuration.
Users list them in the "[options]" section of the settings file
and may include other macro files, so-called feature sets.

It is planned to implement a mini-language with the following features:

* additional actions,
  for example "enable-or-module", "append-string", "include feature-set"

* disable macros if certain conditions are not met ("<macro> if <condition(s)>")

* parameters, mostly for controlling how feature sets are imported


Hardware Detection
------------------

Hardware detection can be viewed as a feature set that consists
of conditional macros: enable one or more config options if a certain
hardware identifier is matched::

   enable-or-module I915 if modalias-match pci:<id_A> or pci:<id_B>

The difficulty here is
creating the hardware identifier to kernel config options mapping.
It is currently planned to examine the "modules.alias" file,
which maps sysfs modalias entries to kernel module names.
The module names have to be translated back into config options by partially
parsing the module Makefiles of the kernel sources.

This is a (run)time-expensive task,
"modules.alias" is generated by depmod and requires to build all kernel
modules, ideally configured with "make allmodconfig",
so kernelconfig will offer a precomputed file.


Package Management Integration
------------------------------

Two approaches are being considered:

* a backwards-compatible one that greps through logs and vdb/ebuilds

  This creates a dynamic feature set
  (its content has to be calculated at runtime).

* feature set catalog,
  a directory structure containing feature set files that can be included
  in the settings file (single packages or as a whole)

  Packages may either generate a feature set file from linux-info.eclass's
  CONFIG_CHECK variable, or install their own feature set files,
  which is more reliable and cross-distro friendly.

  This approach involves writing a new eclass,
  responsible for installing feature set files
  and converting CONFIG_CHECK.


Genkernel Integration
---------------------

to be discussed,
but probably a facility in genkernel that allows to use kernelconfig
instead of genkernel's gen_configkernel.sh->config_kernel().



Deliverables
============

kernelconfig will be mostly written in Python, with smaller parts
like hardware identifier collection (scan through sysfs) in Shell.
Documentation will be written in reStructuredText.


Final (August 15)
-----------------

A kernelconfig python script + modules with the following properties:

   * has access to various curated sources

   * processes macros from the settings file as described below (see
`Mid-term`_)

   * has access to package information (`Package Management Integration`_)

   * maps detected hardware to config options

   * can also be called via genkernel


A shell script for detecting hardware,
with the possibility to run it on one machine (without requiring Python)
and use the reported data on another machine.

An ebuild for installing kernelconfig,
though it may also be used standalone.


Documentation:

   * developer documentation for all features

   * end-user documentation: man pages and usage guide


Mid-term (June 20)
------------------

A kernelconfig python script + modules with the following properties:

   * uses a .config file as configuration basis

   * processes macros from the settings file:

     * checks whether kconfig options are known

     * type-checks options/values

     * detects kconfig symbol dependencies + conflicts, and finds a solution

     * feature sets mechanism (loading of macro files)

   * simple unit test cases around kconfig symbol dependency resolution


Documentation: partially,

   * developer documentation mostly complete for "basic design" and
"macro syntax"

   * end-user documentation: usage guide: "basic program usage",
"settings file / macro syntax"



Timeline
========

May 23 - May 29: write initial documentation: "basic design", "macro syntax"

May 30 - Jun 08: implement "basic design" and "macro syntax",
                 but without dependency resolution (yet)

Jun 09 - Jun 19: add kconfig dependency resolution

Jun 20 - Jun 26: Mid-term evaluations / write documentation

Jun 27 - Jul 10: (re-)add configuration sources

Jul 11 - Jul 17: add hardware detection

Jul 18 - Jul 24: add package management integration

Jul 25 - Jul 31: add genkernel integration

Aug 01 - Aug 23: testing, write/improve documentation



Biography / About me
====================

I'm a twenty-four year old student from Stuttgart, Germany.
I've been using Gentoo for quite a while now and participated in GSoC
with Gentoo twice before, in 2012 and 2013, which was centered around
roverlay, an effort on generating an overlay for R packages.
This would be my third and final summer of code.

My most long-term open source contribution is doing reviews for TLP,
a shell-script based power management tool, since 2011.
I occasionally contribute to other projects, too, for example PyUSB.



Extra Information
=================

Use the tools that you will use in your project to make changes to code
-----------------------------------------------------------------------

My most recent activity on bugs.gentoo.org is bug #574782,
https://bugs.gentoo.org/show_bug.cgi?id=574782.

Only slightly related, I'm a bit familiar with genkernel-next,
which is based on genkernel, see genkernel-next #17,
https://github.com/Sabayon/genkernel-next/issues/17.


Participate in our development community
----------------------------------------

You can find a mailing list entry from me at <this mail>.


Contact info
------------

email: dywi@mailerd.de
irc: dywi at irc.freenode.net

phone number: <removed>
home mailing address: <removed>


Working hours
-------------

Mo - Sa, 7 am - 8 pm UTC,
with a few restrictions until Mid July due to classes

I target an average of 35 hours per week.