Re: [gentoo-dev] [GLEP] Web Application Installation

[Max Kalika <max@gentoo.org>]

Quoting Stuart Herbert <stuart@gentoo.org>:

> Here's an example.  Imagine hosting (oh, I don't know) www.iammax.com and 
> www.maxisgreat.com on the same physical box.  Document roots for each are 
> /var/www/<host>/public_html/ for arguments sake.

Love those names.  I wonder if they're taken? ;-)

> a) A bit of mod_alias magic, and you make the /phpbb/ directories aliases
> for  /usr/share/webapps/phpBB-<version>/files/.  If you do this, though,
> how do  you get each copy of phpBB to use a separate set of configuration
> files?   What happens when the app needs write-access to the directories?
> The  directories are on /usr, and we're all agreed that /usr should be
> mountable  read-only.  And does every webserver have something like
> mod_alias in the  first place?

If not mod_alias, then symlink to the DocRoot, but then we're back to not
knowing where the user configured his/her/virtualhosts's DocRoot.

> If I've understood your eclass correctly, this is what it tries to do,
> yes?

Yes.

> b) A bit of .htaccess magic, and you have the directory structure of the 
> webapp on /var, but directives telling PHP where to find the .php files
> by  setting the include_path.  The config file problem is solved, because
> you can  drop in local config files, and there are real directories that
> can be made  writable.  Only problem with this approach is that not every
> type of web  server has the equivalent of .htaccess files - and the PHP
> SAPI for each type  of web server doesn't necessarily support
> configuration directives in config  files either.  And that's before we
> think about Perl, Python and other ways  that webapps can be implemented.

Local config files?  Where would these be stored?  How are these handled
for virtual hosts?

> c) A bit of find and cp -l, and you've got the directory structure of the 
> webapp on /var, plus links back to the original on /usr.  Again, the
> config  file problem is solved, because local copies can be parachuted
> in, and again  there are real directories that can be made writable.  No
> webserver-specific  tricks are required - as far as the web server is
> concerned, each domain has  its own installation of phpBB.

Ok, I'll pool some replies below...

>> Why base it on the flag?  If the webserver is installed and is supported,
>> configure for it.
> 
> Because that's what USE flags are there for.  If the user puts '-apache2'
> in  their USE flags, it's the job of the ebuild to respect that.
> Otherwise, the  ebuild is broken - and probably in breach of policy too.

Ok, this is debatable, but I'm willing to change the eclass to take
-apache2 into effect.

>> This whole thing needs a separate discussion.  There's no portage support
>> for dealing with more than one installation of the same version of the
>> same package.
> 
> Why a separate discussion?  If this GLEP isn't here to address this 
> fundamental issue, then I'd say that the GLEP has the wrong scope.
> 
> You're right - Portage currently can't do all of this on its own.
> Perhaps it  never will be able to.  As I understand it, that's why
> Robin's volunteered to  write and maintain additional tools to bridge the
> gap.

Keep reading, going to answer this below. :-)

>> Keep in mind it is not completely accurate if the per-package
>> USE flags goes into portage
>> (http://bugs.gentoo.org/show_bug.cgi?id=13616). Things in PUSE will also
>> have to be taken into account.  But this is a completely different
>> matter and will be easy to integrate.
> 
> It's a nice start though.  Ideally, we need a programatic interface to 
> portage, an API we can use to handle these types of queries.  karltk is 
> working on one ;-)

Cool!

>> Thats just more to call from the ebuild.  My goal was to have very small
>> ebuilds -- just declare a few variables and inherit the eclass.
> 
> Hmm ... there's a tradeoff here.  Smaller ebuilds and less flexible (and 
> re-usable) eclasses.  Or larger ebuilds, but more re-usable eclasses.  I 
> guess I prefer the latter.

We can still achieve flexibility along with smaller ebuilds.  If we break
down the eclass install function into smaller functions and optionally call
them from install, we've got ourself a nice tidy system.

>> If config
>> files aren't stored in /etc/webapps/${PN}, then we need to have a way of
>> generating an env.d/${PN} which contains a CONFIG_PROTECT line, otherwise
>> we're forcing sysadins to reconfigure the application at every upgrade.
> 
> Yeah, but if each instance of the installed app has its own config files,
> then  what's the relevance of /etc/webapps at all?
> 
>> And as I mentioned before, there's no way for portage to handle
>> multiple-copy installs, so I'm not sure the best way to go about
>> achieving this goal.
> 
> This is the problem that I think we should be solving - how to support
> the  installation, configuration, and upgrading of multiple-copy installs.

There's nothing I can think of that will make this work, but see below for
some arguments...

>> Why should this be handled any different then the rest of apps handled by
>> portage?  
> 
> Because most of the rest of the apps handled by portage don't run in a
> virtual  domain environment.  Webapps really are a different beast.

They certainly are. :-)

> Here's an example.  Imagine you're running your own hosting firm, and you
> have  a non-trivial number of customers using the same webapp (say, phpBB
> for  arguments sake).  A new version of phpBB comes out.  Some customers
> will ask  for the upgrade, and some explicitly will ask you not to
> upgrade.  So, in  this situation, you need to have two copies of phpBB
> installed on the same  box at the same time.
> 
> Now let's look at what happens when you run 'emerge -u phpBB', with the 
> appropriate ACCEPT_KEYWORDS of course.  Portage goes and installs the new 
> version of phpBB over the top of the old phpBB files.  The old version of 
> phpBB gets overwritten, yes?  I don't see how AUTOCLEAN will prevent that 
> from happening.
> 
> The whole point of SLOTing apps (as I understand it) is to allow you to
> have  multiple versions installed alongside each other.  This is the
> mechanism that  Portage offers us.  

Here's the problem.  First of all, SLOTing won't help that because it
assumes that the versions of the packages *and* the files you are
installing are _different_. I.e. db4 and db3 are SLOTted: they install
libdb-4.so and libdb-3.so respectively and thus can run simultaneously.
Now, wrt to the whole virtual server thing.  Personally if I was running a
server with, as you say, a non-trivial number of virtual domains who have
different needs, I probably wouldn't use a package management system to
begin with, or truly virtualize everything and use something like
user-mode-linux in which case we're back to single-domain installs.

Going along with your example, how can you do an 'emerge -u phpBB' _for a
particular domain_?  You're telling portage to maintain a package, and
upgrade parts of it randomly.  I simply don't think portage will ever have
any way of handling multiple installs of a package in this manner.  How
about we (Gentoo) just provide single domain/host installs and maybe
SLOTing different versions so they won't be overwritten on upgrades.
Beyhond that, we have to let the sysadmins know to cp -l the installed
files into their vhosts.

AFAIK, no other distro has support for virtual hosting because of this very
issue.

> That's up to you.  If you're maintaining webapp ebuilds currently in
> portage,  though, I'd urge you not to stop maintaining them just because
> you're waiting  for a design solution via this GLEP.

Not in charge of anything else in portage.  Though I'd like to help out
with Horde down the road (seeing as I'm on the horde dev team as well).

>> Right, but as we already agreed, not all apps need to have their files
>> owned by apache:apache.  This should be configurable in the eclass.
>> Correct?
> 
> Do we need a new user/group to own most of the files?  And then we just
> make  the files that need write-access owned by the webserver?

The most robust solution would be to create a new group (web?) and make all
the webserver users be a part of that group and make these directory
group-owned by said group.  (Similar to the "mail" group for many
mail-related services)

> In his email in that thread you pointed me at, Robin was explicitly
> talking  about /var being a read-only NFS mount at install time.

Ok, install-time aside, the installed, writeable directories cannot exist
on a readonly mount for day-to-day operations.  Which takes me back to
wanting to support a single instance install of an app that an admin has to
copy in place if the default location is unacceptable.

>> So in any case, we have to pull the
>> apache-specific things out into a separate framework.  Therefore things
>> like DocumentRoot can't even be considered.  So a central location for
>> webapps must be once again taken into account.  
> 
> Yep.  How does this sound as the design of the central layout?  Let's
> agree a  design, so that it can be added to the GLEP.
> 
> * /usr/webapps/<app-name> as the main directory.  
> * /usr/webapps/<app-name>/public_html/ for files served by the web server
> * /usr/webapps/<app-name>/cgi-bin/ for CGI-BIN files
> * /etc/webapps/<app-name>/ to hold the box-default config files
> * <app-name> is ${PN} for non-slotted packages
> * <app-name> is ${P} for slotted packages

I'd say <app-name> should just be ${PF} for both and not worry about what
is SLOTed and what is not -- it works itself out.  Otherwise, this seems ok
to me and is easy to implement in the eclass.  The only thing I would
change is the names public_html and cgi-bin.  There may be no html and not
binary files in them (respectively).  How's about 'public' and 'cgi'?  (I
know, I know, nitpicking).

> a) you need a standard way of detecting which one is installed, and 

Is the approach currently in the eclass unacceptable? (Check for the
existance of the directory/configfile)

> b) you need to honour the 'apache2' USE flag

Yup, see above...no higher...no, higher than that.  Ok there, stop!

--mk

--
gentoo-dev@gentoo.org mailing list