From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 31640 invoked by uid 1002); 4 Aug 2003 22:18:24 -0000 Mailing-List: contact gentoo-dev-help@gentoo.org; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@gentoo.org Received: (qmail 4748 invoked from network); 4 Aug 2003 22:18:24 -0000 From: Stuart Herbert Reply-To: stuart@gentoo.org To: Max Kalika , Troy Dack , gentoo-dev@gentoo.org Date: Mon, 4 Aug 2003 23:16:15 +0100 User-Agent: KMail/1.5.3 References: <42620000.1060017099@valkyrie.lsit.ucsb.edu> In-Reply-To: <42620000.1060017099@valkyrie.lsit.ucsb.edu> MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg=pgp-sha1; boundary="Boundary-02=_zstL/QSlefpgUoc"; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200308042316.19549.stuart@gentoo.org> Subject: Re: [gentoo-dev] [GLEP] Web Application Installation X-Archives-Salt: 8597c91e-e6d6-40e0-958f-0aa79bac2a1a X-Archives-Hash: 962d89d856d2a49f6f1d590cb149c12f --Boundary-02=_zstL/QSlefpgUoc Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Description: signed data Content-Disposition: inline On Monday 04 August 2003 6:11 pm, Max Kalika wrote: > Good morning! Evening, Max. > See, I don't think that running two (or more) copies of an application is > supporting virtual hosts. Upgrading all the files is a pain. Why not ha= ve > one installation of the core files and multiple installations of the conf= ig > files? This is, of course, very application specific -- most of them will > still need database upgrades, but that has to be done by the sysadmin. Because that won't quite work, unless the app is aware of virtual hosting -= =20 and many (if not most) aren't. Think about it. Here's an example. Imagine hosting (oh, I don't know) www.iammax.com and=20 www.maxisgreat.com on the same physical box. Document roots for each are=20 /var/www//public_html/ for arguments sake. Now imagine running phpBB on both domains, to provide separate forums. As = far=20 as phpBB is concerned, when it accesses, say, login.php on www.iammax.com,= =20 the URL is http://www.iammax.com/phpbb/login.php, which translates through = as=20 /var/www/www.iammax.com/public_html/phpbb/login.php. Similarly, login.php = on=20 www.maxisgreat.com translates through as=20 /var/www/www.maxisgreat.com/public_html/phpbb/login.php Given this scenario, how do you make these two sites share the same phpBB c= ode=20 files? Here's a few possible ways, and the problems that they cause. Chip= =20 in with others - because this is the core problem. a) A bit of mod_alias magic, and you make the /phpbb/ directories aliases f= or=20 /usr/share/webapps/phpBB-/files/. If you do this, though, how do= =20 you get each copy of phpBB to use a separate set of configuration files? =20 What happens when the app needs write-access to the directories? The=20 directories are on /usr, and we're all agreed that /usr should be mountable= =20 read-only. And does every webserver have something like mod_alias in the=20 first place? If I've understood your eclass correctly, this is what it tries to do, yes? b) A bit of .htaccess magic, and you have the directory structure of the=20 webapp on /var, but directives telling PHP where to find the .php files by= =20 setting the include_path. The config file problem is solved, because you c= an=20 drop in local config files, and there are real directories that can be made= =20 writable. Only problem with this approach is that not every type of web=20 server has the equivalent of .htaccess files - and the PHP SAPI for each ty= pe=20 of web server doesn't necessarily support configuration directives in confi= g=20 files either. And that's before we think about Perl, Python and other ways= =20 that webapps can be implemented. I believe that this is Robin's basic idea, if I've understood correctly. I= t's=20 a neat solution, but perhaps not a universal one. c) A bit of find and cp -l, and you've got the directory structure of the=20 webapp on /var, plus links back to the original on /usr. Again, the config= =20 file problem is solved, because local copies can be parachuted in, and agai= n=20 there are real directories that can be made writable. No webserver-specifi= c=20 tricks are required - as far as the web server is concerned, each domain ha= s=20 its own installation of phpBB. d) ?? There must be other solutions to the problem that haven't been discus= sed=20 in this thread. Please - contribute! > How do they handle db updates? How do they handle config file updates? I've never asked, and I don't want to know ;-) Maybe Debian just doesn't=20 change that much from year to year . > > 1) Your eclass doesn't correctly (as I understand correct usage to be!) > > support the apache2 USE flag. Easy enough to do - see my webapp-apache > > eclass for an example. > > Why base it on the flag? If the webserver is installed and is supported, > configure for it. Because that's what USE flags are there for. If the user puts '-apache2' i= n=20 their USE flags, it's the job of the ebuild to respect that. Otherwise, th= e=20 ebuild is broken - and probably in breach of policy too. > > 2) Your eclass doesn't specify the permissions that the source files > > should be installed under. Again, easy enough to fix. > > This is very straightforward to add. Agreed. > > 3) Your eclass doesn't provide support for running multiple copies of t= he > > same app on the same machine. This is a showstopper. > > This whole thing needs a separate discussion. There's no portage support > for dealing with more than one installation of the same version of the sa= me > package. Why a separate discussion? If this GLEP isn't here to address this=20 fundamental issue, then I'd say that the GLEP has the wrong scope. You're right - Portage currently can't do all of this on its own. Perhaps = it=20 never will be able to. As I understand it, that's why Robin's volunteered = to=20 write and maintain additional tools to bridge the gap. > > 4) Your eclass requires the admin to stop and start Apache as part of t= he > > install. This is a showstopper. Not every site will want to stop and > > start their web server just because they've installed a new app. > > Imagine a site hosting hundreds of domains, and having to take them > > *all* offline at the same time just because phpBB's been upgraded (for > > example!). Robin's idea of creating .htaccess files under the document > > root deals with this much more managably - although I think we're gonna > > end up using symlinks, as that'll make it easy to support multiple web > > servers. > > ..htaccess files only work in already web-accessable directories (in your > case, DocumentRoot). If we're putting applications in /usr/share/webapps, > the webserver has no idea to look there. Which is what the Alias directive > does in all the .conf files that are generated. However, Alias is not > allowed in .htaccess AFAIK. See about for why I think .htaccess isn't the way to go anyway ;-) > http://httpd.apache.org/docs-2.0/mod/mod_alias.html#alias > > Besides that, the server doesn't have to be restarted -- just HUPed. =46air enough ;-) > > 6) I *like* the idea that check_php() is in this > > eclass, because that check is specific to mod_php under Apache. I'm > > gonna steal that and add it to my webapp-apache eclass ;-) > > Steal away! =20 Done ;-) > Keep in mind it is not completely accurate if the per-package > USE flags goes into portage (http://bugs.gentoo.org/show_bug.cgi?id=3D136= 16). > Things in PUSE will also have to be taken into account. But this is a > completely different matter and will be easy to integrate. It's a nice start though. Ideally, we need a programatic interface to=20 portage, an API we can use to handle these types of queries. karltk is=20 working on one ;-) > > 7) Your variable names are not generic enough for my liking ;-) > > AWEB_CFG, for example, might be better off being WEBAPP_CFG. > > Hey! I originally had them as WEBAPP_* but was told to change the eclass > to apache-webapp to distinguish from other servers so I wanted the > variables to reflect the name of the eclass. I'm as flexible on this as > playdoh. :-) /me bites back his comment about the person who told you to do that ;-) > > 8) Instead of trying to supply an all-encompasing > > apache-webapp_src_install(), relying as it does on defining global > > variables, I'd have supplied a number of individual functions to do ea= ch > > bit. Say, a webapp-install-appconfig, webapp-install-serverconfig each > > taking parameters (this is off the top of my head here ;-) This is a > > personal preference thing. > > Thats just more to call from the ebuild. My goal was to have very small > ebuilds -- just declare a few variables and inherit the eclass. Hmm ... there's a tradeoff here. Smaller ebuilds and less flexible (and=20 re-usable) eclasses. Or larger ebuilds, but more re-usable eclasses. I=20 guess I prefer the latter. > > 9) If I'm not mistaken, your eclass does nothing to ensure that the > > webapp can find the configuration files you've moved into > > /etc/webapps/$PN/. Personally, I'm coming to the conclusion that > > /etc/webapps/$PN/ isn't a good idea, because again it doesn't support > > the idea of running multiple copies of the same app on the same machin= e. > > I'd say we should be shooting for an easy upgrade path first. =20 Good point ;-) But what's the point of installing and upgrading configurat= ion=20 files that the application is never going to look at? ;-) (Sorry, it's 22:3= 0,=20 and still ridiculously hot here) > If config > files aren't stored in /etc/webapps/${PN}, then we need to have a way of > generating an env.d/${PN} which contains a CONFIG_PROTECT line, otherwise > we're forcing sysadins to reconfigure the application at every upgrade. Yeah, but if each instance of the installed app has its own config files, t= hen=20 what's the relevance of /etc/webapps at all? > And as I mentioned before, there's no way for portage to handle > multiple-copy installs, so I'm not sure the best way to go about achieving > this goal. This is the problem that I think we should be solving - how to support the= =20 installation, configuration, and upgrading of multiple-copy installs. > > 10) I'm coming to the conclusion that 'emerge -u ' shouldn't > > overwrite the older version, but should always install alongside, in a > > different slot, so that sites can easily run different versions of apps > > as required. Perhaps this should be configurable somehow? Your eclass > > doesn't make this possible. > > Why should this be handled any different then the rest of apps handled by > portage? =20 Because most of the rest of the apps handled by portage don't run in a virt= ual=20 domain environment. Webapps really are a different beast. > If a sysadmin doesn't want older versions removed, just add > AUTOCLEAN=3Dno to make.conf. Granted, the current eclass doesn't use the > full ${PF} in the target directory, but that is easily changed. Here's an example. Imagine you're running your own hosting firm, and you h= ave=20 a non-trivial number of customers using the same webapp (say, phpBB for=20 arguments sake). A new version of phpBB comes out. Some customers will as= k=20 for the upgrade, and some explicitly will ask you not to upgrade. So, in=20 this situation, you need to have two copies of phpBB installed on the same= =20 box at the same time. Now let's look at what happens when you run 'emerge -u phpBB', with the=20 appropriate ACCEPT_KEYWORDS of course. Portage goes and installs the new=20 version of phpBB over the top of the old phpBB files. The old version of=20 phpBB gets overwritten, yes? I don't see how AUTOCLEAN will prevent that=20 from happening. The whole point of SLOTing apps (as I understand it) is to allow you to hav= e=20 multiple versions installed alongside each other. This is the mechanism th= at=20 Portage offers us. =20 > Ok. Hopefully we can get this thing hashed out and out the door so things > can start getting into shape soon. Hell yes. > > 1) Apache1/Apache2 conundrum > > > > My eclass uses the detection technique adopted for mod_php, and no-one > > has complained about that. If this eclass is invalid, then so's the > > ebuild for mod_php. And I don't think it is. > > Ok, this one is a keeper. Kudos to Robin - it's his algorithm. > > 2) Support for multiple DocumentRoot configurations, and also > > 3) Binary packages installing on machines with different DocumentRoot > > values > > > > Until the GLEP is firmed up and approved, we don't have an agreed > > solution to implement. > > Ah! Ok, so before we have this issues ironed out, the current status-quo > will just be maintained. Fair enough. Agreed. > > Please excuse me, but I don't want to put support for existing ebuilds = on > > hold while we debate the GLEP. I believe that we *have* to continue > > support until we're ready and able to switch. Stopping maintenance > > activities is *not* an option. > > Since I'm the laziest person I know, I didn't want to do the work twice. > I'll just hold off on introducing more ebuilds which will have to be > converted later. :-) That's up to you. If you're maintaining webapp ebuilds currently in portag= e,=20 though, I'd urge you not to stop maintaining them just because you're waiti= ng=20 for a design solution via this GLEP. > > 4) Which user/group to use > > > > My class uses Robin's suggestion, and assumes that Apache is running wi= th > > the default settings of apache.apache. > > Right, but as we already agreed, not all apps need to have their files > owned by apache:apache. This should be configurable in the eclass. > Correct? Do we need a new user/group to own most of the files? And then we just mak= e=20 the files that need write-access owned by the webserver? > > 5) DocumentRoot pointing to a read-only mount > > > > As far as I'm concerned, that's like trying to do 'make bzlilo' with > > /boot not mounted, or run an 'emerge' with /usr mounted read-only. It= 's > > the sysadmin's job to make sure that any necessary filesystems are > > mounted read/write before an installation is attempted. This is not a > > problem unique to web applications. > > The issue is not whether DocumentRoot is a read-only mount during the > install, but during the day-to-day operations. Running 'emerge' with /usr > mounted read-only is not the same as having apache running a webapp that > needs to write to /usr -- one is done seldomly, the other is all the time. In his email in that thread you pointed me at, Robin was explicitly talking= =20 about /var being a read-only NFS mount at install time. > > 6) "it's weak" > > > > That's not an argument, it's an opinion ;-) Anyway, I've taken 5-10 > > lines of broken and incorrectly duplicated code from a number of > > ebuilds, and moved them all into one place where they can be re-used a= nd > > maintained for now. Reduced defects is a strong argument, not a weak > > one. > > I never made the "weak" argument=20 Again, this is in response to the comments in the archived thread - not thi= s=20 one ;-) You didn't make the comment - someone else did. > -- I don't believe in it myself. Things > need to be justified a bit better than that, so I'm completely with you on > this one. :) > Ok, agreed. Troy, is there any chance for another draft with some of the= se > things incorporated? (Thanks a bezillion, btw, for putting up with me) := =2D) Who's "putting up" with you? I don't feel like I am! I'm just grateful th= at=20 we're both interested in finding a solution to this problem. Now, if just = a=20 few more people would chip in and help these discussions ... ;-) > And a successful one at that from what I saw in my emerge --sync this > morning. :-) Thanks! By the time I got Saturday evening, it was all but over. Didn't see a sing= le=20 person in #gentoo-bugs who I could help :( > If you say it is very easy to do, I'm on board. =20 If it's not easy to do, then we'll scrap it and come up with something bett= er. > I can't personally speak > for other webservers, so I'll leave the decision of whether/how to support > others to those with the experience. =20 =46air enough. > So in any case, we have to pull the > apache-specific things out into a separate framework. Therefore things > like DocumentRoot can't even be considered. So a central location for > webapps must be once again taken into account. =20 Yep. How does this sound as the design of the central layout? Let's agree= a=20 design, so that it can be added to the GLEP. * /usr/webapps/ as the main directory. =20 * /usr/webapps//public_html/ for files served by the web server * /usr/webapps//cgi-bin/ for CGI-BIN files * /etc/webapps// to hold the box-default config files * is ${PN} for non-slotted packages * is ${P} for slotted packages > ok, I didn't realize that it wasn't a permanent solution. All is ok. Neat. > >> How is mod_php related to the way applications are installed? > > > > Erm, how about the whole 'do I use Apache 1 or Apache 2' conumdrum? See > > the mod_php ebuild for details. > > Whichever is installed gets configured. Either or both get touched, > depending on what is detected. It is up to the sysadmin to start one or > the either automatically with rc-update. I don't see a problem if both a= re > configured (if detected), but only one is running. Yeah, but as discussed earlier:=20 a) you need a standard way of detecting which one is installed, and=20 b) you need to honour the 'apache2' USE flag Take care, Stu =2D-=20 Stuart Herbert stuart@gentoo.o= rg Gentoo Developer http://www.gentoo.or= g/ Beta packages for download http://dev.gentoo.org/~stuart/package= s/ GnuGP key id# F9AFC57C available from http://pgp.mit.edu Key fingerprint =3D 31FB 50D4 1F88 E227 F319 C549 0C2F 80BA F9AF C57C =2D- --Boundary-02=_zstL/QSlefpgUoc Content-Type: application/pgp-signature Content-Description: signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQA/LtszDC+AuvmvxXwRAjXiAJ0QikCX8FqvAeaw3oWqjhUH7BQWLQCgqtTS OG3w5DWuKw9GisauudX5fy0= =duhW -----END PGP SIGNATURE----- --Boundary-02=_zstL/QSlefpgUoc--