[gentoo-project] [RFC] Reducing use of SMW in favor of something sane

[gentoo-project] [RFC] Reducing use of SMW in favor of something sane

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 8493 bytes --]

Hi,

TL;DR: MediaWiki sucks, SMW even more; let's start moving important
stuff out of it and into something sane. What replacements do you
suggest?


There was a long period during which we've attempted to move a lot of
Gentoo out of fancy XML files and into Wiki. This happened for project
docs, this happened for entire project structure and more more stuff.
On the other hand devmanual is still using fancy XML files in its own
git repository, and the main g.o site is using Jekyll.

Now that the main SMW proponent is MIA, I think it's time we look back
and think whether what has happened over time was good. I think it
wasn't, and I have a number of arguments to support that:

1. SMW lacks proper review platform, and the workflow is not suitable
for project docs.

2. It has a pretty high entry barrier, and is poorly documented.

3. Preview is half-working, and insufficient for more complex work.

4. It lacks proper offline editing capabilities.

5. The watching (mail notifications) are awfully cumbersome.

6. It is slow.

Now I'm going to expand on those points.


1: SMW lacks proper review platform, and the workflow sucks
===========================================================

MediaWiki is strongly bound to a specific workflow: users edit articles
as they see fit, and if they write something bad, someone else reverts
or fixes it. Such a workflow can work well for generic purpose articles
on sites with millions of users such as Wikipedia. However, it isn't
very fit for project documentation or policies where we really expect
every change to be made or at least approved by developers. As a result,
we restrict editing project pages to developers only.

Sadly, with no review platform this makes submitting updates for users
a PITA. All they can do is use project talk (which is usually ignored)
or submit bugs.

This is also a problem for developers -- when I want to work on doc
update that's supposed to be reviewed by others before being applied,
I end up having to copy text over to another page, work there, send it
for reviews (along with copy of the contents to facilitate context
replies) and afterwards merge it back to the original page. Unless I
want to spend a lot of time redoing all the changes separately,
the history ends up being split between the main doc and my working
copy. This also makes editing GLEPs a PITA.


2: It has a pretty high entry barrier, and is poorly documented
===============================================================

MediaWiki might have been cool when there was no alternative. Right now,
I dare say the only markup worse than MW is Textile. We have many decent
alternatives that are more predictable and easier to type -- RST,
Markdown, asciidoc. Users usually meet them anyway for various purposes,
so there's really no point in requiring them to remember MW.

What's even worse is that the basic markup is half-baked anyway. You
can't even make a proper table without having HTML attributes mixed in,
and you frequently end up using inline HTML (sometimes because you just
can't find a way to make MW syntax work).

Everything ends up being a horrible soup, in which you can't even
predict whether X will work without actually testing it. Sometimes you
need to escape stuff (also in unpredictable contexts), and there's not
even a single way to do that for all cases.

And that's just the basic use. If you start doing templates, things go
more complex. If you want to do the fancy SMW stuff, you're entering
the territory of true guesswork. The documentation is of moderate
quality, and is more focused on solving specific problems (i.e. how to
do X with Y) than on fully documenting how things work.


3: Preview is half-working, and insufficient for more complex work
==================================================================

All the above considered, preview becomes a very important feature.
Sadly, it's also mediocre. For a start, it renders some things
differently than the actual page (dunno if maffblaster already looked
at the problem). It's not nice when you spend a lot of time figuring out
why definition lists don't work, just to figure out it's a bug in page
preview.

Then, you can only preview the page you're editing. If you're working
on templates, SMW data and other hard magic, you're on your own.
Basically, you save the updates, wait a few seconds for other pages to
randomly refresh and see if your change hasn't broken them. Then you end
using 'trial and error' until things start working. All that on
the production instance where it could bite people using the wiki.

Not to mention the ability to test or at least reasonably predict that
some fancy SMW changes you won't break the existing data.


4: It lacks proper offline editing capabilities
===============================================

I might be wrong here but I'm not aware of any fully working, SMW-
friendly tools to work on articles offline. There's some git magic to
pull from MediaWiki but my short testing has shown it doesn't work very
well and can't handle the lot of our namespaces.

If you want on docs, you usually need to have your browser open
and a reasonably stable connection. Any offline work usually requires
extra effort, and a lot of handiwork.


5: The watching (mail notifications) are awfully cumbersome
===========================================================

So you can watch articles on MW and get mail notifications when someone
edits them. Cool. But someone thought it would be even cooler if you got
only one notifications per page until you revisited. And there's no
switch to turn this horrible misfeature off!

I get notification that someone did a trivial change on my page.
The description tells me what it was, and I know I don't have to visit
the article to check it. But I need to visit it anyway because otherwise
I wouldn't get notifications for more important changes! In fact, I need
to look at history immediately because this trivial change could already
have been followed by something important that I wasn't be notified of.

So if I read mail on phone, I need to mark mails to visit wiki articles
later (because someone decided to hide login controls on small screens!)
If I'm doing something else, I need to start browser and visit pages
hoping I won't run out-of-memory in the process.

It's real fun when notifications that are supposed to help you end up
enforcing a lot of effort just to keep them working.



6: It is slow
=============

I don't think this really needs much said. SMW is slow *even* for a PHP
software. Our resources could really be better utilized running
something that would benefit everyone rather than testing our patience.


What to move, and what to?
==========================

SMW is not something we can replace overnight. However, I think we
should slowly start moving important stuff out of it.

The project metadata is the first thing I'd like to have out of wiki
ASAP. Keeping it there is inconsistent, inconvenient and seems like
putting a lot of effort just to prove that SMW could be used for
anything.

We already keep developers in LDAP, and dump them onto main site. I see
no reason why we couldn't keep projects in LDAP or in a flat file
(api.gentoo.org), and include them on the main site rather than wiki. We
can still link to wiki entities for documentation, or maybe even figure
out how to include project members & so on from external source
on the wiki.

The next thing I'd like to move are project docs. We'd really use some
proper doc hosting for that, with a review platform and so on. Patrick
has suggested using flat files in a git repository -- and I think it
really makes much more sense than some fancy wiki software. If we need
more power, we could just use Jekyll as for the main site.

Fun fact is, even GitHub got that right. I think it might be even
possible to have GitHub pull requests with some degree of preview for
that. And if everything's in git, we can easily edit and preview it
locally.

I think it would be also reasonable to seek SMW replacement for regular
user docs. Even if we can't figure out something nice and git-backed,
any replacement would be much better. In fact, even Infra runs its own
wiki on DokuWiki which is much faster.


Your comments? Ideas?


-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 988 bytes --]

Re: [gentoo-project] [RFC] Reducing use of SMW in favor of something sane

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1802 bytes --]

On Sat, Jul 22, 2017 at 01:02:55PM +0200, Michał Górny wrote:
> Hi,
> 
> TL;DR: MediaWiki sucks, SMW even more; let's start moving important
> stuff out of it and into something sane. What replacements do you
> suggest?
> 
> 
> There was a long period during which we've attempted to move a lot of
> Gentoo out of fancy XML files and into Wiki. This happened for project
> docs, this happened for entire project structure and more more stuff.
> On the other hand devmanual is still using fancy XML files in its own
> git repository, and the main g.o site is using Jekyll.
> 
> Now that the main SMW proponent is MIA, I think it's time we look back
> and think whether what has happened over time was good. I think it
> wasn't, and I have a number of arguments to support that:
> 
> 1. SMW lacks proper review platform, and the workflow is not suitable
> for project docs.
> 
> 2. It has a pretty high entry barrier, and is poorly documented.
> 
> 3. Preview is half-working, and insufficient for more complex work.
> 
> 4. It lacks proper offline editing capabilities.
> 
> 5. The watching (mail notifications) are awfully cumbersome.
> 
> 6. It is slow.
> 

7. Related to the high entry barrier, it has an even higher entry
barrier for accessibility. Editing documentation in a browser is unwieldy
at best and impossible at worst.

I looked at the git mediawiki module as an alternative, but it is
extremely unwieldy as well and would put a lot of load on the server.

I, too, realize that it would take a lot of time and effort, but I would
definitely support migrating *all* of our documentation out of mw into some kind
of flat files (rst? markdown? some other format?) which could be
edited/previewed locally and rendered on the web easily.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-project] [RFC] Reducing use of SMW in favor of something sane

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

Am Samstag, 22. Juli 2017, 19:13:48 CEST schrieb William Hubbs:
>
> I looked at the git mediawiki module as an alternative, but it is
> extremely unwieldy as well and would put a lot of load on the server.
> 

The git mediawiki module is a nice toy to play with, but I've never managed to 
clone our whole wiki. 

(No namespace support, broken with page names like "/etc/ntpd.conf", ...)

I'll probably back out our patches from git again at some point, since noone 
is pushing them into git upstream and they are not really worth the work 
without further improvements.

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer (council, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-project] [RFC] Reducing use of SMW in favor of something sane

[Daniel Campbell]

[-- Attachment #1.1: Type: text/plain, Size: 7416 bytes --]

On 07/22/2017 04:02 AM, Michał Górny wrote:
> Hi,
> 
> TL;DR: MediaWiki sucks, SMW even more; let's start moving important
> stuff out of it and into something sane. What replacements do you
> suggest?
> 
> 
> There was a long period during which we've attempted to move a lot of
> Gentoo out of fancy XML files and into Wiki. This happened for project
> docs, this happened for entire project structure and more more stuff.
> On the other hand devmanual is still using fancy XML files in its own
> git repository, and the main g.o site is using Jekyll.
> 
> Now that the main SMW proponent is MIA, I think it's time we look back
> and think whether what has happened over time was good. I think it
> wasn't, and I have a number of arguments to support that:

Who is this proponent? Do we know why they are MIA?

> 
> [snip]> However, it isn't
> very fit for project documentation or policies where we really expect
> every change to be made or at least approved by developers. As a result,
> we restrict editing project pages to developers only.

To be sure I'm understanding correctly, you're talking about pages that
really only devs need to be messing with (like most of the Project:
namespace) and not the entire wiki, right?

I agree the workflow can be a pain. I followed your example and began
using a Drafts sub-namespace in my User:Zlg space. It makes things a lot
easier to iteratively improve on, but no easy way to get a review aside
from pinging in #gentoo-wiki. Rather than getting someone's attention
and waiting, I think we need some way to "queue" new edits that can then
be approved by whatever group we decide to maintain the "new"
documentation site.

> 
> [snip]
> 
> 
> 4: It lacks proper offline editing capabilities
> ===============================================
> 
> I might be wrong here but I'm not aware of any fully working, SMW-
> friendly tools to work on articles offline. There's some git magic to
> pull from MediaWiki but my short testing has shown it doesn't work very
> well and can't handle the lot of our namespaces.
> 
> If you want on docs, you usually need to have your browser open
> and a reasonably stable connection. Any offline work usually requires
> extra effort, and a lot of handiwork.
> 

It'd be great to be able to shove this into a git repo and work on it
anywhere. We could adopt similar standards as our other git-related
projects to ease people -- devs or users -- into the new system.

Reliable notifications on edits would be great to have. Currently, I
keep pages in my watchlist and check them each time I need to consult
the wiki. Sometimes that's days or weeks between visits. It'd be easier
to get engaged and stay that way if there was more collaboration,
reviewing, or an accessible, codified documentation standard.

As far as I know, we have a few templates being used to help us find
pages that need attention (I particularly like our ongoing-discussion
template), but part of what makes a wiki work, imo, is people paying
attention to articles and soliciting involvement from the community.
Anything we can do to make more of that happen will be a boon for us, I
think.

> 
> [snip]
> 
> 
> What to move, and what to?
> ==========================
> 
> SMW is not something we can replace overnight. However, I think we
> should slowly start moving important stuff out of it.
> 
> The project metadata is the first thing I'd like to have out of wiki
> ASAP. Keeping it there is inconsistent, inconvenient and seems like
> putting a lot of effort just to prove that SMW could be used for
> anything.
> 
> We already keep developers in LDAP, and dump them onto main site. I see
> no reason why we couldn't keep projects in LDAP or in a flat file
> (api.gentoo.org), and include them on the main site rather than wiki. We
> can still link to wiki entities for documentation, or maybe even figure
> out how to include project members & so on from external source
> on the wiki.
> 
> The next thing I'd like to move are project docs. We'd really use some
> proper doc hosting for that, with a review platform and so on. Patrick
> has suggested using flat files in a git repository -- and I think it
> really makes much more sense than some fancy wiki software. If we need
> more power, we could just use Jekyll as for the main site.
> 
> Fun fact is, even GitHub got that right. I think it might be even
> possible to have GitHub pull requests with some degree of preview for
> that. And if everything's in git, we can easily edit and preview it
> locally.
> 
> I think it would be also reasonable to seek SMW replacement for regular
> user docs. Even if we can't figure out something nice and git-backed,
> any replacement would be much better. In fact, even Infra runs its own
> wiki on DokuWiki which is much faster.
> 
> 
> Your comments? Ideas?
> 
> 

I like DokuWiki, though to be most useful to us it'll need a few
add-ons. It's fairly straightforward to setup and to use, and supports a
wide variety of formatting styles, from DW's own wiki-like syntax to
Markdown, RST, maybe others by now. Markdown in particular is my
favorite since it's ubiquitous, but I'd be willing to learn RST as well.

I'd rather not see reliance on GitHub; but if it's there solely as a
convenience to facilitate better user involvement, then it's hard to
criticize. Devs are already mostly familiar with Git, so flat files in a
repo with a README or other file illustrating the expected hierarchy
would suffice imo.

Jekyll is another option, especially if we're already using it. I'm more
a fan of Pelican (Python-powered instead of Ruby), but either can do the
job with a little template magic and a plugin or two. The source can be
stored in git, and iirc Jekyll will also expect a particular hierarchy
if you configure it that way. In many ways I think they're just about equal.

So I guess count me as +1 for git collaboration and ease of watching
above all else. I like writing documentation in a text editor instead of
a webpage, and if I had a more reliable way to keep an eye on
documentation, it'd motivate me to contribute more to it. DW, Pelican,
and Jekyll are all fine for our use; we might want to look deeper and
consider the work necessary for documentation translation.

I toyed around with Fluxbox's wiki and turned it into a Pelican
installation a few years ago, as a sort of test run. With a plugin, it
can hook right into GNU gettext and uses familiar .po files for layout,
while translated pages have a language extension, e.g. "council.txt"
could be the English page, while the German translation would be
"council.txt.de", and so on. We could also force the requirement of a
language extension on English pages if we wanted to, so every page would
have a language extension. Pelican also natively supports a "Status:
draft" line for pages that need to be linked to directly to facilitate
collaboration, but shouldn't be public-facing.

If anyone's interested in Pelican, I can share the result of that
experiment and assist in migration to it.

Thanks for pulling this discussion together. I look forward to seeing
where it goes.
-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]