From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Sat, 28 Apr 2007 22:35:10 UTC
Message-ID: <fa.ZSPPWmSeGdI2R1NT36bizM+oSOo@ifi.uio.no>

On Sat, 28 Apr 2007, Adrian Bunk wrote:
>
> We are already quite good at ignoring bug reports that come through
> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more
> than 1600 open bugs because this tells how bad we are at handling bugs.

No, it just shows that bugzilla doesn't matter for most of the kernel.

Don't say that "bugzilla tells how bad we are at handling bugs". It tells
how bad *bugzilla* is for handling bugs, nothing more.

Trying to play politics by pointing to bugzilla is pointless. Bugzilla is
used for a few subsystems (ACPI seems to use it actively, for example),
but I doubt most developers use it.

Would be be good to have a better bug-tracking setup? Yes. But I think it
takes man-power, and it would take something *fundamentally* better than
bugzilla.

Maybe the new "http://kernelnewbies.org/known_regressions" thing will
evolve to something worth tracking. Right now, bugzilla isn't it (although
it can be a useful tracking place for individual bugs, *once* you've found
and gotten the right developer involved - but that's a huge step that
bugzilla generally does *not* do for us).

			Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Sat, 28 Apr 2007 23:30:42 UTC
Message-ID: <fa.vuULQSsUsH5KkSDZkBJhkenTbr8@ifi.uio.no>

On Sun, 29 Apr 2007, Adrian Bunk wrote:
>
> That's exactly where Linus' "drop any bug reports that are more than a
> week old" suggestion is completely flawed - no matter what the submitter
> does, how often he tests latest kernels, noone will help him.

You talk, but what do you actually *suggest*?

Talk is cheap. You use to do the walk too, but you've already said that
you're not interested in that any more. So excuse me if I'm not impressed.

The thing is, bugzilla is totally broken because it's designed to help
track bugs, but it's *not* designed to actually handle the much harder
problem, which is to actually get the *right* developers to be aware of
the *right* bugs!

And both of those "right"s are important. Spamming everybody will just
mean that everybody tunes them out. And spamming even the right developers
with useless bugreports will also just cause them to tune out the good
ones.

The thing is, the "tracking bugs" part that bugzilla _can_ do is also
totally _useless_ without that much more important phase, namely the
"connect the parties". That's what you really used to do. You made
developers connect to the reports, because your reports were _useful_ and
not overly noisy.

But go back and look - did you notice that once you connect the dots, it
turns out that bugzilla itself isn't all that wonderful. Quite a lot of
your regression reports had other ways of pointing to the problems,
including very much mailing list archive pointers etc!

So bugzilla isn't actually all it's touted to be even _once_ the
connection between reporter and developer has been established. I really
don't see why you are so hung up about bugzilla, when your own regression
reports didn't generally point to it all that often!

(I just went back and double-checked: you had more than twice as many
pointers to kernel mailing list archives than you had pointers to bugzilla
in the one series I looked at. And I'm _not_ saying that's wrong at all: I
think the mailing list is actually likely to be at LEAST as useful as
bugzilla is as a bug-tracker!)

And bugzilla actually falls down even more than the mailing list does for
the whole (and MUCH MORE IMPORTANT!) phase of connecting developers to bug
reports. And THAT is really the problem here.

And no, I don't actually think that automatically closing entries that
haven't gotten any attention in the last two weeks would "fix" bugzilla.
But I think that it might actually make people
 (a) more likely to look into bugzilla (and thus maybe improve the
     "connecting" developers/reporters thing)
 (b) act as a trivial noise-removal thing, because it would give
     preferential treatment to people who are diligent about their
     bug-reports and are willing to follow up on them and it would also
     remove the noise that comes from obvious things that broke and got
    fixed.

But I think the real fix is to have real humans that help de-noisify the
bug reports. Let's call them the "bunk" group, just to pick a random
four-letter combination. The kinds of people who help turn the mindless
noise that is bugzilla (and the kernel mailing list) into directed
_information_.

No, nothing is ever going to be perfect. And "filtering" the noise will
inevitably also end up dropping real information. But not filtering it
will guarantee that even more is dropped.

		Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Sat, 28 Apr 2007 23:41:19 UTC
Message-ID: <fa.Pa9dulANP4z7dcxjxp/+E2YDLO0@ifi.uio.no>

On Sun, 29 Apr 2007, Markus Rechberger wrote:
>
> I totally disagree here, bugzilla is a very good tool. If someone is
> too lazy to look at it it's his problem.

You must be doing things very differently from a lot of other people if
you think that's the case.

> Kernel Janitors can pick out some bugs which aren't addressed by
> anyone or got left behind.

IF that happened, it would actually be great. That's what I'm arguing for.
And it was basically what Adrian was doing!

> How else should bugs get handled, sending them to the lkml?

Actually, looking at Adrian's regression lists, yes. lkml worked better
than bugzilla did. By at _least_ a factor of two.

> I'm 100% sure some bugreports will also get lost then, but on the lkml
> they'll very likely remain lost whereas in the bugzilla they'll remain
> as open.

What's the difference between bugzilla and lkml.org? Both have search
buttons. Both archive the old stuff. Both can be pointed to.

> what are your suggestions to improve a bugreporting tool, I'm very
> sure that many people, especially people who want to get into existing
> projects here, would love to contribute.

I don't know what the perfect setup is, but I do know that bugzilla is
very close to be totally useless for the top-level maintainers.

Try to think like a person who doesn't maintain *one* specific file in the
kernel, but who can actually make a good judgement about a lot of things,
or at least funnel a problem report to the right person?

And now, imagine that that person is also fairly busy (exactly *because*
he's not looking at a single file, he may be maintaining a huge subsystem
that has multiple submaintainers etc).

And ask yourself whether bugzilla really helps.

> I'd say this is a personal opinion, some people will get along with it
> and some of them will not...

I think bugzilla really only works for very "directed" issues. If you
already know exactly which driver is affected (which is often wrong
anyway: some of the bugs that were due timer breakage got blamed as disk
hangs!) it's almost totally useless.

And yes, maybe that's why you have a much higher opinion of bugzilla than
I do. To _me_ bugzilla is a total mess. There's absolutely _zero_ useful
information there. And I'm pretty certain that is true of a *lot* of other
people too. But if you have a small project, or you maintain a very
specific (and clearly delineated) part of a big project, bugzilla probably
looks a lot more palatable.

			Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Sat, 28 Apr 2007 23:59:18 UTC
Message-ID: <fa.Reln3tgflktcUz2lXLtFNie/KuU@ifi.uio.no>

On Sun, 29 Apr 2007, Adrian Bunk wrote:
>
> Bugzilla has an email interface.
> Andrew forwards bugs from Bugzilla to developers.

Yes. And if you search around, you'll even see that I occasionally use it.

But it's useful only once the bug has been assigned and somebody has
actually *looked* at it. The fact that some people do this is a big credit
(Andrew, Dave J etc)

> There might be small room for improvements, but I don't see how
> man-power or technology could make a big difference in this area.

I'm talking about getting the developers to _look_ at the bugs in the
first place. Bugzilla is not very good at that, because it has no useful
interfaces for doing so (unless you can specify your area of interest so
exactly that you can actually set yourself up as a maintainer of one
particular area).

Almost none of the subtle (and thus harder) bugs tend to fall into that
kind of nice category.

For example, look at suspend/resume bugs. Do you realize that 99% of them
are device driver issues, but how the heck do you connect a "my laptop
does't resume" to the _right_ device driver maintainer?

And do you realize (and acknowledge) that it would be total madness to
send all suspend/resume bugs to _everybody_ who maintains any device
driver at all?

See? THAT is the problem with bugzilla. It only works for the "easy"
cases. It works for the case where a reporter can say with certainty that
the reason his machine doesn't boot is a particular network device driver
(like the sis900 regression we had in 2.6.21). But once you know the
subarea that precisely, bugzilla doesn't even help you that much, and it's
likely easier and more productive to just send email directly to the right
person and cc Jeff and netdev or something!

> "*once* you've found and gotten the right developer involved" is the
> real problem, not how to track bugs.

And I agree 100%.

And bugzilla does *nothing* here.

> *This* is *the* problem.
> And no change in bug tracking will help with this problem.

So why are you pushing bugzilla?

There are actually better bug trackers around. One of them is "google".
For oopses, one of the thngs I do is to put in the most relevant
information (backtrace etc) into google, and ask google to try to find the
pattern. That sometimes actually does pretty well - you can get a real
feel for "oh, there's a pattern here - they're all AMD machines with the
NVidia chipset" kind of thing.

Bugzilla doesn't offer anything even remotely as useful.

It's the "big picture" that tends to be hard to find.

And that's what *you* gave with the regression report. A summary. You even
noted some of the patterns, so people actually had them already somewhat
pre-sorted.

THAT is what we want.

But another way to get the "high-level picture" is to actually just say
"what's active now". And that's why I think the whole "1600 open
bug-reports" kind of thing is useless. Bugzilla does *not* have mail
interfaces for that kind of "these 50 bugs were actively reported on and
unresolved" in the last week summaries!

(And if it did, it would still almost certainly need some human care and
understanding)

		Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Sun, 29 Apr 2007 16:08:58 UTC
Message-ID: <fa.cfi8a2XQorMdcwWSZ/UqBMjEkbw@ifi.uio.no>

On Sun, 29 Apr 2007, Andi Kleen wrote:
> Linus Torvalds <torvalds@linux-foundation.org> writes:
> >
> > The thing is, bugzilla is totally broken because it's designed to help
> > track bugs, but it's *not* designed to actually handle the much harder
> > problem, which is to actually get the *right* developers to be aware
> > of the *right* bugs!
>
> This means we need people who figure out who to assign bugs too.
> Aka bugmasters.

Yes. But not using bugzilla.

I don't know if you've noticed already, but I'm not the only one that
doesn't have a very high opinon of bugzilla ;)

What works is somebody who is a bugmaster, and it doesn't really matter
*what* bug tracker he points to (bugzilla being one of the possibilities,
although not necessarily the best, and absolutely NOT the only choice),
and turn them into emails. Once they are emails, bugzilla can track them.

Andrew does this, and yes, Adrian did it.

> In theory it could be nearly automated. Figure out what files related
> to the bug and assign to the last 5 people who submitted patches
> for them and/or signed off.

I do think you're pretty optimistic. I think it's true for trivial driver
bugs, but even for trivial driver bugs the initial report is often not
enough to pinpoint the driver.

Let's take the sis900 bug as an example (not because I want to rag on that
being a problem, but it happened to be a _trivial_ bug in 2.6.21, so it's
a good case of something really really easy - and if that easy case isn't
handled trivially and obviously, then the bug-reporting doesn't work).

In that case, the initial report was (condensed version, but fairly
accurate): "system hangs on boot at random points. 2.6.21-rc7 worked
well".

Now, realistically, if that entry had been in bugzilla, what would you do?

Equally realistically, let's ignore bugzilla for a moment, and ask what
the best method for handling something like this would be? Have an open
mind, no rules on "have to use bug tracker XYZ".

You know what? The report went to me and the kernel mailing list as email.
And that was the *right* thing in that case. There was no good sign of who
it should go to, and while there wasn't a whole lot of information there,
there *was* a very tight timeframe (ie it could pinpoint to within about a
week when it started). But the only thing I could really ask for was for
the person to bisect it.

Would bugzilla have helped? HELL NO. It would have been a disaster. It
would have wasted reporter time, it would have wasted developer time, and
it would likely have been ignored because the bug report wasn't specific
enough to really trigger any good queries or trigger a maintainer.

So bugzilla wouldn't have helped even for a _trivial_ bug.

I personally suspect that bugzilla helps more if a bug actually has a real
history, and people want to actually save that history because they are
stumped. But I think it should come in *then*, not immediately. Because it
is actually too intrustive for the simple stuff - and most bugs actually
are simple, it's just that they also get resolved so simply that people
don't even think of them as bugs (ie we have a lot of "duh" kind of fixes
too).

> BTW one big problem in our current bugzilla is that a lot of people
> cannot reassign bugs they don't own. I sometimes see bugs that I don't
> own bug I know who is responsible, but bugzilla doesn't allow me to do it.

I think that's a problem, but I think the thing that I react to most (and
I know some other people do too) is that I just don't want a broken mousy
interface that I have to explicitly go and look at. I'd much rather get
involved by email once it's clear that I need to be involved. Having a
link to "more information at xyz" (where xyz _can_ be bugzilla too, but
doesn't have to be!) is fine, but but it really cannot be the first
interface, not in the current form, at least.

> So I think what would help:
>
> - Ask more people to just categorize and reassign bugs (anybody interested?)
> - Give more people in bugzilla the power to reassign arbitary bugs
>   (bugzilla maintainers would need to do that)

I think both of these are good ideas, but I don't think it's enough. There
must be some better form of bug tracking than bugzilla. Some relly
distributed way of letting people work together, *without* having to
congregate on some central web-site kind of thing. A "git for bugs", where
you can track bugs locally and without a web interface.

And I think it would be something much closer to "distributed email mbox
with status tracking facilities", _not_ a web clicky interface.

		Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Sun, 29 Apr 2007 17:48:08 UTC
Message-ID: <fa.7SqOJXBgETmQMAwF7pEUswQR6cE@ifi.uio.no>

On Sun, 29 Apr 2007, Andi Kleen wrote:
> On Sun, Apr 29, 2007 at 09:07:43AM -0700, Linus Torvalds wrote:
> >
> > Yes. But not using bugzilla.
>
> And what would you use instead?

Didn't you even *read* my email?

I already told you: we have real bugs getting reported and fixed that
don't hit bugzilla or any bugtracker AT ALL.

This is not a "all or nothing" situation. There is absolutely _zero_ real
reason to think that everything has to go through bugzilla.

> > I don't know if you've noticed already, but I'm not the only one that
> > doesn't have a very high opinon of bugzilla ;)
>
> Sure, but do you have alternatives?

Yes. The ones that *work*. Plain email is preferably over bugzilla 90% of
the time.

But quite frankly, if you think you can make bugzilla work (and realize
that a lot of people will _not_ be looking at it or reporting bugs into
it), go ahead. I don't care. The only think I care about is *REALITY*, and
that means:

 - a lot of reporters will not use bugzilla, because it's damn
   inconvenient even for reporting. If you propose something that uses
   _only_ bugzilla, you'd better also have the people who enter other
   peoples bugreports into there.

 - a lot of developers will not use bugzilla, because it's even more
   inconvenient for developers, with no sane ways to interact with the
   right people. So if you propose using bugzilla, you'd really better
   have the manpower to turn bugzilla into emails (and no, the bugzilla
   cc list etc is _not_ the primary one - the email cc's are the primary
   ones, because that's where it is much easier to bring in new people)

In other words, anything that thinks that bugzilla is "primary" is just
broken. It can be a _part_ of the thing, but drop the belief of it being a
primary tracker. It's just too inconvenient.

		Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Sun, 29 Apr 2007 17:51:23 UTC
Message-ID: <fa.oq8g5KLE4bL6fqJAcWUjEiY+894@ifi.uio.no>

On Sun, 29 Apr 2007, Andi Kleen wrote:
>
> Besides the primary point of bug tracking is not to be friendly
> to someone, but to (a) fix the bugs and (b) know how many bugs
> there for a given release. Any replacement would need to solve
> this problem too.
>
> Email does not solve it as far as I can see.

Email fixes a _lot_ more bugs than bugzilla does.

End of story. I don't think anybody who cannot accept that UNDENIABLE FACT
should even participate in this discussion. Wake up and look at all the
bugs we fix - most of them have never been in bugzilla.

That's a FACT.

Don't go around ignoring reality.

		Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Sun, 29 Apr 2007 18:59:32 UTC
Message-ID: <fa.yYBW4UZM4wy0hw0fWgKJLMD4Om0@ifi.uio.no>

On Sun, 29 Apr 2007, Rafael J. Wysocki wrote:
>
> - You are required to select a category and 'component' for your report, which
>   often is difficult (especially if you're not a kernel expert)
> - You need to have a bugzilla account (or to create one, if you don't)

Amen.

Both of those are show-stoppers. It may be ok for kernel developers, but
it's not good for random people. It's one of the reasons I don't generally
use bugzilla for other projects - if they require me to sign up etc, they
can take care of their own bugs.

(For the same reason I consider closed mailing lists to be useless for
bugreports. If you have to be a member to send email, it's not a
bug-report thing, it's just a secret society).

I realize that spam is a problem, but there are better spam solutions than
alienating the people who just want to send a report.

Also, I don't know if anybody has ever tried to avoid sending duplicate
bugs, but every time I use bugzilla to send a bug-report (which I do for
things like FC7 live CD's breaking etc where I care enough, and I have a
bugzilla account on that bugzilla _anyway_ for other reasons), I am ready
to _kill_ somebody whenever I see that "humorous"

	zarro bugs found

message, after it made it almost impossible for me to even figure out how
to do a good search in the first place!

So I gnash my teeth, and fill in the bug report anyway, and if it's a
duplicate becasue bugzilla didn't have any sane way to get any kind of
overview at all, hey, it's a duplicate. Nothing I can do about it.

And the reason I gnash my teeth is that I realize that because I can't
even get any overview of the bugzilla entries as a random submitter (and
dammit, I probably have a better idea of what the problem might be than
most people), I sure as hell can't expect a random user to be better. So I
bet my bad reports are a lot better than the average, and a lot of people
probably just give up and don't report anything at all.

		Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Sun, 29 Apr 2007 21:25:19 UTC
Message-ID: <fa.hS4SiNm/mkZPAqCZHzb/SgC9cTs@ifi.uio.no>

On Sun, 29 Apr 2007, Adrian Bunk wrote:
> >
> > Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
>
> OK, how do you suggest to track bugs in a way that doesn't suck?

I've tried to explain.

Bugzilla can be one _part_ of it, but anybody who thinks it's the "main
part" is really not being realistic. It's too cumbersome, and it's too
stupid.

Quite frankly, "lkml + google" is probably in many ways a *better* way to
search for problems. But yes, some manual smarts (and the _occasional_
pointer to bugzilla) is probably currently the only option.

Exactly because I don't think anybody has shown any better automation than
bugzilla. But that doesn't make bugzilla "the One Choice". That's not how
it works. If there is no automation, manual tracking is still better than
*crap* automation.

> Bug reports to linux-kernel have the big problem that they are lost if
> no developer immediately picks them up.

..and this is different from bugzilla exactly _how_?

Those things are lost too. As you yourself have pointed out. The fact that
you can search for them is _exactly_ as relevant as the fact that you can
search for lkml on google.

> What I do know is that the majority of them has never been proper
> debugged by a kernel developer knowing the subsystem in question.

And you blame the developers, but not bugzilla? Why are you so unable to
see bugzilla as part of the *cause* of the problem? You're perfectly happy
to blame other things, but bugzilla is somehow above blame?

		Linus

From: Theodore Tso <tytso@mit.edu>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Sun, 29 Apr 2007 23:56:31 UTC
Message-ID: <fa.yKxdIndlq91YvsHGyO46BLs0Qtw@ifi.uio.no>

On Sun, Apr 29, 2007 at 03:15:42PM +0200, Andi Kleen wrote:
> This means we need people who figure out who to assign bugs too.
> Aka bugmasters.
>
> BTW one big problem in our current bugzilla is that a lot of people
> cannot reassign bugs they don't own. I sometimes see bugs that I don't
> own bug I know who is responsible, but bugzilla doesn't allow me to do it.
>
> So I think what would help:
>
> - Ask more people to just categorize and reassign bugs (anybody interested?)
> - Give more people in bugzilla the power to reassign arbitary bugs
> (bugzilla maintainers would need to do that)

Folks might want to take a look at the Debian Bug Tracking System
(BTS).  It has a web interface which you can use to query history, but
*everything* is e-mail driven, and the way you submit, close, update,
tag/classfy bugs --- everything --- is via e-mail.

More importantly, anyone is allowed to recategorize and reassign bugs.
If someone does so maliciously or incorrectly, you can always revert
it, and if someone is being truly malicious, you can always blacklist
that one person.  It this respect, it is far more wiki-like than
bugzilla, which has always been too much like a straightjacket.

It's not perfect, but it's better than bugzilla --- but then again,
just about *anything* would be better than bugzilla.  (Hmm, except
maybe SourceForge's very tragic bug tracking system... :-)

Of course, as Linus has said, it's not a complete solution --- you
still need humans to be smart about things --- but if the goal is to
make it easier to archive and track information about a bug, at
*least* with the Debian BTS, when you reply to an e-mail message, the
reply is automatically appended to the bug log!

						- Ted

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Mon, 30 Apr 2007 18:21:33 UTC
Message-ID: <fa.CjsU70SLMi5U2jALelEjscP4IEw@ifi.uio.no>

On Mon, 30 Apr 2007, Adrian Bunk wrote:
>
> My ideal was always that reported bugs should be fixed.

..and this is where we differ.

OF COURSE bugs should be fixed. But you seem to think that there is
something magical and special about every single bug-report.

You have a new home assignment: watch the "every sperm is sacred" thing
from Monty Python's "Meaning of Life". Google for it.

And if you cannot appreciate the absurdity and humor of that thing, maybe
you should think about it a bit more.

And once you _can_ appreciate the humor of that song/skit, look yourself
in the mirror, and ask yourself: "is every bug report sacred?"

Really?

> If you accept that this is anyway impossible because more bugs get added
> than could get fixed you might not need any tracking at all.

That's a TOTALLY IDIOTIC argument.

That goes from "every sperm is sacred" to "sperm doesn't count at all".

Can you not see how stupid that statement of yours really is? Can you not
see that anybody who thinks in those kinds of black-and-white terms is
simply not FUNCTIONAL!

Bugs are neither sacred, _nor_ should they be ignored.

Ponder that, grasshopper. And until you can see that things are not
"either-or", "black-and-white", "all or nothing", I don't think I really
can have anything worthwhile to add in this discussion to you. People who
think in absolutes are simply not worth talking to.

			Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.21
Date: Mon, 30 Apr 2007 18:28:31 UTC
Message-ID: <fa.tObCx1Uwu0PgZXtO7yIpkNzsuv4@ifi.uio.no>

On Mon, 30 Apr 2007, Linus Torvalds wrote:

> Ponder that, grasshopper. And until you can see that things are not
> "either-or", "black-and-white", "all or nothing", I don't think I really
> can have anything worthwhile to add in this discussion to you. People who
> think in absolutes are simply not worth talking to.

This is another case of "perfect is the enemy of good".

Tryng to reach perfect is not only guaranteed to fail, but trying to reach
it AND NOT REALIZING that it's stupid and wrong is actually much WORSE
than just trying to do a reasonable job.

And if you put some _totally_idiotic_ expectation that all bugreports can
be fixed, and should always be totally blocking, that's guaranteed to just
cause a totally unusuable bug reporting system.

And your bugzilla arguments seem to be exactly that. A naïve and totally
unrealistic expectation of "every bugreport is sacred" is BAD.

In other words: perfection not only isn't even possible, BUT IT IS NOT
EVEN WORTH TRYING TO REACH FOR!

			Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: This is [Re:] How to improve the quality of the kernel[?].
Date: Tue, 19 Jun 2007 15:04:55 UTC
Message-ID: <fa.Hf8vimdQv67gDPNXrRd2XuPMWew@ifi.uio.no>

On Tue, 19 Jun 2007, Adrian Bunk wrote:
>
> The goal is to get all patches for a maintained subsystem submitted to
> Linus by the maintainer.

Well, to be honest, I've actually over the years tried to have a policy of
*never* really having black-and-white policies.

The fact is, some maintainers are excellent. All the relevant patches
*already* effectively go through them.

But at the same time, other maintainers are less than active, and some
areas aren't clearly maintained at all.

Also, being a maintainer often means that you are busy and spend a lot of
time talking to *people* - it doesn't necessarily mean that you actually
have the hardware and can test things, nor does it necessarily mean that
you know every detail.

So I point out in Documentation/ManagementStyle (which is written very
much tongue-in-cheek, but at the same time it's really *true*) that
maintainership is often about recognizing people who just know *better*
than you!

> The -mm kernel already implements what your proposed PTS would do.
>
> Plus it gives testers more or less all patches currently pending
> inclusion into Linus' tree in one kernel they can test.
>
> The problem are more social problems like patches Andrew has never heard
> of before getting into Linus' tree during the merge window.

Not really. The "problem" boils down to this:

	[torvalds@woody linux]$ git-rev-list --all --since=100.days.ago | wc -l
	7147
	[torvalds@woody linux]$ git-rev-list --no-merges --all --since=100.days.ago | wc -l
	6768

ie over the last hundred days, we have averaged over 70 changes per day,
and even ignoring merges and only looking at "pure patches" we have more
than an average of 65 patches per day. Every day. Day in and day out.

That translates to five hundred commits a week, two _thousand_ commits per
month, and 25 thousand commits per year. As a fairly constant stream.

Will mistakes happen? Hell *yes*.

And I'd argue that any flow that tries to "guarantee" that mistakes don't
happen is broken. It's a sure-fire way to just frustrate people, simply
because it assumes a level of perfection in maintainers and developers
that isn't possible.

The accepted industry standard for bug counts is basically one bug per a
thousand lines of code. And that's for released, *debugged* code.

Yes, we should aim higher. Obviously. Let's say that we aim for 0.1 bugs
per KLOC, and that we actually aim for that not just in _released_ code,
but in patches.

What does that mean?

Do the math:

	git log -M -p --all --since=100.days.ago | grep '^+' | wc -l

That basically takes the last one hundred days of development, shows it
all as patches, and just counts the "new" lines. It takes about ten
seconds to run, and returns 517252 for me right now.

That's *over*half*a*million* lines added or changed!

And even with the expectation that we do ten times better than what is
often quoted as an industry average, and even with the expectation that
this is already fully debugged code, that's at least 50 bugs in the last
one hundred days.

Yeah, we can be even more stringent, and actually subtract the number of
lines _removed_ (274930), and assume that only *new* code contains bugs,
and that's still just under a quarter million purely *added* lines, and
maybe we'd expect just new 24 bugs in the last 100 days.

[ Argument: some of the old code also contained bugs, so the lines added
  to replace it balance out. Counter-argument: new code is less well
  tested by *definition* than old code, so.. Counter-counter-argument: the
  new code was often added to _fix_ a bug, so the code removed had an even
  _higher_ bug rate than normal code..

  End result? We don't know. This is all just food for thought. ]

So here's the deal: even by the most *stringent* reasonable rules, we add
a new bug every four days. That's just something that people need to
accept. The people who say "we must never introduce a regression" aren't
living on planet earth, they are living in some wonderful world of
Blarney, where mistakes don't happen, developers are perfect, hardware is
perfect, and maintainers always catch things.

> The problem is that most problems don't occur on one well-defined
> kind of hardware - patches often break in exactly the areas the patch
> author expected no problems in.

Note that the industry-standard 1-bug-per-kloc thing has nothing to do
with hardware. Somebody earlier in this thread (or one of the related
ones) said that "git bisect is only valid for bugs that happen due to
hardware issues", which is just totally *ludicrous*.

Yes, hardware makes it harder to test, but even *without* any hardware-
specific issues, bugs happen. The developer just didn't happen to trigger
the condition, or didn't happen to notice it when he *did* trigger it.

So don't go overboard about "hardware". Yes, hardware-specific issues have
their own set of problems, and yes, drivers have a much higher incidence
of bugs per KLOC, but in the end, even *without* that, you'd still have to
face the music. Even for stuff that isn't drivers.

So this whole *notion* that you can get it right the first time is
*insane*.

We should aim for doing well, yes.

But quite frankly, anybody who aims for "perfect" without taking reality
into account is just not realistic. And if that's part of the goal of some
"new process", then I'm not even interested in listening to people discuss
it.

If this plan cannot take reality into account, please stop Cc'ing me. I'm
simply not interested.

Any process that tries to "guarantee" that regressions don't happen is
crap. Any process that tries to "guarantee" that we release only kernels
without bugs can go screw itself. There's one thing I _can_ guarantee, and
that's as long as we add a quarter million new lines per 100 days (and
change another quarter million lines), we will have new bugs.

No ifs, buts or maybe's about it.

The process should aim for making them *fewer*. But any process that aims
for total eradication of new bugs will result in one thing, and one thing
only: we won't be getting any actual work done.

The only way to guarantee no regressions is to make no progress.

		Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: This is [Re:] How to improve the quality of the kernel[?].
Date: Tue, 19 Jun 2007 17:07:02 UTC
Message-ID: <fa.R8HakB5aNX4UcuOUF85F4W8iOFg@ifi.uio.no>

On Tue, 19 Jun 2007, Oleg Verych wrote:
>
> I'm proposing kind of smart tracking, summarized before. I'm not an
> idealist, doing manual work. Making tools -- is what i've picked up from
> one of your mails. Thus hope of having more opinions on that.

Don't get me wrong, I wasn't actually responding to you personally, I was
actually responding mostly to the tone of this thread.

So I was responding to things like the example from Bartlomiej about
missed opportunity for taking developer review into account (and btw, I
think a little public shaming might not be a bad idea - I believe more in
*social* rules than in *technical* rules), and I'm responding to some of
the commentary by Adrian and others about "no regressions *ever*".

These are things we can *wish* for. But the fact that we might wish for
them doesn't actually mean that they are really good ideas to aim for in
practice.

Let me put it another way: a few weeks ago there was this big news story
in the New York Times about how "forgetting" is a very essential part
about remembering, and people passed this around as if it was a big
revelation. People think that people with good memories have a "good
thing".

And personally, I was like "Duh".

Good memory is not about remembering everything. Good memory is about
forgetting the irrelevant, so that the important stuff stands out and you
*can* remember it. But the big deal is that yes, you have to forget stuff,
and that means that you *will* miss details - but you'll hopefully miss
the stuff you don't care for. The keyword being "hopefully". It works most
of the time, but we all know we've sometimes been able to forget a detail
that turned out to be crucial after all.

So the *stupid* response to that is "we should remember everything". It
misses the point. Yes, we sometimes forget even important details, but
it's *so* important to forget details, that the fact that our brains
occasionally forget things we later ended up needing is still *much*
preferable to trying to remember everything.

The same tends to be true of bug hunting, and regression tracking.

There's a lot of "noise" there. We'll never get perfect, and I'll argue
that if we don't have a system that tries to actively *remove* noise,
we'll just be overwhelmed. But that _inevitably_ means that sometimes
there was actually a signal in the noise that we ended up removing,
because nobody saw it as anything but noise.

So I think people should concentrate on turning "noise" into "clear
signal", but at the same time remember that that inevitably is a "lossy"
transformation, and just accept the fact that it will mean that we
occasionally make "mistakes".

This is why I've been advocating bugzilla "forget" stuff, for example. I
tend to see bugzilla as a place where noise accumulates, rather than a
place where noise is made into a signal.

Which gets my to the real issue I have: the notion of having a process for
_tracking_ all the information is actually totally counter-productive, if
a big part of the process isn't also about throwing noise away.

We don't want to "save" all the crud. I don't want "smart tracking" to
keep track of everything. I want "smart forgetting", so that we are only
left with the major signal - the stuff that matters.

		Linus