Index Home About Blog
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 16:27:49 UTC
Message-ID: <fa.zqn9ooVcM3cJM6FGMmm5ir3yegw@ifi.uio.no>

On Thu, 11 Jun 2009, Christoph Hellwig wrote:
>
> Err, no.   This adds tons of userspace code into tools/ which
> should not be in the kernel tree but a proper package.

I disagree.

We've had tons of cases where we tried to "separate" the user-land code
and the kernel code, in the name of "beauty" of whatever.

It's almost invariably a disaster.

Look at oprofile. F*ck me, what a horrid piece of crap. It took literally
months for the user mode tools to catch up and get the patches to support
new functionality into CVS (or is it SVN?), and after that it took even
longer for them to become part of a release and be picked up by
distributions. In fact, I'm not sure it is part of a release even now - I
had to make a bug report to Fedora to get atom and Nehalem support in my
tools: I think they took the unofficial patch.

Or look at the crazy things we used to do for X. It's going away (slowly),
because some of the most incestuous things are actually just being
integrated into the kernel, and so there's less of the "two broken pieces"
approach, and more of a "one working piece" kind of thing.

So I'd much rather have kernel tools with the kernel, than have to depend
on some external entity that doesn't really care.

			Linus


From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 16:39:57 UTC
Message-ID: <fa.HnZrvvKDN4KS3Q1k/tQvLyEavk8@ifi.uio.no>

On Thu, 11 Jun 2009, Marcel Holtmann wrote:
>
> so do you expect us to merge stuff like ip, iw, rfkill, crda, the WiMAX
> tools, the Bluetooth ones and whatever we have that are all have the
> same issues to be merged into the kernel source code as well.

No. Only stuff that I expect to be really close to hardware, and used for
kernel purposes.

> Also please consider the distro point of view. All these distros have
> already a hard time to keep up with the kernel patches etc. It is a lot
> easier to update a userspace package then having to provide a patches
> kernel source.

Feel free to split it all up if it turns out to be stable later.

But I refuse to go through another oprofile.

		Linus


From: Al Viro <viro@ZenIV.linux.org.uk>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 16:52:50 UTC
Message-ID: <fa.itpsJKO5BE0V+estJruihtXMsgA@ifi.uio.no>

On Thu, Jun 11, 2009 at 09:26:55AM -0700, Linus Torvalds wrote:
>
>
> On Thu, 11 Jun 2009, Christoph Hellwig wrote:
> >
> > Err, no.   This adds tons of userspace code into tools/ which
> > should not be in the kernel tree but a proper package.
>
> I disagree.
>
> We've had tons of cases where we tried to "separate" the user-land code
> and the kernel code, in the name of "beauty" of whatever.
>
> It's almost invariably a disaster.
>
> Look at oprofile. F*ck me, what a horrid piece of crap.

Yes.  So's sysfs, so's udev, so's hal, so's any number of revolting
strings of intertwined copulating tapeworms hanging off the kernel's arse.

Do you consider "put into tools/" as permission to change interface at will?
More to the point, do the authors of that stuff consider it as such?




From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 16:55:34 UTC
Message-ID: <fa.vWMo3kZSpi+7+7boRGz0SPCrSVU@ifi.uio.no>

On Thu, 11 Jun 2009, Christoph Hellwig wrote:
>
> Did you take a look a tools/perf/?  There is nothing close to hardware
> at all.  It's all pretty highly abstracted away from anything resembling
> the hardware through the perfcounters interface.

The thing is, the raw perfcounters interface isn't going to be useful as
is. And I have seen where things go when you split them up. So when I get
the choice, I'll go down the road of unproven failure, in the hope that it
will be successful, rather than doing the same mistake once more.

   "Insanity: doing the same thing over and over, expecting to get
    different results."

And I'm not insane.

Anyway, feel free to disagree. I just don't care.

			Linus


From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 16:59:43 UTC
Message-ID: <fa.pO3GPIP20maNwXdn/rJxVaDsxWQ@ifi.uio.no>

On Thu, 11 Jun 2009, Al Viro wrote:
>
> Yes.  So's sysfs, so's udev, so's hal, so's any number of revolting
> strings of intertwined copulating tapeworms hanging off the kernel's arse.

Those are about a different thing, though - they are largely about policy.
Very different from something like profiling (or graphics acceleration).

I do like your visuals, though.

			Linus

From: Al Viro <viro@ZenIV.linux.org.uk>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 17:13:35 UTC
Message-ID: <fa.KeqHeCT7N6wNRrKPLzpxzJ22770@ifi.uio.no>

On Thu, Jun 11, 2009 at 10:05:02AM -0700, Ray Lee wrote:
> On Thu, Jun 11, 2009 at 10:00 AM, Christoph Hellwig<hch@infradead.org> wrote:
> > On Thu, Jun 11, 2009 at 06:56:18PM +0200, Peter Zijlstra wrote:
> >> No, once a kernel with this syscall gets released we most certainly
> >> intend to maintain its ABI.
> >
> > So what point is there in keeping it in-tree except making life hell for
> > packagers?
>
> Packagers are quite used to taking a single source tree and building
> multiple packages out of it. This isn't rocket science. It's the
> multiple separate trees that need to be released in lock-step that are
> headaches.

Wrong.  Remember the fun bisecting around sysfs/udev incompatible change?
Oops, went back past the cutoff line, got to downgrade udev for the next
boot.  Oh, it oopses?  Too fucking bad, can't just boot the previous kernel,
should've kept _two_ working ones so that with any userland state we could
come back to working system.

This isn't a rocket science, this is a goddamn load of horse manure.
Packages that need to be updated in the lock-step *are* headaches from
hell when you are trying to do development.  Even if you have all of
them already built.


From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 18:35:42 UTC
Message-ID: <fa.p/qljTdTOewiLNwUZ8U2L1NrazE@ifi.uio.no>

On Thu, 11 Jun 2009, Martin Bligh wrote:
>
> We actually ended up coming to the same conclusion as you for some of the
> internal tools we use that are tightly tied to the kernel. There is one hitch,
> which is that if you boot between different kernel versions, you need multiple
> userspace versions of the tools, so you may need to put them in
> /lib/modules/<kernel-version> or something equivalent, not one fixed place.

So I actually think this is broken.

No tool should ever be _that_ tightly tied to a kernel. If they are, they
are broken, plain and simple.

A stable user-space ABI is still a requirement.

What the "keep it in the kernel sources" approach hopefully allows is

 - taking advantage of new features in a timely manner.

   NOT with some ABI breakage, but simply things like supporting a new CPU
   architecture or new counters. The thing that oprofile failed at so
   badly in my experience.

 - Make it easier for developers, and _avoiding_ the horrible situation
   where you have two different groups that don't talk well to each other
   because one is a group of user-space weenies, and the other is a group
   of manly kernel people, and there is no common ground.

And no, I'm not going to "guarantee" that this works well. Again, I just
know that the separation didn't work. Let's just _try_ to do it this way,
and see how it works.

But at no point will it be acceptable to have kernel version dependencies.
Install the newest version of the binaries, and it should support older
kernels too (within reason).

The "within reason" is because (a) it's new, so early on you might see
breakage, and (b) because we do tend to allow system tools to break
occasionally. Not nearly often enough to make it valid to design around
it, though.

		Linus


From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 19:25:07 UTC
Message-ID: <fa.3isPe9XsFmWWmaBEcbfXAQrCp+g@ifi.uio.no>

On Fri, 12 Jun 2009, David Newall wrote:
>
> You seem to be saying that putting the code in kernel tree will make
> user-level developers more responsive. FWIW (very little) I would have
> quietly guessed the opposite result.

No. I'm saying that if there's a big overlap with _kernel_ developers
(which there is), then they can maintain the tree.

		Linus


From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 19:30:48 UTC
Message-ID: <fa.zHTLWa5VU0HWDb248dLefjureYY@ifi.uio.no>

On Thu, 11 Jun 2009, Linus Torvalds wrote:
>
> No. I'm saying that if there's a big overlap with _kernel_ developers
> (which there is), then they can maintain the tree.

To take the oprofile example that decided it for me: the code to actually
support new processors was all done by basically kernel developers. And it
didn't hit user land for almost a year, because the user-land tools didn't
take the patch and propagate it up.

		Linus


From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 19:50:33 UTC
Message-ID: <fa.JlKJaGUEEq1uoYcqjo5juZfLbi8@ifi.uio.no>

On Fri, 12 Jun 2009, David Newall wrote:

> Linus Torvalds wrote:
> > To take the oprofile example that decided it for me: the code to actually
> > support new processors was all done by basically kernel developers. And it
> > didn't hit user land for almost a year, because the user-land tools didn't
> > take the patch and propagate it up.
>
> Bad developer, Spot, you only did half the job. Not sure there's much
> more one can say.

Umm. The kernel developer _did_ do the job. The patch to the user land
side was available for that whole year. It just didn't get merged, and
then didn't get merged some more, and then got merged but only in a SVN
tree, not a release, and then finally when I did a bugzilla request to
fedora, they took the patch and put it in their distro.

Anyway, it's clearly not worth discussing this with you. I've tried. I
give up. Happily, I don't _need_ to convince you.

		Linus


From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 20:10:44 UTC
Message-ID: <fa.ui1JcdkJ/AXeFxWf73J0SfDJbT8@ifi.uio.no>

On Thu, 11 Jun 2009, Andrew Morton wrote:
>
> It could be that shipping userspace code in the kernel bundle will
> improve that situation.  So let's give it a try.  If it turns out to be
> good, let's do it again.  If it turns out to be bad, let's move perf
> out of the kernel tree and not do it again.

Exactly. Right now, I use the oprofile experience as a reason for why we
should try to do this. But hey, who knows, in one year, maybe people will
use _this_ experience as a reason why we should never do it again.

We just don't know yet. But that's no reason not to try. Either way, we'll
hopefully learn something.

Or to quote Edison:
 "I have not failed 700 times. I have not failed once. I have succeeded in
  proving that those 700 ways will not work. When I have eliminated the
  ways that will not work, I will find the way that will work."

			Linus


From: Al Viro <viro@ZenIV.linux.org.uk>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Thu, 11 Jun 2009 23:20:26 UTC
Message-ID: <fa.V6NkJL6XUavtc+7kGQSI31KdK0o@ifi.uio.no>

On Thu, Jun 11, 2009 at 03:27:36PM -0700, Linus Torvalds wrote:
>
>
> On Fri, 12 Jun 2009, Jiri Slaby wrote:
> >
> > Bah, having 40M .src.rpm for a 5k binary package?
>
> Why do people who don't even know how packaging works bother to even
> participate in the discussion?
>
> Look at how many git binary packages there are some day. For CVS users,
> for SVN people, graphical tools etc. Do you think that each of them has a
> source package?
>
> No.
>
> You can generate multiple binary packages from the same source package
> (trivial example: debug builds etc). But you want to make a point, and
> then YOU USE SOME DAMN IDIOTIC AND IGNORANT argument to do so.

Linus, the real question that needs to be answered is this:

	What shall be done to ABI-breaking changes when users of that ABI are
in tools/*?

_That_ is the real issue.  Because I can guarantee that there will be attempts
to use that as an excuse for ABI breakage.  We have one specimen in this
thread already, complete with "oh, bisect problems don't matter, just rebuild
all packages" (and install them where, exactly? if it, say, break-the-boot
kind of incompatibility, how does one recover from running into a b0rken
kernel during bisect?)

If you are willing to ban that kind of crap - great; there are real remaining
issues (mostly with choosing the dependencies between binary packages), but
that's more or less survivable.  If not... we'll have one hell of a PITA
to deal with when that kind of excuse gets actually used.


From: Al Viro <viro@ZenIV.linux.org.uk>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Fri, 12 Jun 2009 00:27:06 UTC
Message-ID: <fa.YBacxl3u9FAMtheKfLQ1xMFMlFA@ifi.uio.no>

On Thu, Jun 11, 2009 at 04:25:19PM -0700, Linus Torvalds wrote:
>
>
> On Fri, 12 Jun 2009, Al Viro wrote:
> >
> > Linus, the real question that needs to be answered is this:
>
> No it's not.
>
> People have already told you that the intent isn't to change the ABI. So
> your whole "hard-hitting journalism" is just bogus posturing.
>
> What does this have to do with anything?

Oh, for...  I can bloody well read, I've seen the reply from Peter and
I've no reasons to doubt his words (and if I had, I would've said so).
Not the issue.  I don't know who you are confusing me with, but for the
record - I have no problem with this particular code being in tree.

I do have a problem with another thing: suggestions I've heard quite a few
times before; basically, "let's allow special breakable ABIs for use by
userland code living in kernel tree and tied to specific version".  No,
I'm not saying that this is what's happening with that merge.  But your
support for userland code in the tree (and BTW, I agree that it's a good
idea - hell, mount(8) makes a good candidate as far as I'm concerned) will
be parsed as green light for that.  Has been already, in this thread.

So could you please clarify the situation?  If the ABI compatibility
requirements remain the same as they used to be, whether the userland code
is in-tree or not, I'm fine with the entire thing.  If they do not (and *ONLY*
in that case), I think we have a real problem.

For the record, I don't give a damn about packaging-related arguments and
theories about keeping userland source separate as a matter of some principle.
As far as I'm concerned, it's not a problem - as long as we take care of
later version's $TOOL working on older kernel as well as $TOOL from that
older kernel used to work, I'm fine with it.

I realize that multi-side flamefests are messy, but let's keep track of
who's saying what, OK?


From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: [GIT PULL] Performance Counters for Linux
Date: Fri, 12 Jun 2009 03:00:08 UTC
Message-ID: <fa.+mKHmPbR0SlZXxkFMrpoq7AlwEU@ifi.uio.no>

On Fri, 12 Jun 2009, Al Viro wrote:
>
> So could you please clarify the situation?  If the ABI compatibility
> requirements remain the same as they used to be, whether the userland code
> is in-tree or not, I'm fine with the entire thing.  If they do not (and *ONLY*
> in that case), I think we have a real problem.

I think the ABI requirements are the same.

That said, I also suspect that as with oprofile itself, we'll end up
having expansions of the ABI that may well be CPU-specific. I also suspect
that there will probably be breakage early on just because things will
inevitably settle.

And I think that for something like a profiling tool, such breakage is
much more acceptable than for the actual binaries you'd profile. It's not
like we're talking about breaking the boot or functionality of a machine,
as happens when we break the X server (which has happened).

		Linus


Index Home About Blog