Index
Home
About
Blog
Newsgroups: fa.linux.kernel
From: Christoph Hellwig <hch@lst.de>
Subject: silent semantic changes with reiser4
Original-Message-ID: <20040824202521.GA26705@lst.de>
Date: Tue, 24 Aug 2004 20:26:47 GMT
Message-ID: <fa.e31rl4a.rjq2ac@ifi.uio.no>
After looking trough the code and mailinglists I'm quite unhappy with
a bunch of user-visible changes that Hans sneaked in and make reiser4
incompatible with other filesystems and have a slight potential to break
even in the kernel.
o files as directories
- O_DIRECTORY opens succeed on all files on reiser4. Besides breaking
.htaccess handling in apache and glibc compilation this also renders
this flag entirely useless and opens up the races it tries to
prevent against cmpletely useless
- meaning of the -x permission. This one has different meanings on
directories vs files on UNIX systems. If we want to support
directories as files we'll probably have to find a way to work
around this.
- dentry aliasing. I can't find a formal guarantee in the code this
can't happen
o metafiles - ..metas as a magic name that's just taken out of the
namespace doesn't sound like a good idea. If we want this it should
be a VFS-level option and there should be a translation-layer to
xattrs. Not doing this will again confuse applications greatly that
expect uniform filesystem behaviour.
Given these problems I request that these interfaces are removed from
reiser4 for the kernel merge, and if added later at the proper VFS level
after discussion on linux-kernel and linux-fsdevel, like we did for
xattrs.
Newsgroups: fa.linux.kernel
From: viro@parcelfarce.linux.theplanet.co.uk
Subject: Re: silent semantic changes with reiser4
Original-Message-ID: <20040824205343.GE21964@parcelfarce.linux.theplanet.co.uk>
Date: Tue, 24 Aug 2004 20:57:02 GMT
Message-ID: <fa.n5ctbir.1hhuojf@ifi.uio.no>
On Tue, Aug 24, 2004 at 04:38:25PM -0400, Jeff Garzik wrote:
> Christoph Hellwig wrote:
> > o files as directories
> > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking
> > .htaccess handling in apache and glibc compilation this also renders
> > this flag entirely useless and opens up the races it tries to
> > prevent against cmpletely useless
>
>
> Ouch.
>
> I would definitely classify this as a security hole, since userland
> definitely uses O_DIRECTORY to avoid races.
Feh. That's far from the worst parts of the mess introduced by "hybrid"
crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit
higher on the suckitude scale, IMO.
Newsgroups: fa.linux.kernel
From: viro@parcelfarce.linux.theplanet.co.uk
Subject: Re: silent semantic changes with reiser4
Original-Message-ID: <20040824212232.GF21964@parcelfarce.linux.theplanet.co.uk>
Date: Tue, 24 Aug 2004 21:25:33 GMT
Message-ID: <fa.n5ctaqq.1jhupbe@ifi.uio.no>
On Tue, Aug 24, 2004 at 09:53:44PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> Feh. That's far from the worst parts of the mess introduced by "hybrid"
> crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit
> higher on the suckitude scale, IMO.
While we are at it - consider these hybrids vetoed until
a) sys_link()/sys_link() deadlock is fixed
b) sys_link()/sys_rename() deadlock is fixed
c) correctness proof of the locking scheme (in
Documentation/filesystems/directory-locking) is updated to match the
presense of the file/directory hybrids.
Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince
us that there's no more crap of that kind". IMO a reasonable request, seeing
that the first look at the patches in -mm4 had turned up two exploits in
that area, despite the *YEARS* of warnings about potential trouble and need
to be careful there (actually, I've given Hans too much credit and assumed
that link/link never happens since nobody would be dumb enough to provide
->link() method for non-directory inodes; turns out that somebody is dumb
enough and link/link is as exploitable as link/rename).
Newsgroups: fa.linux.kernel
From: Linus Torvalds <torvalds@osdl.org>
Subject: Re: The argument for fs assistance in handling archives
Original-Message-ID: <Pine.LNX.4.58.0409020030220.2295@ppc970.osdl.org>
Date: Thu, 2 Sep 2004 07:39:14 GMT
Message-ID: <fa.gstpvu3.bi4oph@ifi.uio.no>
On Thu, 2 Sep 2004, Hans Reiser wrote:
> Linus Torvalds wrote:
> >
> >. Doing transactions on one file is
> >only the beginning - you'll find people who want transactions across file
> >boundaries etc.
>
> Yup, that's what reiser4 is aiming at and where things get exciting for
> version control systems and the like. We are just missing the api code
> for it at the moment, all the infrastructure is there, and the api is in
> progress in sys_reiser4.
But _my_ point is, no user program is going to take _advantage_ of
anything that only one filesystem on one system offers.
So there's no point.
It's much saner (from _any_ app standpoint) to roll their own database in
user space - that way it just works.
In other words, nobody is really ever going to take advantage of this.
This is _not_ how technical advanncement happens. The way you get people
to take advantage of something is to have a nice gradual ramp-up, not a
sudden new feature that they can't realistically use.
So before you do transactions in the filesystem, you have to be able to do
them in user space - and then make the filesystem version be _compatible_
with that user space library. That way you can get people to use the
library ("hey, it works anywhere") and then you can get them to use your
filesystem ("hey, if you use our super-duper filesystem, you can do what
you are already doing five times faster").
See? You're starting at the wrong end. Give me a portable interface to use
_first_. Then do transactions in the filesystem.
(Now, as an academic exercise and as a learning experience and prototyping
work it might make sense to do the filesystem first. But then you have to
really _consider_ it a prototype, and realize that to make it useful you'd
still have to do the user thing before you can "sell" people on the idea).
Linus
Newsgroups: fa.linux.kernel
From: Linus Torvalds <torvalds@osdl.org>
Subject: Re: The argument for fs assistance in handling archives
Original-Message-ID: <Pine.LNX.4.58.0409021041440.2295@ppc970.osdl.org>
Date: Thu, 2 Sep 2004 17:48:05 GMT
Message-ID: <fa.gudu064.828p1g@ifi.uio.no>
On Thu, 2 Sep 2004, Hans Reiser wrote:
>
> Apple does not have this problem....
Deep, Hans. Deep.
You're ignoring the issue. You're flaming about not "embracing progress",
when I was talking about how progress happens. You don't seem to
understand _how_ people embrace progress, Hans. Railing at windmills isn't
real smart.
Linus
Newsgroups: fa.linux.kernel
From: Linus Torvalds <torvalds@osdl.org>
Subject: Re: The argument for fs assistance in handling archives
Original-Message-ID: <Pine.LNX.4.58.0409021048340.2295@ppc970.osdl.org>
Date: Thu, 2 Sep 2004 18:02:56 GMT
Message-ID: <fa.gttu06b.bi8p1p@ifi.uio.no>
On Thu, 2 Sep 2004, James Bruce wrote:
> Linus Torvalds wrote:
>
> >In other words, nobody is really ever going to take advantage of this.
> >This is _not_ how technical advanncement happens. The way you get people
> >to take advantage of something is to have a nice gradual ramp-up, not a
> >sudden new feature that they can't realistically use.
>
> Sure, but there are plenty of existing interfaces that you could
> emulate.
Absolutely. We should look at them, and whether they solve any issues.
I'm not saying that Hans would have to make up a new interface. Quite the
reverse. One interface I suggested to use for attributes (and everybody
hates ;) was the existing one from Solaris. Similarly, there are probably
perfectly fine interfaces for the issue of transactions..
Linus
Newsgroups: fa.linux.kernel
From: Linus Torvalds <torvalds@osdl.org>
Subject: Re: The argument for fs assistance in handling archives
Original-Message-ID: <Pine.LNX.4.58.0409021717220.2295@ppc970.osdl.org>
Date: Fri, 3 Sep 2004 00:29:25 GMT
Message-ID: <fa.gte7vea.b2ao9m@ifi.uio.no>
On Thu, 2 Sep 2004, David Masover wrote:
>
> reiser4 kernel will contain knowledge of fs type contained in a file.
That's a disaster, btw.
There is no one "fs type" of a file. Files have at _least_ one type
(bytestream), but most have more. Which is why automatically doing the
right thing (in the sense you seem to want) in kernel space is simply not
possible.
Linus
From: Theodore Ts'o <tytso@mit.edu>
Newsgroups: fa.linux.kernel
Subject: Re: reiser4 plugins
Date: Fri, 24 Jun 2005 23:12:30 UTC
Message-ID: <fa.d8odcmh.1u56sbb@ifi.uio.no>
Original-Message-ID: <20050624230644.GA20185@thunk.org>
On Fri, Jun 24, 2005 at 12:21:18PM -0700, Hans Reiser wrote:
> There is an area where we suffered from writing fsck last. When there
> are two leaf nodes with the same key range AND the bitmap cannot be
> trusted to tell us which is the valid one, we don't know which is the
> most recent, and pick arbitrarily. Also, if you store a backup of V3,
> and you don't compress it, and you wipe out the bitmap blocks and need
> to use fsck, we don't know what blocks are backup image and what blocks
> are the fs. We advise users to never store a V3 backup on V3 without
> compressing it.
Unfortunately, there are plenty of reasons why you might want to store
a filesystem image on disk besides for backup purposes:
* Regression tests --- I have some 70+ small filesystem images
used for e2fsck's regression test suite. (I am always
amazed how many filesystem fsck programs don't have
regression test suites.)
* Initial ram-disk images
* Image files for qemu or user-mode-linux
... and probably many more. None of these are safe to store on a
reiserfs3 filesystem if you're worried about fsck being robust after a
disk failure.
Funny thing. When I tell system administrators who have been around
the block more than a few times about this particular "feature" of
reiserfs3, they usually very quickly decide that it's time to switch
to another filesystem.....
- Ted
From: Theodore Ts'o <tytso@mit.edu>
Newsgroups: fa.linux.kernel
Subject: Re: reiser4 plugins
Date: Mon, 27 Jun 2005 21:35:17 UTC
Message-ID: <fa.d48heur.1olqt31@ifi.uio.no>
Original-Message-ID: <20050627212628.GB27805@thunk.org>
On Mon, Jun 27, 2005 at 12:46:23PM -0700, Hans Reiser wrote:
> A difference between us is
> that I tell them that with all the major linux filesystems (I include
> XFS and JFS in this) it is by this time far more likely to be hardware
> that caused corruption than the filesystem software, whereas I guess you
> tell them something else.
Oh, I agree with this, and I do tell people that. The question though
is how the filesystem recovers from said hardware-caused corruption
once it does happen. You've admitted that reiserfs3 has less than
optimal recovery characteristics from hardware-induced corruptions if
said filesystem contains reiserfs filesystem images; that would be an
example of a filesystem not being as robust as it could be. (It'll be
interesting to see if SuSE will support reiserfsv3 in combination with
the Xen hypervisor or other virtualization systems, which makes use of
filesystem images.) Another example would be DMA'ing garbage into the
hard drive after a power failure --- how does a filesystem respond to
this eventuality?
You probably hear more stories people who got unlucky with
hardware-induced corruptions with ext2/3, and I probably hear more
from users who have sworn off of reiserfs just because are sample sets
are somewhat biased. Such are the dangers of relying on anecdotal
evidence.
However, logically speaking, if a filesystem is designed such that in
certain cases, the fsck program has to brute-force search every single
disk block looking for data structures that _look_ like they might be
part of the filesystem data, well, that's always going to be more
error prone than one where the filesystem metadata is in
easily-predicted locations. It sounds like you've added some more
checks in reiser4, and that's definitely a good thing. Time will tell
whether they are sufficient or not.
Regards,
- Ted
From: Theodore Tso <tytso@mit.edu>
Newsgroups: fa.linux.kernel
Subject: Re: reiserFS?
Date: Sun, 16 Jul 2006 16:57:18 UTC
Message-ID: <fa.u0wkW+sp8RA8YR/WvJz7Pgqdt1c@ifi.uio.no>
Original-Message-ID: <20060716165648.GB6643@thunk.org>
On Sun, Jul 16, 2006 at 06:28:31PM +0200, Christian Trefzer wrote:
> I don't quite understand. You are supposed to dd_rescue the whole block
> device to a working drive and use fsck on the copy. Whatever is lost in
> the process must of course be restored from a recent backup. But, as a
> friend of mine put it recently, people don't need backup, they only need
> restore ; )
If the disk is known to be bad, yes, and the number of bad blocks is
growing. On the other hand, disks can and will have a few bad blocks,
or bad writes that don't mean the disk is going bad, and a modern
filesystem should be robust enough that a single failed sector doesn't
cause the filesystem to go completely kaput.
In fact, one of the scary trends with hard drives is that size is
continuing to grow exponentially, access times linearly (more or less),
and error rates (errors per kilobytes per unit time) are remaining
more or less constant.
The fact that reiserfs uses a single B-tree to store all of its data
means that very entertaining things can happen if you lose a sector
containing a high-level node in the tree. It's even more entertaining
if you have image files (like initrd files) in reiserfs format stored
in reiserfs, and you run the recovery program on the filesystem.....
Yes, I know that reiserfs4 is alleged to fix this problem, but as far
as I know it is still using a single unitary tree, with all of the
pitfalls that this entails.
Now, that being said, that by itself is not a reason not to decide not
to include reiserfs4 into the mainline sources. (I might privately
get amused when system administrators use reiserfs and then report
massive data loss, but that's my own failure of charity; I'm working
on it.) For the technical reasons why resierfs4 hasn't been
integrated, please see the mailing list archives.
- Ted
From: Theodore Tso <tytso@mit.edu>
Newsgroups: fa.linux.kernel
Subject: Re: reiserFS?
Date: Sun, 16 Jul 2006 17:48:19 UTC
Message-ID: <fa.ijq98TN4wD5n1SH7SawlX8jHm+8@ifi.uio.no>
Original-Message-ID: <20060716174804.GA23114@thunk.org>
On Sun, Jul 16, 2006 at 06:26:03PM +0100, Lexington Luthor wrote:
> I read the archives, and most of the problems pointed out during the
> review were fixed relatively quickly, followed by a flame war due to
> some suggesting that reiser4 should not be able to affect VFS semantics,
> and other such matters (which IMO should be outside of the scope of a
> code review). There has been no follow-up review as far as I can tell.
As far as I know not all of the problems were fixed. And it has been
observed that given the abuse and accusations that were directed at
the people who did decide to review it, that it would not at all
surprising if some (all?) of reviewers may have decided they had
better things to do. Getting things merged into mainline is not a
right, and the reviewers are volunteers.....
Speaking for myself, since I don't enjoy being accused of partisanship
and being ascribed of having a desire to backstab reiserfs, I have a
personal policy to avoid reiserfs review, and recuse myself from any
votes within program committee discussions regarding Hans Reiser.
Being accused of taking unfair advantage of my volunteer activities is
something I allow myself to get into once.
- Ted
From: Theodore Tso <tytso@mit.edu>
Newsgroups: fa.linux.kernel
Subject: Re: reiser4: maybe just fix bugs?
Date: Thu, 03 Aug 2006 07:47:37 UTC
Message-ID: <fa.GeypwuHTKSnTqI1j7fOdS5MoQ6Q@ifi.uio.no>
Original-Message-ID: <20060803074651.GA27835@thunk.org>
On Tue, Aug 01, 2006 at 11:55:57AM -0500, David Masover wrote:
>
> If I understand it right, the original Reiser4 model of file metadata is
> the file-as-directory stuff that caused such a furor the last big push
> for inclusion (search for "Silent semantic changes in Reiser4"):
The furor was caused by concerns Al Viro expressed about
locking/deadlock issues that reiser4 introduced.
The bigger issue with xattr support is two-fold. First of all, there
are the progams that are expecting the existing extended attribute
interface, and not implementing it will simply mean that as far as
Samba, and other applications, Reiser4 simply will not have xattr
support.
More importantly are the system-level extended attributes, such as
those used by SELINUX, which by definition are not supposed to be
visible to the user at all, but only are supposed to be there for the
SELINUX (or some other kernel layer, in general) to access.
Not supporting xattrs means that those distro's that use SELINUX by
default (i.e., RHEL, Fedora, etc.) won't want to use reiser4, because
SELINUX won't work on reiser4 filesytstems.
Whether or not Hans cares about this is up to him....
- Ted
From: Theodore Tso <tytso@mit.edu>
Newsgroups: fa.linux.kernel
Subject: Re: Reiser4. BEST FILESYSTEM EVER.
Date: Sun, 08 Apr 2007 17:03:59 UTC
Message-ID: <fa.v84nT5EA3JIzeY7Nn255zbHobJQ@ifi.uio.no>
The reason why I ignore the tar+gzip tests is that in the past Hans
has rigged the test by using a tar ball which was generated by
unpacking a set of kernel sources on a reiser4 filesystem, and then
repacking them using tar+gzip. The result was a tar file whose files
were optimally laid out so that reiser4 could insert them into the
filesystem b-tree without doing any extra work.
I can't say for sure whether or not this set of benchmarks has done
this (there's not enough information describing the benchmark setup),
but the sad fact of the matter is that people trying to pitch Reiser4
have generated for themselves a reputation for using rigged
benchmarks. Hans's used of a carefully stacked and ordered tar file
(which is the same as stacking a deck of cards), and your repeated use
of the bonnee++ benchmarks despite being told that it is a meaningless
result given the fact that well, zero's compress very well and most
people are interested in storing a file of all zeros, has caused me to
look at any benchmarks cited by Reiser4 partisans with a very
jaundiced and skeptical eye.
Fortunately for you, it's not up to me whether or not Reiser4 makes it
into the kernel. And if it works for you, hey, go wild. You can
always patch it into your own kernel and encourage others to do the
same with respect to getting it tested and adopted. My personal take
on it is that Reiser3, Reiser4 and JFS suffer the same problems, which
is to say they have a very small and limited development community,
and this was referenced in Novell's decision to drop Reiser3:
http://linux.wordpress.com/2006/09/27/suse-102-ditching-reiserfs-as-it-default-fs/
SuSE has deprecated Reiser3 *and* JFS, and I believe quite strongly it
is the failure of the organizations to attract a diverse development
community is ultimately what doomed them in the long term, both in
terms of support as the kernel migrated and new feature support. It
is for that reason that Hans' personality traits that tend to drive
away those developers who would help them, beyond those that he hires,
is what has been so self-destructive to Reiser4. Read the
announcement Jeff Mahoney from SUSE Labs again; he pointed out was
that reiser3 was getting dropped even though it performs better than
ext3 in some scenarios. There are many other considerations, such as
a filesystem's robustness in case on-disk corruption, long term
maintenance as the kernel maintains, availability of developers to
provide bug fixes, how well the system performs on systems with
multiple cores/CPU's, etc.
- Ted
From: Theodore Tso <tytso@mit.edu>
Newsgroups: fa.linux.kernel
Subject: Re: Question about Reiser4
Date: Mon, 23 Apr 2007 22:57:15 UTC
Message-ID: <fa.Ag3QiJKX3hM+O2hC57qjORzf2hM@ifi.uio.no>
On Mon, Apr 23, 2007 at 06:52:16AM -0700, Eric Hopper wrote:
> Oh, two things really interest me about Reiser4. First, I despise
> having to care about how many tiny files I leave lying around when
> writing a program. Berkeley DB and its ilk are evil, evil programs that
> obscure data and make things harder. Secondly, the moves Reiser4 has
> made towards having actual transactions at the filesystem level also
> intrigue me.
>
> I want to use the filesystem as a DB. IMHO, there is no reason that
> filesystems shouldn't be a DB sans query language. If there were a more
> DB-like way to deal with filesystems, I think that it would be that much
> easier to make something that was a decent replacement for NFS and
> actually worked.
One of the big problems of using a filesystem as a DB is the system
call overheads. If you use huge numbers of tiny files, then each
attempt read an atom of information from the DB takes three system
calls --- an open(), read(), and close(), with all of the overheads in
terms of dentry and inode cache.
Hans of course had a solution to this problem --- namely the
sys_reiser4 system call, where you download a program to the kernel to
execute a open/read/close via a single system call, and which returns
the combined results to userspace. But now you have more complexity
since there is now a reiser4-specific interpreter embeddeed in the
kernel, the userspace application needs to write the equivalent of an
channel program such as what was found in an IBM/360 mainframe (need I
mention this can be a rich source of security bugs), and then the
userspace application *still* needs to parse the result returned by
the sys_reiser4() system call?
So it adds a huge amount of complexity, and at the end of the day,
given that you don't have the search capability, it is (a) less
functional, (b) more complexitated, and (c) probably less performant
than simply calling out to a database.
> Sadly, unless someone pays me to maintain it, I can't do the fork
> myself, and I likely wouldn't anyway as being a kernel hacker of
> something as important as a filesystem is a full-time job and I have
> other things that interest me a lot more.
Unfortunately, the way OSS works is that you either (a) have to do the
work yourself, (b) convince someone else to do the work, or (c)
convince someone that it's worth paying you to do it.
Personally, if I controlled large budget for Linux filesystem
development, I'd put a lot more money into something like Val's
chunkfs idea than resier4. Being able to have filesystems designed
for fast recovery given disks getting larger and larger (but not more
reliable), is a whole lot more improtant than trying to create an
alternate solution to an already solved problem --- namely that of a
database. When you consider that a similar idea, WinFS, was partially
responsible for delaying Vista by years due to the complexity of
shoving a database where it has no place being, it's another reason
why I personally think that chunkfs is a much more promising avenue
for future filesystem investment than reiserfs.
But hey, the advantage of Open Source is that if *you* want to work on
Reiser4, you're perfectly free to do so. My personal opinion is that
it'd be a waste of your time, but you're free to spend your time
whichever way that you want. What you don't get do is whine about how
other people get to spend *their* time, or *their* money.
- Ted
From: Theodore Tso <tytso@mit.edu>
Newsgroups: fa.linux.kernel
Subject: Re: Question about Reiser4
Date: Tue, 24 Apr 2007 00:20:19 UTC
Message-ID: <fa.SuGZMEZGMWm3fgkVYkHS9I2utd8@ifi.uio.no>
On Mon, Apr 23, 2007 at 04:53:03PM -0700, H. Peter Anvin wrote:
> Theodore Tso wrote:
> >
> >One of the big problems of using a filesystem as a DB is the system
> >call overheads. If you use huge numbers of tiny files, then each
> >attempt read an atom of information from the DB takes three system
> >calls --- an open(), read(), and close(), with all of the overheads in
> >terms of dentry and inode cache.
> >
>
> Now, to be fair, there are probably a number of cases where
> open/lseek/readv/close and open/lseek/writev/close would be worth doing
> as a single system call. The big problem as far as I can see involves
> EINTR handling; such a system call has serious restartability implications.
Sure, but Hans wants to change /etc/inetd.conf into /etc/inetd.conf.d,
where you have: /etc/inetd.conf.d/telnet/port,
/etc/inetd.conf.d/telnet/protocol, /etc/inetd.conf.d/telnet/wait,
/etc/inetd.conf.d/telnet/userid, /etc/inetd.conf.d/telnet/daemon,
etc. for each individual line in /etc/inetd.conf. (And where each
file might only contains 2-4 characters each: i.e., "23", "tcp",
"root", etc.)
So it's not enough just to collapse open/pread/close into a single
system call; in order to gain back the performance squandered by all
of these itsy-bitsy tiny little files. You want to collapse the
open/pread/close for many of these little files into a single system
call, hence Hans's insistence on sys_reiser4(); otherwise his scheme
doesn't work all that well at all.
- Ted
From: Theodore Tso <tytso@mit.edu>
Newsgroups: fa.linux.kernel
Subject: Re: Question about Reiser4
Date: Tue, 24 Apr 2007 01:18:58 UTC
Message-ID: <fa.6PS97ien32s1QE569ScjXqfP4UI@ifi.uio.no>
On Mon, Apr 23, 2007 at 05:31:29PM -0700, H. Peter Anvin wrote:
> Heh. sys_read_tree() -- walk a directory tree and return it as a data
> structure in memory :)
But maybe you don't want every single file in the directory, but some
subset of the files in the directory tree. So before you know it:
sys_fs_sql("SELECT port,userid,daemon FROM /etc/inetd.conf.d "
"WHERE protocol=='tcp'", buf, sizeof(buf));
The question is where do you stop on the slippery slope, and is it
really all that harder than simply parsing a /etc/gitconfig or
/etc/e2fsck.conf file. There are plenty of parsers or database
libraries already written, and many of them are quite efficient. And
personally, I'd much rather edit a single /etc/gitconfig or
/etc/e2fsck.conf file using emacs than have to cd through 3 or 4
levels of directories to edit each 2-3 byte file one at a time. But
to each their own....
- Ted
Index
Home
About
Blog