Commit graph

43 commits

Author SHA1 Message Date
David Chinner
2a82b8be8a [XFS] Concurrent Multi-File Data Streams
In media spaces, video is often stored in a frame-per-file format. When
dealing with uncompressed realtime HD video streams in this format, it is
crucial that files do not get fragmented and that multiple files a placed
contiguously on disk.

When multiple streams are being ingested and played out at the same time,
it is critical that the filesystem does not cross the streams and
interleave them together as this creates seek and readahead cache miss
latency and prevents both ingest and playout from meeting frame rate
targets.

This patch set creates a "stream of files" concept into the allocator to
place all the data from a single stream contiguously on disk so that RAID
array readahead can be used effectively. Each additional stream gets
placed in different allocation groups within the filesystem, thereby
ensuring that we don't cross any streams. When an AG fills up, we select a
new AG for the stream that is not in use.

The core of the functionality is the stream tracking - each inode that we
create in a directory needs to be associated with the directories' stream.
Hence every time we create a file, we look up the directories' stream
object and associate the new file with that object.

Once we have a stream object for a file, we use the AG that the stream
object point to for allocations. If we can't allocate in that AG (e.g. it
is full) we move the entire stream to another AG. Other inodes in the same
stream are moved to the new AG on their next allocation (i.e. lazy
update).

Stream objects are kept in a cache and hold a reference on the inode.
Hence the inode cannot be reclaimed while there is an outstanding stream
reference. This means that on unlink we need to remove the stream
association and we also need to flush all the associations on certain
events that want to reclaim all unreferenced inodes (e.g. filesystem
freeze).

SGI-PV: 964469
SGI-Modid: xfs-linux-melb:xfs-kern:29096a

Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Barry Naujok <bnaujok@sgi.com>
Signed-off-by: Donald Douwsma <donaldd@sgi.com>
Signed-off-by: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Tim Shimmin <tes@sgi.com>
Signed-off-by: Vlad Apostolov <vapo@sgi.com>
2007-07-14 15:40:53 +10:00
David Chinner
92821e2ba4 [XFS] Lazy Superblock Counters
When we have a couple of hundred transactions on the fly at once, they all
typically modify the on disk superblock in some way.
create/unclink/mkdir/rmdir modify inode counts, allocation/freeing modify
free block counts.

When these counts are modified in a transaction, they must eventually lock
the superblock buffer and apply the mods. The buffer then remains locked
until the transaction is committed into the incore log buffer. The result
of this is that with enough transactions on the fly the incore superblock
buffer becomes a bottleneck.

The result of contention on the incore superblock buffer is that
transaction rates fall - the more pressure that is put on the superblock
buffer, the slower things go.

The key to removing the contention is to not require the superblock fields
in question to be locked. We do that by not marking the superblock dirty
in the transaction. IOWs, we modify the incore superblock but do not
modify the cached superblock buffer. In short, we do not log superblock
modifications to critical fields in the superblock on every transaction.
In fact we only do it just before we write the superblock to disk every
sync period or just before unmount.

This creates an interesting problem - if we don't log or write out the
fields in every transaction, then how do the values get recovered after a
crash? the answer is simple - we keep enough duplicate, logged information
in other structures that we can reconstruct the correct count after log
recovery has been performed.

It is the AGF and AGI structures that contain the duplicate information;
after recovery, we walk every AGI and AGF and sum their individual
counters to get the correct value, and we do a transaction into the log to
correct them. An optimisation of this is that if we have a clean unmount
record, we know the value in the superblock is correct, so we can avoid
the summation walk under normal conditions and so mount/recovery times do
not change under normal operation.

One wrinkle that was discovered during development was that the blocks
used in the freespace btrees are never accounted for in the AGF counters.
This was once a valid optimisation to make; when the filesystem is full,
the free space btrees are empty and consume no space. Hence when it
matters, the "accounting" is correct. But that means the when we do the
AGF summations, we would not have a correct count and xfs_check would
complain. Hence a new counter was added to track the number of blocks used
by the free space btrees. This is an *on-disk format change*.

As a result of this, lazy superblock counters are a mkfs option and at the
moment on linux there is no way to convert an old filesystem. This is
possible - xfs_db can be used to twiddle the right bits and then
xfs_repair will do the format conversion for you. Similarly, you can
convert backwards as well. At some point we'll add functionality to
xfs_admin to do the bit twiddling easily....

SGI-PV: 964999
SGI-Modid: xfs-linux-melb:xfs-kern:28652a

Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Tim Shimmin <tes@sgi.com>
2007-07-14 15:28:50 +10:00
Nathan Scott
4cc929ee30 [XFS] Don't grow filesystems past the size they can index.
When growing a filesystem we don't check to see if the new size overflows
the page cache index range, so we can do silly things like grow a
filesystem page 16TB on a 32bit. Check new filesystem sizes against the
limits the kernel can support.

SGI-PV: 957886
SGI-Modid: xfs-linux-melb:xfs-kern:28563a

Signed-Off-By: Nathan Scott <nscott@aconex.com>
Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Tim Shimmin <tes@sgi.com>
2007-07-14 15:21:29 +10:00
Lachlan McIlroy
5478eead85 [XFS] Re-initialize the per-cpu superblock counters after recovery.
After filesystem recovery the superblock is re-read to bring in any
changes. If the per-cpu superblock counters are not re-initialized from
the superblock then the next time the per-cpu counters are disabled they
might overwrite the global counter with a bogus value.

SGI-PV: 957348
SGI-Modid: xfs-linux-melb:xfs-kern:27999a

Signed-off-by: Lachlan McIlroy <lachlan@sgi.com>
Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Tim Shimmin <tes@sgi.com>
2007-02-10 18:36:29 +11:00
Kevin Jamieson
c97be73605 [XFS] Fix block reservation changes for non-SMP systems.
SGI-PV: 956323
SGI-Modid: xfs-linux-melb:xfs-kern:27940a

Signed-off-by: Kevin Jamieson <kjamieson@bycast.com>
Signed-off-by: David Chatterton <chatz@sgi.com>
Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Tim Shimmin <tes@sgi.com>
2007-02-10 18:36:23 +11:00
David Chinner
dbcabad19a [XFS] Fix block reservation mechanism.
The block reservation mechanism has been broken since the per-cpu
superblock counters were introduced. Make the block reservation code work
with the per-cpu counters by syncing the counters, snapshotting the amount
of available space and then doing a modifcation of the counter state
according to the result. Continue in a loop until we either have no space
available or we reserve some space.

SGI-PV: 956323
SGI-Modid: xfs-linux-melb:xfs-kern:27895a

Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Tim Shimmin <tes@sgi.com>
2007-02-10 18:36:17 +11:00
David Chinner
20f4ebf2bf [XFS] Make growfs work for amounts greater than 2TB
The free block modification code has a 32bit interface, limiting the size
the filesystem can be grown even on 64 bit machines. On 32 bit machines,
there are other 32bit variables in transaction structures and interfaces
that need to be expanded to allow this to work.

SGI-PV: 959978
SGI-Modid: xfs-linux-melb:xfs-kern:27894a

Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Tim Shimmin <tes@sgi.com>
2007-02-10 18:36:10 +11:00
Lachlan McIlroy
1f9b3b64d4 [XFS] remove unused xflags parameter from sync routines
SGI-PV: 959137
SGI-Modid: xfs-linux-melb:xfs-kern:27710a

Signed-off-by: Lachlan McIlroy <lachlan@sgi.com>
Signed-off-by: Eric Sandeen <sandeen@sandeen.net>
Signed-off-by: Tim Shimmin <tes@sgi.com>
2007-02-10 18:35:33 +11:00
David Chinner
03135cf726 [XFS] Fix UP build breakage due to undefined m_icsb_mutex.
SGI-PV: 952227
SGI-Modid: xfs-linux-melb:xfs-kern:27692a

Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Lachlan McIlroy <lachlan@sgi.com>
Signed-off-by: Tim Shimmin <tes@sgi.com>
2007-02-10 18:35:15 +11:00
David Chinner
20b642858b [XFS] Reduction global superblock lock contention near ENOSPC.
The existing per-cpu superblock counter code uses the global superblock
spin lock when we approach ENOSPC for global synchronisation. On larger
machines than this code was originally tested on this can still get
catastrophic spinlock contention due increasing rebalance frequency near
ENOSPC.

By introducing a sleeping lock that is used to serialise balances and
modifications near ENOSPC we prevent contention from needlessly from
wasting the CPU time of potentially hundreds of CPUs.

To reduce the number of balances occuring, we separate the need rebalance
case from the slow allocate case. Now, a counter running dry will trigger
a rebalance during which counters are disabled. Any thread that sees a
disabled counter enters a different path where it waits on the new mutex.
When it gets the new mutex, it checks if the counter is disabled. If the
counter is disabled, then we _know_ that we have to use the global counter
and lock and it is safe to do so immediately. Otherwise, we drop the mutex
and go back to trying the per-cpu counters which we know were re-enabled.

SGI-PV: 952227
SGI-Modid: xfs-linux-melb:xfs-kern:27612a

Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Lachlan McIlroy <lachlan@sgi.com>
Signed-off-by: Tim Shimmin <tes@sgi.com>
2007-02-10 18:35:09 +11:00
Nathan Scott
215101c360 [XFS] Fix kmem_zalloc_greedy warnings on 64 bit platforms.
SGI-PV: 955302
SGI-Modid: xfs-linux-melb:xfs-kern:26907a

Signed-off-by: Nathan Scott <nathans@sgi.com>
Signed-off-by: Tim Shimmin <tes@sgi.com>
2006-09-28 11:04:43 +10:00
Christoph Hellwig
fe48cae9ed [XFS] remove bhv_lookup, _range version works aswell and has more useful
semantics.

SGI-PV: 954580
SGI-Modid: xfs-linux-melb:xfs-kern:26563a

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nathan Scott <nathans@sgi.com>
Signed-off-by: Tim Shimmin <tes@sgi.com>
2006-09-28 10:58:52 +10:00
Nathan Scott
f6c2d1fa63 [XFS] Remove version 1 directory code. Never functioned on Linux, just
pure bloat.

SGI-PV: 952969
SGI-Modid: xfs-linux-melb:xfs-kern:26251a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-06-20 13:04:51 +10:00
Nathan Scott
8285fb58e7 [XFS] Resolve a namespace collision on remaining vtypes for FreeBSD
porters.

SGI-PV: 953338
SGI-Modid: xfs-linux-melb:xfs-kern:26108a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-06-09 17:07:12 +10:00
Nathan Scott
67fcaa73ad [XFS] Resolve a namespace collision on vnode/vnodeops for FreeBSD porters.
SGI-PV: 953338
SGI-Modid: xfs-linux-melb:xfs-kern:26107a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-06-09 17:00:52 +10:00
Nathan Scott
b83bd13881 [XFS] Resolve a namespace collision on vfs/vfsops for FreeBSD porters.
SGI-PV: 9533338
SGI-Modid: xfs-linux-melb:xfs-kern:26106a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-06-09 16:48:30 +10:00
Nathan Scott
7d04a335b6 [XFS] Shutdown the filesystem if all device paths have gone. Made
shutdown vop flags consistent with sync vop flags declarations too.

SGI-PV: 939911
SGI-Modid: xfs-linux-melb:xfs-kern:26096a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-06-09 14:58:38 +10:00
Olaf Weber
3e57ecf640 [XFS] Add parameters to xfs_bmapi() and xfs_bunmapi() to have them report
the range spanned by modifications to the in-core extent map.  Add
XFS_BUNMAPI() and XFS_SWAP_EXTENTS() macros that call xfs_bunmapi() and
xfs_swap_extents() via the ioops vector. Change all calls that may modify
the in-core extent map for the data fork to go through the ioops vector. 
This allows a cache of extent map data to be kept in sync.

SGI-PV: 947615
SGI-Modid: xfs-linux-melb:xfs-kern:209226a

Signed-off-by: Olaf Weber <olaf@sgi.com>
Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-06-09 14:48:12 +10:00
Nathan Scott
764d1f89a5 [XFS] Implement the silent parameter to fill_super, previously ignored.
SGI-PV: 951299
SGI-Modid: xfs-linux-melb:xfs-kern:25632a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-03-31 13:04:17 +10:00
Nathan Scott
c41564b5af [XFS] We really suck at spulling. Thanks to Chris Pascoe for fixing all
these typos.

SGI-PV: 904196
SGI-Modid: xfs-linux-melb:xfs-kern:25539a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-03-29 08:55:14 +10:00
Nathan Scott
b8b0f54656 [XFS] Remove a couple of no-longer-used macros/types from XFS.
SGI-PV: 950556
SGI-Modid: xfs-linux-melb:xfs-kern:25377a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-03-14 13:47:32 +11:00
David Chinner
01e1b69cfc [XFS] using a spinlock per cpu for superblock counter exclusion results in
a preēmpt counter overflow at 256p and above. Change the exclusion
mechanism to use atomic bit operations and busy wait loops to emulate the
spin lock exclusion mechanism but without the preempt count issues.

SGI-PV: 950027
SGI-Modid: xfs-linux-melb:xfs-kern:25338a

Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-03-14 13:29:16 +11:00
David Chinner
e8234a6871 [XFS] Add support for hotplug CPUs to the per-CPU superblock counters by
registering a notifier callback that listens to CPU up/down events to
modify the counters appropriately.

SGI-PV: 949726
SGI-Modid: xfs-linux-melb:xfs-kern:25214a

Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-03-14 13:23:52 +11:00
David Chinner
8d280b98cf [XFS] On machines with more than 8 cpus, when running parallel I/O
threads, the incore superblock lock becomes the limiting factor for
buffered write throughput. Make the contended fields in the incore
superblock use per-cpu counters so that there is no global lock to limit
scalability.

SGI-PV: 946630
SGI-Modid: xfs-linux-melb:xfs-kern:25106a

Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-03-14 13:13:09 +11:00
Nathan Scott
9f4cbecd7e [XFS] XFS propagates MS_NOATIME through two levels internally but doesn't
actually use it.  Kill this dead code.	Signed-off-by: Christoph Hellwig
<hch@lst.de>

SGI-PV: 904196
SGI-Modid: xfs-linux-melb:xfs-kern:25086a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-03-14 13:05:30 +11:00
Nathan Scott
ce8e922c0e [XFS] Complete the pagebuf -> xfs_buf naming convention transition,
finally.

SGI-PV: 947038
SGI-Modid: xfs-linux-melb:xfs-kern:24866a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-01-11 15:39:08 +11:00
Nathan Scott
13059ff04c [XFS] Reverse the sense of COMPAT_ATTR and ATTR2, keeps it simple and
consistent.

SGI-PV: 941645
SGI-Modid: xfs-linux-melb:xfs-kern:202961a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2006-01-11 15:32:01 +11:00
Jes Sorensen
794ee1baee [PATCH] mutex subsystem, semaphore to mutex: XFS
This patch switches XFS over to use the new mutex code directly as
opposed to the previous workaround patch I posted earlier that avoided
the namespace clash by forcing it back to semaphores. This falls in the
'works for me<tm>' category.

Signed-off-by: Jes Sorensen <jes@trained-monkey.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2006-01-09 15:59:21 -08:00
Nathan Scott
c11e2c369d [XFS] Rework fid encode/decode wrt 64 bit inums interacting with NFS.
SGI-PV: 937127
SGI-Modid: xfs-linux:xfs-kern:24201a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2005-11-02 15:11:45 +11:00
Nathan Scott
e718eeb4fe [XFS] Rework the final mount options flag bit to make room for more.
SGI-PV: 943866
SGI-Modid: xfs-linux:xfs-kern:24030a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2005-11-02 15:09:22 +11:00
Nathan Scott
7b71876980 [XFS] Update license/copyright notices to match the prefered SGI
boilerplate.

SGI-PV: 913862
SGI-Modid: xfs-linux:xfs-kern:23903a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2005-11-02 14:58:39 +11:00
Nathan Scott
a844f4510d [XFS] Remove xfs_macros.c, xfs_macros.h, rework headers a whole lot.
SGI-PV: 943122
SGI-Modid: xfs-linux:xfs-kern:23901a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2005-11-02 14:38:42 +11:00
Nathan Scott
fc1f8c1ca3 [XFS] Track external log/realtime device names for correct reporting in
/proc/mounts.

SGI-PV: 942984
SGI-Modid: xfs-linux:xfs-kern:23862a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2005-11-02 11:44:33 +11:00
Nathan Scott
f74dee4276 [XFS] Ondisk format extension for extended attributes (attr2). Basically,
the data/attr forks now grow up/down from either end of the literal area,
rather than dividing the literal area into two chunks and growing both
upward.  Means we can now make much more efficient use of the attribute
space, incl. fitting DMF attributes inline in 256 byte inodes, and large
jumps in dbench3 performance numbers.  It is self enabling, but can be
forced on/off via the attr2/noattr2 mount options.

SGI-PV: 941645
SGI-Modid: xfs-linux:xfs-kern:23837a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2005-11-02 10:35:56 +11:00
David Chinner
e8c8b3a79d [XFS] Introduce two new mount options (nolargeio/largeio) to allow
filesystems to expose the filesystem stripe width in stat(2) rather than
the page cache size. This allows applications requiring high bandwidth to
easily determine the optimum I/O size for the underlying filesystem. The
default is to report the page cache size (i.e. "nolargeio").

SGI-PV: 942818
SGI-Modid: xfs-linux:xfs-kern:23830a

Signed-off-by: David Chinner <dgc@sgi.com>
Signed-off-by: Nathan Scott <nathans@sgi.com>
2005-11-02 10:33:05 +11:00
Nathan Scott
ee34807a65 [XFS] Provide a mechiansm for flushing delalloc before quota reporting.
SGI-PV: 942815
SGI-Modid: xfs-linux:xfs-kern:23829a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2005-11-02 10:32:38 +11:00
Christoph Hellwig
f538d4da8d [XFS] write barrier support Issue all log sync operations as ordered
writes.  In addition flush the disk cache on fsync if the sync cached
operation didn't sync the log to disk (this requires some additional
bookeping in the transaction and log code). If the device doesn't claim to
support barriers, the filesystem has an extern log volume or the trial
superblock write with barriers enabled failed we disable barriers and
print a warning.  We should probably fail the mount completely, but that
could lead to nasty boot failures for the root filesystem.  Not enabled by
default yet, needs more destructive testing first.

SGI-PV: 912426
SGI-Modid: xfs-linux:xfs-kern:198723a

Signed-off-by: Christoph Hellwig <hch@sgi.com>
Signed-off-by: Nathan Scott <nathans@sgi.com>
2005-11-02 10:26:59 +11:00
Nathan Scott
c8ad20ffeb [XFS] Add support for project quota, based on Dan Knappes earlier work.
SGI-PV: 932952
SGI-Modid: xfs-linux:xfs-kern:22805a

Signed-off-by: Nathan Scott <nathans@sgi.com>
2005-06-21 15:38:48 +10:00
Christoph Hellwig
ba0f32d460 [XFS] mark various symbols static Patch from Adrian Bunk
SGI-PV: 936255
SGI-Modid: xfs-linux:xfs-kern:192760a

Signed-off-by: Christoph Hellwig <hch@sgi.com>
Signed-off-by: Nathan Scott <nathans@sgi.com>
2005-06-21 15:36:52 +10:00
Nathan Scott
f403b7f452 [XFS] Cleanup use of loff_t vs xfs_off_t in the core code.
SGI Modid: xfs-linux:xfs-kern:22378a

Signed-off-by: Nathan Scott <nathans@sgi.com>
Signed-off-by: Christoph Hellwig <hch@sgi.com>
2005-05-05 13:33:40 -07:00
Nathan Scott
24e17b5fb9 [XFS] Use the right offset when ensuring a delayed allocate conversion has covered the offset originally requested. Can cause data corruption when multiple processes are performing writeout on different areas of the same file. Quite difficult to hit though.
SGI Modid: xfs-linux:xfs-kern:22377a

Signed-off-by: Nathan Scott <nathans@sgi.com>
Signed-off-by: Christoph Hellwig <hch@sgi.com>
.
2005-05-05 13:33:20 -07:00
Nathan Scott
1f443ad70d [XFS] Allow initial XFS delayed allocation size to be increased beyond 64KB.
SGI Modid: xfs-linux:xfs-kern:22261a

Signed-off-by: Nathan Scott <nathans@sgi.com>
Signed-off-by: Christoph Hellwig <hch@sgi.com>
2005-05-05 13:28:29 -07:00
Linus Torvalds
1da177e4c3 Linux-2.6.12-rc2
Initial git repository build. I'm not bothering with the full history,
even though we have it. We can create a separate "historical" git
archive of that later if we want to, and in the meantime it's about
3.2GB when imported into git - space that would just make the early
git days unnecessarily complicated, when we don't have a lot of good
infrastructure for it.

Let it rip!
2005-04-16 15:20:36 -07:00