There are a number of places where we multiply a dgrp_t with
s_blocks_per_group expecting that we will get a blk64_t. This
requires a cast, or using the convenience function
ext2fs_group_first_block2().
This audit was suggested by Eric Sandeen.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Use ext2fs_[un]mark_block_range2() functions to reduce the CPU
overhead of resizing large file systems by 45%, primarily by
reducing the time spent in fix_uninit_block_bitmaps().
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Add a new debug flag which prints how much time is consumed by the
various parts of resize2fs's processing.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
This caused the free blocks count in the superblock to be incorrect
after resizing a 64-bit file system if the number of free blocks
overflowed a 32-bit value.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Fix a 32-bit overflow bug caused by a missing blk64_t cast which can
cause the block bitmap to get corrupted when doing an off-line resize
of a 64-bit file system.
This problem can be reproduced as follows:
rm -f foo.img; touch foo.img
truncate -s 8T foo.img
mke2fs -F -t ext4 -O 64bit foo.img
e2fsck -f foo.img
truncate -s 21T foo.img
resize2fs foo.img
e2fsck -fy foo.img
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Now that we are reserving all of the bg-specific metadata before we
try to allocate the metadata for the new block groups, we don't have
to temporarily disable the flex_bg feature flag while we allocate the
new metadata blocks --- this allows the newly created block groups to
have a much more optimized layout, instead of fragmenting the inode
table and block/inode bitmaps in sepraate block groups.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
With flex_bg file systems, bg-specific metadata (i.e., bitmaps and the
inode table blocks) can be located in another block group. Hence,
when we grow the number of block group descriptors, we need to check
if we need to relocate metadata blocks not just for the block group
where the bgd blocks are located, but in all block groups.
This change fixes the following test case:
rm -f foo.img; touch foo.img
truncate -s 32G foo.img
mke2fs -F -t ext4 -E resize=12582912 foo.img
e2fsck -f foo.img
truncate -s 256G foo.img
./resize2fs foo.img
e2fsck -fy foo.img
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
For flex_bg file systems, if we need to relocate an allocation bitmap
or inode table, we need to make sure that all metadata blocks have
been reserved, lest we end up overwriting a metadata block belonging
to a different block group.
This change fixes the following test case:
rm -f foo.img; touch foo.img
truncate -s 32G foo.img
mke2fs -F -t ext4 -E resize=12582912 foo.img
e2fsck -f foo.img
truncate -s 64G foo.img
./resize2fs foo.img
e2fsck -fy foo.img
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
This is the first commit to add support for off-line resizing using
flex_bg without the assist of using the resize_inode to reserve gdt
blocks. This functionality has been broken up into separate commits
which are hopefully obviously correct to make them easier to review
for correctness.
In this first step, we break up the for loop at the end of
blocks_to_move() so that we first mark all of the metadata blocks
which don't need to be moved in the reserve_blocks bitmap, and then
try to allocate the metadata blocks are new or which need to moved
second.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
One of these fixes was triggering failures when running:
./test_scripts --valgrind r_move_itable r_inline_xattr r_resize_inode
It should be a false positive, but it fixing this makes it easier to
see real problems.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
If the uninit_bg feature is enabled and the kernel supports
lazy_itable_init, skip zeroing the inode table so that the resize
operation can go much more quickly. Also set the itable_unused fields
so that the first e2fsck after the resize will run faster.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The file system overhead calculation in calculate_minimum_resize_size
was incorrect meta_bg file systems. This caused the minimum size to
underflow for very large file systems, which threw resize2fs into a
loop generally lasted longer than the user's patience.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
calculate_minimum_resize_size() forgot to account s_first_data_block
into minimum filesystem size. Thus in case the size of filesystem was
such that the last group had the minimal size (50 blocks + metadata
overhead), the code in adjust_fs_info() decided the group is unneeded,
removed it, and in some cases the resizing then failed with ENOSPC.
Fix the issue by properly accounting for s_first_data_block in
calculate_minimum_resize_size().
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
The ext2fs_file_acl_block() and ext2fs_set_file_acl_block() needs to
only check i_file_acl_high if the 64-bit flag is set. This is needed
because otherwise we will run into problems on Hurd systems which
actually use that field for h_i_mode_high.
This involves an ABI change since we need to pass ext2_filsys to these
functions. Fortunately these functions were first included in the
1.42-WIP series, so it's OK for us to change them now. (This is why
we have 1.42-WIP releases. :-)
Addresses-Sourceforge-Bug: #3379227
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
This commit fixes a failure when running the commands:
dd if=/dev/zero of=fs bs=1k count=100k; mke2fs fs; resize2fs -Mp fs
We should not try truncating the file system if there is only a single
block group in the file system.
Addresses-Sourceforge-Bug: #3404051
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The DEFS line in MCONFIG had gotten so long that it exceeded 4k, and
this was starting to cause some tools heartburn. It also made "make
V=1" almost useless, since trying to following the individual commands
run by make was lost in the noise of all of the defines.
So fix this by putting the configure-generated defines in lib/config.h
and the directory pathnames to lib/dirpaths.h.
In addition, clean up some vestigal defines in configure.in and in the
Makefiles to further shorten the cc command lines.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Code to count the number of blocks in the last partial
group is cut and pasted around the e2fsprogs codebase
a few times.
Making this a helper function should improve matters.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
The write_journal_inode() code is only setting the low 32-bit i_size
for the journal size, even though it is possible to specify a journal
up to 10M blocks in size. Trying to create a journal larger than 2GB
will succeed, but an immediate e2fsck would fail. Store i_size_high
for the journal inode when creating it, and load it upon access.
Use s_jnl_blocks[15] to store the journal i_size_high backup. This
field is currently unused, as EXT2_N_BLOCKS is 15, so it is using
s_jnl_blocks[0..14], and i_size is in s_jnl_blocks[16].
Rename the "size" argument "num_blocks" for the journal creation functions
to clarify this parameter is in units of filesystem blocks and not bytes.
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
I ran into odd behavior where mkfs.ext4 of a 16T filesystem would
create a resize inode with 0 reserved blocks, and mark the resize_inode
feature.
A subsequent slight downward resize of the filesystem would remove
the resize inode, making any further offline resizing impossible.
This is especially odd in light of the fact that a large downward
resize (say, to 8T) will actually add blocks to the resize inode -
so a small resize removes it, a large resize expands it ...
commit 8ade268cf2 had added this:
If the filesystem is grown to the point where the resize_inode is no
longer needed, clean it up properly so e2fsck doesn't have to.
but, it seems e2fsck does not care about this situation, either.
So, simply leave the resize_inode intact in this case, and everything
seems to be happy.
Note, this is for the 1.41.xx branch.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Commit 74128f8 added tests for uninit groups, but it could access past
the end of the group_desc[] array after processing the last group:
==19668== Invalid read of size 2
==19668== at 0x40518C: resize_fs (resize2fs.c:1824)
==19668== by 0x405A46: main (main.c:451)
==19668== Address 0x5a0d002 is not stack'd, malloc'd or (recently) free'd
==19668==
==19668== Invalid read of size 2
==19668== at 0x405391: resize_fs (resize2fs.c:1864)
==19668== by 0x405A46: main (main.c:451)
==19668== Address 0x5a0d002 is not stack'd, malloc'd or (recently) free'd
==19668==
It was found by Eric Sandeen running the regression suite through
valgrind.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Resizing a filesystem with an external journal fails when it tries
to read inode 0:
# touch testfs
# truncate testfs 1342177280
# touch testjournal
# truncate testjournal 134217728
# mke2fs -O journal_dev testjournal
# losetup /dev/loop0 testjournal
# mkfs.ext4 -J device=/dev/loop0 testfs 127680
# resize2fs testfs
resize2fs 1.41.9 (22-Aug-2009)
Resizing the filesystem on testfs to 327680 (4k) blocks.
resize2fs: Illegal inode number while trying to resize testfs
Please run 'e2fsck -fy testfs' to fix the filesystem
after the aborted resize operation.
I think the right, simple thing to do is just bail out early
for an external journal here, as there are no backup blocks
to update.
Reported-by: mjevans1983@gmail.com
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
After cleaning up ext2fs_bg_flag_set() and ext2fs_bg_flag_clear(),
we're left with ext2fs_bg_flag_test(). Convert it to
ext2fs_bg_flags_test().
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The ext2fs_bg_flag* functions were confusing.
Currently we have this:
void ext2fs_bg_flags_set(ext2_filsys fs, dgrp_t group, __u16 bg_flags);
void ext2fs_bg_flags_clear(ext2_filsys fs, dgrp_t group,__u16 bg_flags);
(_set (unused) sets exactly bg_flags; _clear clears all and ignores bg_flags)
and these, which can twiddle individual bits in bg_flags:
void ext2fs_bg_flag_set(ext2_filsys fs, dgrp_t group, __u16 bg_flag);
void ext2fs_bg_flag_clear(ext2_filsys fs, dgrp_t group, __u16 bg_flag);
A better interface, after the patch below, is just:
ext2fs_bg_flags_zap(fs, group) /* zeros bg_flags */
ext2fs_bg_flags_set(fs, group, flags) /* adds flags to bg_flags */
ext2fs_bg_flags_clear(fs, group, flags) /* clears flags in bg_flags */
and remove the original ext2fs_bg_flags_set / ext2fs_bg_flags_clear.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
When flex_bg is on, calculate_minimum_resize_size() should add more meta
blocks for newly added flex_bg.
Addresses-RedHat-Bugzilla: #519131
Signed-off-by: Peng Tao <bergwolf@gmail.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
If the resize operation fails in the middle of the operation, mark the
filesystem as needing to be checked, and tell the user that they
should run e2fsck -fy on the device.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
This was reported in Fedora, since the livecd creator does
a lot of resizing.
If we've moved the journal blocks during resize (more likely now,
due to the journal being in the middle) the backup blocks in the
superblock don't get updated, and a subsequent e2fsck will find
issues:
e2fsck 1.41.6 (30-May-2009)
Backing up journal inode block information.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/mnt/test/img: ***** FILE SYSTEM WAS MODIFIED *****
/mnt/test/img: 11/16000 files (0.0% non-contiguous), 17789/38400 blocks
This can be shown in a simple test:
# dd if=/dev/zero of=img bs=1 count=0 seek=3000M
# mke2fs -t ext4 -F img
# resize2fs img 150M
# e2fsck -f img
(thanks to the Fedora reporter Mads Kiilerich for the testcase!
https://bugzilla.redhat.com/show_bug.cgi?id=506105#c2)
So, update the backup journal in the superblock before resize2fs exits.
Addresses-RedHat-Bugzilla: #505339
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The extra padding added to the minimum size calculations:
/*
* We need to reserve a few extra blocks if extents are
* enabled, in case we need to grow the extent tree. The more
* we shrink the file system, the more space we need.
*/
if (fs->super->s_feature_incompat & EXT3_FEATURE_INCOMPAT_EXTENTS)
blks_needed += (fs->super->s_blocks_count - blks_needed)/500;
can go quite wrong if we've already added up more "blks_needed"
than our current size, and the above subtraction wraps. This can
easily happen for a filesystem which is almost completely full.
In this case, just return the current fs size as the minimum and
be done with it.
With this fix we could probably call calculate_minimum_resize_size()
for each resize2fs invocation and refuse to resize smaller than that?
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Due to a fencepost bug, when skipping a block group whose block bitmap
was uninitialized (and hence could not contain any blocks eligible for
relaocation), the block immediately following the block group wasn't
checked as well. If it was in use and required relocation, it
wouldn't get properly relocated, with the result that an inode using
such a block would end up, post resize, with a pointer to a block now
outside the bounds of the filesystem.
This commit fixes this fencepost error.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
If we need to shrink the inode table, we need to make sure the inodes
contained in the part of the inode table we are vacating don't get
reused as part of the filesystem shrink operation. This wasn't a
problem with ext3 filesystems, since the inode table was located in
the block group that was going away, so that location was not eligible
for reallocation.
However with ext4 filesystems with flex_bg enabled, it's possible for
a portion of the inode table in the last flex_bg group to be
deallocated, but in a part of the filesystem which could be used as
data blocks. So we must mark those blocks as reserved to prevent
their reuse, and adjust the minimum filesystem size calculation to
assure that we don't shrink a filesystem too small for the resize
operation to succeed.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
When allocating a new set of block group metadata as part of growing
the filesystem, the resize2fs code assumes that the bitmap and inode
table blocks are in their own block group; an assumption which is
changed by the flex_bg feature. This commit works around the problem
by temporarily turning off flex_bg while allocating the new block
group metadata, to avoid potentially overwriting previously allocated
data blocks.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Some extra blocks may be needed to expand some extent allocation trees
while we are shrinking the filesystem. We don't know exactly how
much, so we use a hueristic.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Previously resize2fs assumed that bitmap and inode table blocks were
always located in their respective block group. However, this is no
longer true with flex_bg. So it is necessary to check all of the
block groups which will be truncated to see if they have metadata
blocks that need to be marked as no longer being in use in the new,
shrunk filesystem.
This bug fixes resize2fs -M, which would otherwise fail because
without the released blocks, there would not be enough space in the
filesystem. This bug also avoids (mostly harmless) filesystem
corruptions reported by e2fsck regarding blocks marked in use but not
actually used (these being the bitmap and inode table blocks
associated with the truncated block groups).
Note: in theory it is possible to have block group N utilize bitmap
and inode table blocks in block group N+X with flex_bg. At the moment
neither mke2fs nor e2fsck will create filesystems like this, which is
good, because resize2fs doesn't handle this case correctly.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
In the function blocks_to_move(), when checking to see if a block
group's block bitmap is initialized, we need to check the old_fs's
block group descriptors, not the new file system's (already truncated)
group descriptor data structures. Otherwise we will end up
derferencing past the end of the array boundary, and the resulting
garbage value may indicate that the bitmap is uninitialized, and so
all of the blocks in that block group will be skipped, resulting in
some blocks not getting marked as needing relocation.
This showed up in the following test case:
mke2fs -t ext4 -b 1024 test.img 1048576
resize2fs test.img 80000
The journal inode after the resize operation looked like this:
debugfs: stat <8>
Inode: 8 Type: regular Mode: 0600 Flags: 0x80000
...
BLOCKS:
(IND):35385, (0-5836):2356-8192, (5837-21959):8454-24576, (21960-32506):24838-35
384, (32507-32767):434177-434437
TOTAL: 32769
The blocks 434177-434437 were not moved because block group 53 was
wrongly thought to have an unitialized block group.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Some of these could affect filesystems between 2^31 and 2^32-1 blocks.
Thanks to Valerie Aurora Henson for pointing out the problems in
lib/ext2fs/alloc_tables.c, which led me to do a "make gcc-wall" scan
over the source tree.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Theodore Ts'o