Currently there are many uses of ext2fs_close() which might be wrong.
First of all ext2fs_close() does not set the ext2_filsys pointer to NULL
so the caller is responsible for clearing it, however there are some
cases there we do not do it.
Second of all very small number of users of ext2fs_close() actually
check the return value. If there is a problem in ext2fs_close() it will
not even free the ext2_filsys structure, but majority of users expect it
to do so.
To fix both problems this commit introduces a new helper
ext2fs_close_free() which will not only check for the return value and
free the ext2_filsys structure if the call to ext2fs_close2() failed,
but it will also set the ext2_filsys pointer to NULL.
Replace every use of ext2fs_close() in e2fsprogs tools with
ext2fs_close_free() - there is no real reason to keep using
ext2fs_close().
Signed-off-by: Lukas Czerner <lczerner@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
My old patch:
resize2fs: don't print minimum size if fs is not clean
almost did this, but it still calculated the size; it just didn't print
it. Which is a bit silly.
Jes had a pretty badly corrupted image which made the minimum size
calculation go off into the weeds. It was corrupted, and also marked
as having an error.
We'll eventually bail out for an unmounted filesystem if it's marked
as being in an error state anyway; just move that test & bail-out
to a much earlier point, and remove the now-duplicate one under the
print_min_size block.
This will catch & block all resize operations on an offline filesystem
with errors, in one central place.
Reported-by: Jes Sorensen <Jes.Sorensen@redhat.com>
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
In reserve_sparse_super2_last_group, the old_desc check should only be
performed if ext2fs_super_and_bgd_loc2() gave us a location -- a
return value of 0 means that there is no old-style GDT block.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
In the case where the new location of the inode table is before the
old inode table, the optimization which tries to optimize zero block
moves breaks. Fix it.
This fixes a bug that was tickled by the reproduction described in the
previous commit.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The previous commit exposed bugs in the calculation for flex_bg file
systems. The problem is that since (by default) we keep the metadata
blocks for the flex_bg in the first block group of the flex_bg, and
because we don't want to overwrite metadata blocks used by the
original file system with data blocks make life easier in case the
resize is aborted for some reason, we need to treat all of the
metadata blocks in the existing flex_bg has in use for the purposes of
calculate_minimum_resize_size().
Even though this means we need to reserve more data blocks to avoid
running out of space, the net result of these two commits is a net
savings in how much we can shrink a file system.
Using the following test sequence:
mke2fs -F -t ext4 /tmp/foo.img 2T
resize2fs -M /tmp/foo.img
resize2fs -M /tmp/foo.img
resize2fs -M /tmp/foo.img
Here is the comparison in the resulting file systems between the old
and new resize2fs (units are in 4k blocks):
resize #1 resize #2 resize #3
old resize2fs 1117186 45679 43536
new resize2fs 48784 37413 37392
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
For extent-mapped file systems, we need to reserve some extra space in
case we need to grow the extent tree. Calculate the safety margin
more intelligently, so we don't overestimate the amount of space
required.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Reported-by: Dmitry Monakhov <dmonakhov@openvz.org>
When we're moving blocks around the filesystem, ensure that freeing
the old blocks only frees the clusters if they're not in use by other
metadata.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
When freeing a block group descriptor block, be careful not to free
metadata clusters belonging to other groups!
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
To check the coverage of e2fsprogs's regression test, do the
following:
configure --enable-gcov
make -j8 ; make -j8 check ; make coverage.txt
The coverage information will be the coverage.txt and *.gcov files in
the build directories.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
ext2fs_free() does not set the ext2_filsys pointer to null so the
caller is responsible to setting it himself if it is needed.
This patch fixes some places where caller did not set ext2_filsys
pointer to NULL after ext2fs_free() which might result in use after
free. Fix it.
Signed-off-by: Lukas Czerner <lczerner@redhat.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
In practice, it is **extremely** rare for users to try to use more
than the first backup superblock located at the beginning of block
group #1. (i.e., at block number 32768 for file systems with a 4k
block size). This new compat feature restricts the backup superblock
to block group #1 and the last block group in the file system.
Aside from reducing the overhead of the file system by a small number
of blocks, by eliminating the rest of the backup superblocks, it
allows us to have a much more flexible metadata layout. For example,
we can force all of the allocation bitmaps and inode table blocks to
the beginning of the disk, which allows most of the disk to be
exclusively used for contiguous data blocks.
This simplifies taking advantage of certain HDD specific features,
such as Shingled Magnetic Recording (aka Shingled Drives), and the
TCG's OPAL Storage Specification where having a simple mapping between
LBA block ranges and the data blocks used by the file system can make
life much simpler.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Newer versions of autoconf pull in AC_PROG_GCC as part of
AC_CANONICAL_HOST. So we need check for WITH_DIET_LIBC earlier in
configure.in.
Also, e2fsprogs now needs functions which are found in diet libc's
compat library. So add support for autoconf's LIBS function, and
automatically set libs to include -lcompat.
Finally, disable compiling e4defrag by deault if --with-diet-libc is
specified because the program has too many glibc dependencies.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Mostly by adding static and removing excess extern qualifiers. Also
convert a few remaining non-ANSI function declarations to ANSI.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Compiling with LLVM generates a large number of warnings due
to the use of _() for wrapping strings for i18n:
warning: format string is not a string literal
(potentially insecure) [-Wformat-security]
./nls-enable.h:4:14: note: expanded from macro '_'
#define _(a) (gettext (a))
^~~~~~~~~~~~
These warnings are fixed by using "%s" as the format string,
and then _() is used as the string argument.
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Run sparse against source files when building e2fsprogs with 'make C=1'. If
instead C=2, it configures basic ext2 types for bitwise checking with sparse,
which can help find the (many many) spots where conversion errors are
(possibly) happening.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
If the file system is being shrunk, and a block group's inode table
falls beyond the end of the inode table, we need to try to relocate
the inode table blocks.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
If the file system's inode table blocks in the last block group are
located in the middle or the end of the block group, it's possible for
resize2fs -M to use a size which will require relocating the inode
table blocks in the last block group. This can lead to all sorts of
problems, so solve it by simply guaranteeing that we will never do
that.
Reported-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
free_gdp_blocks needs to be taught to use 64-bit fields and the appropriate
getters, otherwise it'll truncate high block numbers (when, say, resizing a
>16T fs) and mark the low numbered group descriptor blocks as free. Yikes.
Reported-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
It is possible to have a flex_bg filesystem with block groups
which have inode & block bitmaps at some point well past the
start of the group.
If an offline shrink puts the new size somewhere between
the start of the block group and the (old) location of
the bitmaps, they can be left beyond the end of the filesystem,
i.e. result in fs corruption.
Check each remaining block group for whether its bitmaps
are beyond the end of the new filesystem, and reallocate
them in a new location if needed.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
When doing an off-line resize2fs of an initially very small file
system, it's possible to run out of reserved gdt blocks (which are
reserved via the resize inode). Once we run out, we need to move the
allocation bitmaps and inode table out of the way to grow the gdt
blocks. Unfortunately, when moving these metadata blocks, it was
possible that a block that had been just been newly allocated for a
new block group could also get allocated for a metadata block for an
existing block group that was being moved.
To prevent this, after we grow the gdt blocks and allocate the
metadata blocks for the new block groups, make sure all of these
blocks are marked as reserved.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Reported-by: John Jolly <john.jolly@gmail.com>
Fixes resize2fs so it correctly calculates the number of free clusters
in each block group for file systems with the bigalloc feature
enabled.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
This should be made into a more formal, automated test case, but for
now, save this as script since it's useful for validating resize2fs's
handling of very large file systems.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The old online resize ioctl interfaces are still present in the
kernel, and we want to make it easy to test both the kernel code for
those older interfaces, and resize2fs's use of those interfaces in a
relatively easy manner.
To do this, resize2fs will now check the environment variable
RESIZE2FS_KERNEL_VERSION. If the version given is less than 3.3, then
do not try using the new resizing ioctl, but instead use the resizing
ioctls that were used before Linux version 3.3.
If the version given is less than 3.7, then emulate sanity checks
which get done to protect against the fact that the new resizing ioctl
prior to 3.7 did not handle meta_bg resizing. (This was previously
tested via the presence of the RESIZE2FS_NO_META_BG_RESIZE environment
variable. But the new environment variable, RESIZE2FS_KERNEL_VERISON,
is more general.)
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Reviewed-by: Carlos Maiolino <cmaiolino@redhat.com>
There are a number of places where we multiply a dgrp_t with
s_blocks_per_group expecting that we will get a blk64_t. This
requires a cast, or using the convenience function
ext2fs_group_first_block2().
This audit was suggested by Eric Sandeen.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Use ext2fs_[un]mark_block_range2() functions to reduce the CPU
overhead of resizing large file systems by 45%, primarily by
reducing the time spent in fix_uninit_block_bitmaps().
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Add a new debug flag which prints how much time is consumed by the
various parts of resize2fs's processing.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
This caused the free blocks count in the superblock to be incorrect
after resizing a 64-bit file system if the number of free blocks
overflowed a 32-bit value.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Fix a 32-bit overflow bug caused by a missing blk64_t cast which can
cause the block bitmap to get corrupted when doing an off-line resize
of a 64-bit file system.
This problem can be reproduced as follows:
rm -f foo.img; touch foo.img
truncate -s 8T foo.img
mke2fs -F -t ext4 -O 64bit foo.img
e2fsck -f foo.img
truncate -s 21T foo.img
resize2fs foo.img
e2fsck -fy foo.img
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Now that we are reserving all of the bg-specific metadata before we
try to allocate the metadata for the new block groups, we don't have
to temporarily disable the flex_bg feature flag while we allocate the
new metadata blocks --- this allows the newly created block groups to
have a much more optimized layout, instead of fragmenting the inode
table and block/inode bitmaps in sepraate block groups.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
With the bug fixes from the last two commits, resize2fs can now fully
support off-line resizing of file systems with flex_bg even if the
resize_inode feature is not present; so we no longer need to disallow
this combination.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
With flex_bg file systems, bg-specific metadata (i.e., bitmaps and the
inode table blocks) can be located in another block group. Hence,
when we grow the number of block group descriptors, we need to check
if we need to relocate metadata blocks not just for the block group
where the bgd blocks are located, but in all block groups.
This change fixes the following test case:
rm -f foo.img; touch foo.img
truncate -s 32G foo.img
mke2fs -F -t ext4 -E resize=12582912 foo.img
e2fsck -f foo.img
truncate -s 256G foo.img
./resize2fs foo.img
e2fsck -fy foo.img
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
For flex_bg file systems, if we need to relocate an allocation bitmap
or inode table, we need to make sure that all metadata blocks have
been reserved, lest we end up overwriting a metadata block belonging
to a different block group.
This change fixes the following test case:
rm -f foo.img; touch foo.img
truncate -s 32G foo.img
mke2fs -F -t ext4 -E resize=12582912 foo.img
e2fsck -f foo.img
truncate -s 64G foo.img
./resize2fs foo.img
e2fsck -fy foo.img
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
This is the first commit to add support for off-line resizing using
flex_bg without the assist of using the resize_inode to reserve gdt
blocks. This functionality has been broken up into separate commits
which are hopefully obviously correct to make them easier to review
for correctness.
In this first step, we break up the for loop at the end of
blocks_to_move() so that we first mark all of the metadata blocks
which don't need to be moved in the reserve_blocks bitmap, and then
try to allocate the metadata blocks are new or which need to moved
second.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
One of these fixes was triggering failures when running:
./test_scripts --valgrind r_move_itable r_inline_xattr r_resize_inode
It should be a false positive, but it fixing this makes it easier to
see real problems.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
If the uninit_bg feature is enabled and the kernel supports
lazy_itable_init, skip zeroing the inode table so that the resize
operation can go much more quickly. Also set the itable_unused fields
so that the first e2fsck after the resize will run faster.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Enhance the online resizing code to be more nuanced about resizing
restrictions. If the kernel supports meta_bg resizing, then we can
skip all of the restrictions. If the kernel does not support meta_bg
resizing, check more carefully to make sure there are enough reserved
gdt blocks, so that the user gets a clearer error message.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Resize2fs can't handle resizing flex_bg file systems that do not have
the resize inode, but when the kernel adds support for resizing using
the meta_bg layout, we should allow it be able to resize the file
system.
So move the flex_bg/resize_inode check to the just before we start
doing the off-line resize, instead of doing it earlier where it would
prohibit these file systems for both on-line and off-line resizes.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The file system overhead calculation in calculate_minimum_resize_size
was incorrect meta_bg file systems. This caused the minimum size to
underflow for very large file systems, which threw resize2fs into a
loop generally lasted longer than the user's patience.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The 16TB limit must be enforced regardless of whether the new size is
specified on the command line or implied by the size of the device,
but only if the file system does not support 64-bit block sizes, or
the kernel does not advertise support of meta_bg resizing.
Previously we were unconditionally enforcing it when it was implied by
the device size, but not if the new size was specified on the command
line.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Also fixed a number of other minor nits in the resize2fs and e2image
man pages.
Addresses-Debian-Bug: #674453, #674694
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
By passing EXT2_FLAG_64BITS to ext2fs_open2() we can avoid some
unnecessary redirection in critical paths. While resize2fs does not
currently otherwise support so big filesystems that this would matter,
passing this flag is entirely harmless and only tells libext2fs that
the caller has been recompiled against current headers.
With this change the CPU time needed to shrink a 100G filesystem drops
by 20%.
Signed-off-by: Sami Liedes <sami.liedes@iki.fi>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Clean up some compile warnings related to fstat64(), which is
verbosely deprecated on OSX.
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
calculate_minimum_resize_size() forgot to account s_first_data_block
into minimum filesystem size. Thus in case the size of filesystem was
such that the last group had the minimal size (50 blocks + metadata
overhead), the code in adjust_fs_info() decided the group is unneeded,
removed it, and in some cases the resizing then failed with ENOSPC.
Fix the issue by properly accounting for s_first_data_block in
calculate_minimum_resize_size().
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Linux's compat_sys_ioctl() function, which is run when executing a
ioctl using a 32-bit binary on a 64-bit kernel, returns EINVAL when an
inode does not exist. Sigh. See /usr/src/linux/fs/compat_ioctl.c.
This is probably a kernel bug, but work around it for now.
Addresses-Debian-Bug: #644989
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Lukas Czerner