ext2fs_flush2() unconditionally writes the block group descriptors to
disk even if the underlying FS isn't marked dirty. This causes the
following error message on a fsck -n run:
e2fsck 1.43-WIP (09-Jul-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Error writing block 2 (Attempt to write block to filesystem resulted in short write). Ignore error? no
Error writing block 2 (Attempt to write block to filesystem resulted in short write). Ignore error? no
Error writing file system info: Attempt to write block to filesystem resulted in short write
Since ext2fs_close2() only calls flush if the dirty flag is set,
modify e2fsck to exhibit the same behavior so that we don't spit out
write errors for a read only check.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
This test checks to make sure resize2fs can properly handle a file
system which started life as a normal ext4 file system and then was
grown to a size where meta_bg was enabled, and then shrunk back below
the point where the meta_bg format is still needed.
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
The test verifies that e2fsck can properly fix a file system where the
value of s_first_meta_bg in the superblock is larger than the number
of block group descriptors in the file system. E2fsck will fix this
by clearing the meta_bg feature.
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Place the allocation bitmaps and inode table blocks so they are
adjacent, even in the last flexbg.
Previously, after running "mke2fs -t ext4 DEV 286720", the layout of
the last few block groups would look like this:
Group 32: (Blocks 262145-270336) [INODE_UNINIT, ITABLE_ZEROED]
Block bitmap at 262145 (+0), Inode bitmap at 262161 (+16)
Inode table at 262177-262432 (+32)
Group 33: (Blocks 270337-278528) [INODE_UNINIT, BLOCK_UNINIT, ITABLE_ZEROED]
Block bitmap at 262146 (bg #32 + 1), Inode bitmap at 262162 (bg #32 + 17)
Inode table at 262433-262688 (bg #32 + 288)
Group 34: (Blocks 278529-286719) [INODE_UNINIT, ITABLE_ZEROED]
Block bitmap at 262147 (bg #32 + 2), Inode bitmap at 262163 (bg #32 + 18)
Inode table at 262689-262944 (bg #32 + 544)
Now, they look like this:
Group 32: (Blocks 262145-270336) [INODE_UNINIT, ITABLE_ZEROED]
Block bitmap at 262145 (+0), Inode bitmap at 262148 (+3)
Inode table at 262151-262406 (+6)
Group 33: (Blocks 270337-278528) [INODE_UNINIT, BLOCK_UNINIT, ITABLE_ZEROED]
Block bitmap at 262146 (bg #32 + 1), Inode bitmap at 262149 (bg #32 + 4)
Inode table at 262407-262662 (bg #32 + 262)
Group 34: (Blocks 278529-286719) [INODE_UNINIT, ITABLE_ZEROED]
Block bitmap at 262147 (bg #32 + 2), Inode bitmap at 262150 (bg #32 + 5)
Inode table at 262663-262918 (bg #32 + 518)
This reduces the free space fragmentation in a freshly created file
system. It also allows the following mke2fs command to succeed:
mke2fs -t ext4 -b 4096 -O ^resize_inode -G $((2**20)) DEV 2130483
(Note that while this allows people to run mke2fs with insanely large
flexbg sizes, this is not a recommended practice, as the kernel may
refuse to resize such a file system while mounted, since it currently
tries to allocate an in-memory data structure based on the size of the
flexbg, and so a file system with a very large flexbg size will cause
the memory allocation to fail. This will hopefully be fixed in a
future kernel release, but if the goal is to force all of the metadata
blocks to be at the beginning of the file system, it's better to use
the packed_meta_blocks configuration parameter in mke2fs.conf.)
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
The f_badcluster output format depends on how libreadline formats
and outputs the commands read from stdin. Instead of trying to
handle these differences, use an input command file, which does
not depend on external components to be consistent.
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
This should have been part of commit 9a1d614df2 ("e2fsck: fix
rule-violating lblk->pblk mappings on bigalloc filesystems") but it
accidentally got dropped when the patch was applied.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
If we think we're going to need to repair either the root directory or
the lost+found directory, reserve a block at the end of pass 1 to
reduce the likelihood of an e2fsck abort while reconstructing
root/lost+found during pass 3.
If / and/or /lost+found are corrupt and duplicate processing in pass
1b allocates all the free blocks in the FS, fsck aborts with an
unusable FS since pass 3 can't recreate / or /lost+found. If either
of those directories are missing, an admin can't easily mount the FS
and access the directory tree to move files off the injured FS and
free up space; this in turn prevents subsequent runs of e2fsck from
being able to continue repairs of the FS.
(One could migrate files manually with debugfs without the help of
path names, but it seems easier if users can simply mount the FS and
use regular FS management tools.)
[ Fixed up an obvious C trap: const char * and const char [] are not
the same thing when you are taking the size of the parameter.
People, run your regression tests! Like spinach, it's good for you. :-)
-- tytso ]
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Provide an API to set i_size in an inode and take care of all required
feature flag modifications. Refactor the code to use this new
function.
[ Moved the function to lib/ext2fs/blk_num.c, which is the rest of
these sorts of functions live, and renamed it to be
ext2fs_inode_size_set() instead of ext2fs_inode_set_size() to be
consistent with the other functions in in blk_num.c -- tytso ]
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
We rely on a nasty hack to adjust the free block count where we pass
signed value into ext2fs_free_blocks_count_add(), which takes an
64-bit unsigned value, and relies on overflow and C's signed->unsigned
semantics to do the subtraction. This works, so long as a 64-bit
signed value is used.
Unfortunately, ext2fs_block_alloc_stats2() and
ext2fs_block_alloc_stats_range(), this is not true, so on a 64-bit
file system, the free blocks accounting can get screwed up.
A simple way to demonstrate the problem is:
mke2fs -F -t ext4 -O 64bit /tmp/foo.img 1M
e2fsck -fy /tmp/foo.img
... which will result in the following e2fsck complaint:
Pass 5: Checking group summary information
Free blocks count wrong (4294968278, counted=982).
Fix? yes
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Directories can't have uninitialized extents, so offer to clear the
uninit flag when we find this situation. The actual directory blocks
will be checked in pass 2 and 3 regardless of the uninit flag.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Currently, directories cannot be fallocated, which means that the only
way they get bigger is for the kernel to append blocks one by one.
Therefore, if we encounter a logical block offset that is too big, we
needn't bother adding it to the dblist for pass2 processing, because
it's unlikely to contain a valid directory block. The code that
handles extent based directories also does not add toobig blocks to
the dblist.
Note that we can easily cause e2fsck to fail with ENOMEM if we start
feeding it really large logical block offsets, as the dblist
implementation will try to realloc() an array big enough to hold it.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Always iterate logical block 0 in a directory, even if no physical
block has been allocated. Pass 2 will notice the lack of mapping and
offer to allocate a new directory block; this enables us to link the
directory into lost+found.
Previously, if there were no logical blocks mapped, we would fail to
pick up even block 0 of the directory for processing in pass 2. This
meant that e2fsck never allocated a block 0 and therefore wouldn't fix
the missing . and .. entries for the directory; subsequent e2fsck runs
would complain about (yet never fix) the problem.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
If we notice a hole in the block map of an extent-based directory,
offer to collapse the hole by decreasing the logical block # of the
extent. This saves us from pass 3's inefficient strategy, which fills
the holes by mapping in a lot of empty directory blocks.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
In the loop in ext2fs_get_free_blocks2, we ask the bitmap if there's a
range of free blocks starting at "b" and ending at "b + num - 1".
That quantity is the number of the last block in the range. Since
ext2fs_blocks_count() returns the number of blocks and not the number
of the last block in the filesystem, the check is incorrect.
Put in a shortcut to exit the loop if finish > start, because in that
case it's obvious that we don't need to reset to the beginning of the
FS to continue the search for blocks. This is needed to terminate the
loop because the broken test meant that b could get large enough to
equal finish, which would end the while loop.
The attached testcase shows that with the off by one error, it is
possible to throw e2fsck into an infinite loop while it tries to
find space for the inode table even though there's no space for one.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
The -t option is documented as deprecated in GNU's mktemp, and
FreeBSD's mktemp doesn't support it at all.
Replace it with the construct "mktemp ${TMPDIR:-/tmp}/foo.XXXXXX"
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
In addition, incorporate the test name into the e2fsprogs-tmp to make
it easier to debug left-over temp files in the future.
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
It can be made simpler because there is no need to differentiate between
having an internal journal inode and having an external journal device.
Signed-off-by: Benno Schulenberg <bensberg@justemail.net>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
The default configuration still has quota disabled, but
runs the f_quota test unconditionally, so we fail by
default.
Fix that...
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
The strptime() function does not update fields in struct tm that are
not specified in the input format. The glibc implementation sets the
tm_yday field (%j) when any of the year (%Y), month (%m), or day (%d)
fields are changed, but the MacOS strptime() does not set tm_yday in
this case. This caused string_to_time() to calculate the wrong Unix
epoch on MacOS. If tm_yday is unset, compute it in string_to_time().
This also fixes test regression failures for FreeBSD.
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
We no longer need to reference https://ext4.wiki.kernel.org/index.php/Quota
since we've fixed the nasty bugs associated with e2fsck and the quota
feature. The wiki page will be updated once we've done a release that
includes these fixes indicated the verison which these problems have
been fixed.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Reviewed-by: Aditya Kali <adityakali@google.com>
The previous commit exposed bugs in the calculation for flex_bg file
systems. The problem is that since (by default) we keep the metadata
blocks for the flex_bg in the first block group of the flex_bg, and
because we don't want to overwrite metadata blocks used by the
original file system with data blocks make life easier in case the
resize is aborted for some reason, we need to treat all of the
metadata blocks in the existing flex_bg has in use for the purposes of
calculate_minimum_resize_size().
Even though this means we need to reserve more data blocks to avoid
running out of space, the net result of these two commits is a net
savings in how much we can shrink a file system.
Using the following test sequence:
mke2fs -F -t ext4 /tmp/foo.img 2T
resize2fs -M /tmp/foo.img
resize2fs -M /tmp/foo.img
resize2fs -M /tmp/foo.img
Here is the comparison in the resulting file systems between the old
and new resize2fs (units are in 4k blocks):
resize #1 resize #2 resize #3
old resize2fs 1117186 45679 43536
new resize2fs 48784 37413 37392
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Old distros may not have the "truncate" tool, so use "dd" instead.
If tmpfs cannot handle a 2GB temp file (e.g. old RHEL5 and SLES 11
kernels) then skip the test instead of failing it. If this fails,
try to report better error messages instead of failing silently.
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
To check the coverage of e2fsprogs's regression test, do the
following:
configure --enable-gcov
make -j8 ; make -j8 check ; make coverage.txt
The coverage information will be the coverage.txt and *.gcov files in
the build directories.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The "mkswap" program is not available on MacOS, so just use the
existing swap0.img.bz2 and swap1.img.bz2 files directly.
Because MacOS HFS+ doesn't support sparse files (welcome to the 80's)
the m_bigjournal test takes forever to zero out the whole 42GB test
filesystem. Skip this test for Darwin kernels for now.
Unfortunately, neither "df -T" nor "stat -f -c %T" is available on
MacOS to directly determine the filesystem type, and I'm too lazy
to parse the output of "mount" and match it to the path of the test
directory in shell, so it just checks the kernel type and assumes
the filesystem type is HFS and skips the test.
Since this test runs on Linux the majority of the time, the loss of
test coverage is minimal. If MacOS should ever get a real filesystem,
this can be revisited.
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Now that libext2fs marks group metadata in the fs block bitmap, adjust
the expected test output to reflect expanded use of block_uninit and
the fact debugfs no longer prints block bitmap data that fails to
account for group data blocks.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
The dietlibc doesn't support the TZ environment variable, which is
required by the standard. Work around this so that we can run the
regression test suite when building with dietlibc. (This is useful
for finding problems.)
With this change, the only thing which doesn't work as far as dietlibc
is concerned is the posix_memalign test, and the MMP support tests
(because posix_memalign isn't provided by dietlibc, sigh.)
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Newer versions of autoconf pull in AC_PROG_GCC as part of
AC_CANONICAL_HOST. So we need check for WITH_DIET_LIBC earlier in
configure.in.
Also, e2fsprogs now needs functions which are found in diet libc's
compat library. So add support for autoconf's LIBS function, and
automatically set libs to include -lcompat.
Finally, disable compiling e4defrag by deault if --with-diet-libc is
specified because the program has too many glibc dependencies.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Previously, this message used 8193 as the example alternate
superblock. But for most file systems, the backup superblock is
located at 32768 (since most file systems have a block size of 4k, and
not 1k).
Addresses-Debian-Bug: #719185
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
If given at least one offset and only one file, assume source
and dest are the same, and do an in place move.
Signed-off-by: Phillip Susi <psusi@ubuntu.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Fix the f_extent_oobounds test so that it uses binaries built in the
tree, instead of the binaries in the system PATH (which might not
exist in a chroot environment) when creating the test image.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The s_desc_size in the superblock specifies the group descriptor
size in bytes, but in various places the EXT4_FEATURE_INCOMPAT_64BIT
flag implies that the descriptor size is EXT2_MIN_DESC_SIZE_64BIT
(64 bytes) instead of checking the actual size. In other places,
the s_desc_size field is used without checking for INCOMPAT_64BIT.
In the case of ext2fs_group_desc() the s_desc_size was being ignored,
and assumed to be sizeof(struct ext4_group_desc), which would result
in garbage for any but the first group descriptor. Similarly, in
ext2fs_group_desc_csum() and print_csum() they assumed that the
maximum group descriptor size was sizeof(struct ext4_group_desc).
Fix these functions to use the actual superblock s_desc_size if
INCOMPAT_64BIT.
Conversely, in ext2fs_swap_group_desc2() s_desc_size was used
without checking for INCOMPAT_64BIT being set.
The e2fsprogs behaviour is different than that of the kernel,
which always checks INCOMPAT_64BIT, and only uses s_desc_size to
determine the offset of group descriptors and what range of bytes
to checksum.
Allow specifying the s_desc_size field at mke2fs time with the
"-E desc_size=NNN" option. Allow a power-of-two s_desc_size
value up to s_blocksize if INCOMPAT_64BIT is specified. This
is not expected to be used by regular users at this time, so it
is not currently documented in the mke2fs usage or man page.
Add m_desc_size_128, f_desc_size_128, and f_desc_bad test cases to
verify mke2fs and e2fsck handling of larger group descriptor sizes.
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Since commit 5ad07acad if $TMP cannot hold large test filesystems
for resize testing the resize_test creates temporary test files
in the local working directory. Since it overrides TMPFILE locally
the calling program does not delete the generated file correctly.
Delete the large $TMPFILE within resize_test if it passes, but leave
it for debugging if the test failed.
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The new resize tests create 2TB test files, but tmpfs in kernels
before 3.1 have a max file size of 256GB. Ext3 may also have
a size limit for smaller blocksize filesystems.
Fix the resize_test script to verify that $TMPFILE can be resized
to the final test size, and if that fails try creating the file on
the local filesystem instead of in $TMPDIR. If that cannot hold
the large filesystem, skip the test.
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Tested-by: Zheng Liu <wenqing.lz@taobao.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Update all superblock copies when disabling the quota feature.
Added basic tests for the quota feature.
Signed-off-by: Niu Yawei <niu@whamcloud.com>
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Run sparse against source files when building e2fsprogs with 'make C=1'. If
instead C=2, it configures basic ext2 types for bitwise checking with sparse,
which can help find the (many many) spots where conversion errors are
(possibly) happening.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Eric Sandeen reported that Fedora's mke2fs when compiled for ppc was
creating a file system which caused problems with resize2fs -M.
Closer examination showed that the problem was file system which
looked like this:
Filesystem features: ext_attr dir_index filetype sparse_super
Inode count: 512
Block count: 1247
...
Group 0: (Blocks 1-1024)
Primary superblock at 1, Group descriptors at 2-2
Block bitmap at 66 (+65), Inode bitmap at 67 (+66)
Inode table at 68-99 (+67)
Group 1: (Blocks 1025-1246)
Backup superblock at 1025, Group descriptors at 1026-1026
Block bitmap at 1090 (+65), Inode bitmap at 1091 (+66)
Inode table at 1092-1123 (+67)
It's not obvious to me why Fedora's ppc mke2fs is creating file
systems like this (I can't reproduce this on debian ppc systems), but
resize2fs -M should be able to deal with such file systems, which is
what this test is designed to check.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Commit d3f32c2db8 was intended to detect extents found outside their
proper location in the extent tree, including invalid extents at the
end of an extent block. However, it incorrectly reported legal
uninitialized extents created by fallocate() at the end of file with
the FALLOC_FL_KEEP_SIZE flag as false positives. xfstests
generic/263 (among others) caught this problem, while the e2fsprogs
test f_uninit_ext_past_eof did not. The latter test failed to
detect the problem in part because it uses a test file whose i_size
is 0.
Add a test derived from the fsx-based test case in xfstests
generic/263 consisting of a file with non-zero length, more than
four extents total, and two uninitialized extents past EOF to
reliably reproduce commit d3f32c2db8's false positive behavior.
Signed-off-by: Eric Whitney <enwlinux@gmail.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
The function deallocate_inode() in e2fsck/pass2.c was buggy in that it
would clear out the inode's mode and flags fields before trying to
deallocate any blocks which might belong to the inode.
The good news is that deallocate_inode() is mostly used to free inodes
which do not have blocks: device inodes, FIFO's, Unix-domain sockets.
The bad news is that if deallocate_inode() tried to free an invalid
extent-mapped inode, it would try to interpret the root of the extent
node as block numbers, and would therefore mark various file system
metadata blocks (the superblock, block group descriptors, the root
directory, etc.) as free and available for allocation. This was
unfortunate.
(Try running an older e2fsck against the test file system image in the
new test f_invalid_extent_symlink, and then run e2fsck a second time
on the fs image, and weep.)
Fortunately, this kind of file system image corruption appears to be
fairly rare in actual practice, since it would require a very unlucky
set of bits to be flipped, or a buggy file system implementation.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
E2fsck was missing a check for directories with logical blocks so
large that i_size > 2GB. Without this check the test image found in
the new test f_toobig_extent_dir will cause e2fsck to die with a
memory allocation failure:
Error storing directory block information (inode=12, block=0, num=475218819): Memory allocation failed
e2fsck: aborted
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Reported-by: Andrey Melnikov <temnota.am@gmail.com>
The environment variable EXT2FS_NO_MTAB_OK will suppress the error
code EXT2_NO_MTAB_FILE when the /etc/mtab file can not be found. This
allows the e2fsprogs regression test suite to be run in chroots which
might not have an /etc/mtab file.
By default will still want to complain if the /etc/mtab file is
missing, since we really don't want to discourage distributions and
purveyors of embedded systems from running without an /etc/mtab file.
But if it's missing it only results in a missing sanity check that
might cause file system corruption if the file system is mounted when
programs such as e2fsck, tune2fs, or resize2fs is running, so there is
no potential security problems that might result if this environment
variable is set inappropriately.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Don't change the root directory's UID/GID automatically just because
mke2fs was run as a non-root user. This can be confusing for users,
and is not flexible for non-root installation tools that need to
create a filesystem with different ownership from the current user.
Add the "-E root_owner[=uid:gid]" option to mke2fs so that the user
and group can be explicitly specified for the root directory. If
the "=uid:gid" argument is not specified, the current UID and GID
are extracted from the running process, as was done in the past.
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
A minor cleanup to order the command-line option parsing in
alphabetical order, except for "-E" and "-R", which need to
be co-located.
Print a message that the "-R" option is deprecated. It has
been deprecated since 2005 (commit c6a44136b9).
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Darrick J. Wong