Turns out a "should never happen" error can indeed happen very easily
if a directory with an htree index has an incorrect, and too-large,
i_size field. This patch fixes this so that we handle this situation
gracefully, allowing filesystems with this error to be fixed.
In another patch I will clean up the specific problem which caused the
internal "should never happen" error from happening at all, but patch
will prevent e2fsck from crashing, and prompt the user to remove the
htree index, so it can be rebuilt again after pass 3.
Thanks to Bas van Schaik at Tetra for giving me access to his system
so this problem could be debugged.
Addresses-Launchpad-Bug: #129395
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Stop clearing the EXT2_FEATURE_RO_COMPAT_LARGE_FILE flag automatically
if there are no large files in the filesystem. It's been almost a
decade since there have been kernels that don't support this flag, and
e2fsck clears it quietly without telling the user why the filesystem
has been changed.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Factor out code to clear a bogus inode and update e2fsck's internal
data structures accordingly into a common routine,
e2fsck_clear_inode(). This saves about 200 bytes in the compiled x86
e2fsck executable, and makes the code more maintainable in the
long-term.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
If e2fsck adds or deletes any of the feature bitmasks, clear
EXT2_FLAG_MASTER_SB_ONLY so the backup superblocks are updated when
e2fsck finishes.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
This patch instruments the libext2fs unix I/O manager and adds bytes
read/written and data rate to e2fsck -tt pass/overall timing output.
Signed-off-by: Jim Garlick <garlick@llnl.gov>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Recently, one of our customers found this message in pass2 of e2fsck
while doing some regression testing:
"Entry '4, 0x695a, 0x81ff, 0x0040, 0x8320, 0xa192, 0x0021' in ??? (136554) has
rec_len of 14200, should be 26908."
Both the displayed rec_len and the "should be" value are bogus. The
reason is that salvage_directory sets a offset beyond blocksize
leading to bogus messages.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Fix a problem byte-swapping fast symlinks inodes that contain extended
attributes.
Addresses Red Hat Bugzilla: #232663
Addresses LTC Bugzilla: #27634
Signed-off-by: "Bryn M. Reeves" <breeves@redhat.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Add better ehandler_operation() markers so it is clearer what e2fsck was
doing when an I/O error is reported.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
If e2fsck.conf configures a scratch_files directory which is available,
and the number of directories exceeds scratch_files.numdirs_threshold,
then try to use the tdb library to store the inode count abstraction.
This allows us to check very large filesystems without needing as much
physical memory.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Change the iterator abstraction and replace e2fsck_get_dir_info() with
e2fsck_dir_info_{set,get}_{parent,dotdot} so that we can support an
on-disk dirinfo implementation. This allows e2fsck to check very large
filesystems on systems with smaller amounts of memory and/or address
space.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The e2fsprogs and kernel implementation of directory hash tree has a
bug which causes the implementation to be dependent on whether
characters are signed or unsigned. Platforms such as the PowerPC,
Arm, and S/390 have signed characters by default, which means that
hash directories on those systems are incompatible with hash
directories on other systems, such as the x86.
To fix this we add a new flags field to the superblock, and define two
new bits in that field to indicate whether or not the directory should
be signed or unsigned. If the bits are not set, e2fsck and fixed
kernels will set them to the signed/unsigned value of the currently
running platform, and then respect those bits when calculating the
directory hash. This allows compatibility with current filesystems,
as well as allowing cross-architectural compatibility.
Addresses Debian Bug: #389772
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
- EXT4_FEATURE_RO_COMPAT_HUGE_FILE (0x0008) - change i_blocks to be
in units of s_blocksize units instead of 512-byte sectors, use
l_i_frag and l_i_fsize as i_blocks_hi (could also be part of 64BIT).
E2fsck and debugfs changed to support i_blocks_hi instead of l_i_frag and
l_i_fsize.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Don't core dump if there is a corrupt htree interior node. If the block
number is larger than the number of blocks in the directory, don't write
past the end of malloc'ed memory.
Addresses SourceForge Bug: #1512778
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Change the format string(%d, %ld) for a block number and inode number
to %u or %lu.
Signed-off-by: Takashi Sato <sho@tnes.nec.co.jp>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
This was actually caused by two bugs. The first bug is that if the
inode has been fully fixed up, the code will attempt to remove the
inode from the inode_bad_map without checking to see if this bitmap is
present. Since it is cleared at the end of pass 2, if
e2fsck_process_bad_inode is called in pass 4 (as it is for
disconnected inodes), this would result in a core dump.
The first bug was mostly hidden by a second bug, which caused
e2fsck_process_bad_inode() to consider all inodes without an extended
attribute to be not fixed.
Note: This bug was introduced in e2fsprogs 1.36.
(Addresses Debian Bug: #316736)
incorrectly treat as valid symlinks created with SE Linux
(Debian bug #228723) as well as failing the f_journal test case on
big endian systems due to the backup journal blocks not being swapped.
a single directory block (because this is the easy case;
we don't currently check for duplicates that span
directory blocks, for now. Eventually for htree
directories we can do this by searching for all directory
blocks that have a hash overflow, and then searching the
adjacent blocks to find all other potential duplicates.)
When byte-swapping a filesystem on a PPC architecture, byte-swap
the bitmaps since the historical big-endian ext2 variant had
byte-swapped bitmaps, and the ext2fs library assumes this. Otherwise
the regression test suite will fail...
Add additional checks to HTREE directories. We now check the count
and limit fields in the htree header, as well as assuring that the
hash table in each interior node is in ascending order. We also
check to make sure all leaf nodes are have the expected depth in
the tree.
Updated test cases to deal with all of the above.
to determine whether or not a directory entry is a
completely empty leaf block or leaf node. Otherwise
e2fsck might get confused into thinking that a valid dxdir
was corrupted.
directories.
Speed up e2fsck slightly by only updating the master superblock;
there is no point to update the backup superblocks.
Fix a small bug in the rehashing code which could leave the indexed
flag set even after the directory was compressed instead of indexed.
(Not fatal, since the kernel will deal with this, but technically
it filesystem isn't consistent, and the filesystem will be marked
as being in error when the kernel comes across the directory. It
should also never happen in real life, since directories that small
will never be indexed, but better safe than sorry.)
Also change the threshold of when directories are indexed, so that
directories of size 2 blocks will be indexed. Otherwise they will
never be indexed by the kernel when they grow.
using a non-zero hash version (i.e., half MD4 or TEA hash).
The hash version wasn't getting copied into dx_dir->hashversion and
this caused the kernel to treat all directories if they were using the
legacy hash, which was Bad.
make sure we gracefully clean up and only exit at safe points.
For fsck, we pass the SIGINT/SIGTERM signal to the child processes,
so they can do their own cleanup.
Andreas's symlink code; check_blocks() was unconditionally testing
inode_bad_map without checking to see if it existed first. Fixed
problem a different way; we now no longer check inode_bad_map at all,
since the file might not get deleted in pass 2 anyway. We move the
large file feature reconciliation code to to e2fsck_pass2(), and in
deallocate_inode() in pass2.c, we decrement the large files counter if
we're about to delete a large file.
pass1.c (mark_inode_bad): Replace alloc_bad_map with a function which
sets the bit in the bad inode bitmap.
(e2fsck_pass1): Check for fast symlinks with an invalid size, and set
the bad inode map in that case.
pass2.c (e2fsck_process_bad_inode): Check for fast symlinks with an
invalid size and prompt the user if the inode should be cleared.
problem.h, problem.c (PR_2_SYMLINK_SIZE): Added new problem code.
pass1b.c: Change routines to use PR_1B_BLOCK_ITERATE when reporting
problems rather than using com_err directly.
problem.c, problem.h (PR_1B_BLOCK_ITERATE): Add new problem code.
message.c (expand_percent_expression): Add safety check. If ctx->str
is NULL, print "NULL" instead of dereferencing the null pointer.
pass1b.c, pass2.c, pass3.c: Change calls to ext2fs_block_iterate to
ext2fs_block_iterate2, to support 64-bit filesizes and to speed things
up slightly by avoiding the use of the ext2fs_block_iterate's
compatibility shim layer.
version.h:
Update for WIP release.
unix.c (main): If compression is enabled on the filesystem, print a
warning message (for now).
message.c: Add new compression shortcut: @c == compress
problem.c, problem.h (PR_1_COMPR_SET): Add new error code.
pass1.c (check_blocks): If the inode has EXT2_COMPRBLK_FL flag set,
check to see if the filesystem supports compression. If it does pass
this information down to process_block() so it can treat the
compressed block flag words correctly. If not, offer to clear the
flag, since it shouldn't be set.
(process_block): If an inode has the compressed inode flag set, allow
EXT2FS_COMPRESSED_BLKADDR.
pass1b.c (process_pass1b_block, delete_file_block, clone_file_block):
pass2.c (deallocate_inode_block): Use HOLE_BLKADDR to check to see if
the block can be skipped.
ChangeLog, Makefile.in:
Makefile.in: Exclude the internationalization files from being
distributed.
ChangeLog, configure, configure.in:
configure.in: Add support for --enable-compression. This is
experimental code only for now, which is why it's under --enable test.
Once it's stable, it will always be compiled in.
TODO:
Commit additional TODO items.
pass1.c (e2fsck_pass1): If the filesystem does not support imagic
inodes, if an inode has the imagic flag set, offer to clear the imagic
flag. If a valid device/fifo/socket has the immutable flag set, call
the new helper function check_immutable() to offerto clear the
immutable flag.
pass2.c (check_filetype): Use the new ext2_file_type() helper function
instead of calculating the file_type information manually.
pass3.c (e2fsck_reconnect_file): When adding a link to lost+found,
calculate the filetype information so that ext2fs_link() can use the
information if applicable. (get_lost_and_found): Create the
/lost+found directory with the correct filetype information if
applicable.
util.c (ext2_file_type), e2fsck.h: New function which returns the
directory entry file type information given the inode's mode bits.
problem.c, problem.h: Added new problem codes PR_1_SET_IMAGIC and
PR_1_SET_IMMUTABLE.
ChangeLog, mke2fs.8.in:
mke2fs.8.in: Update manual page so that the sparse_option filesystem
option is properly named.
problem.c, problem.h (PR_2_NULL_NAME): Add new problem code.
pass2.c (check_dir_block): Require that the length of the directory
entry be at least 12 bytes. Check to see if the filename is
zero-length, and flag that as an error.
pass2.c (check_filetype): If the filetype filesystem feature is not
set, and a directory entry has a dirent feature, offer to clear it
(since 2.0 kernels will do complain will interpret it as a very large
name length field).
pass4.c (e2fsck_pass4): If an inode is set in the inode_imagic_map
bitmap, don't check to see if it is disconnected from the inode tree
(because it almost certainly will be). Free inode_imagic_map at the
end of pass 4.
pass2.c (check_dir_block, check_filetype): If the FILETYPE feature is
set, check the directory entry's filetype information field, and
fix/set it if necessary. (e2fsck_pass2): Free the inode_reg_map
bitmap at the end of pass 2.
pass1.c (e2fsck_pass1, alloc_imagic_map): Allocate and fill in
information for inode_reg_map and inode_imagic_map, which indicates
which inodes are regular files and AFS inodes, respectively.
Since only the master superblock is written during a restart, force
that superblock to be used after a restart; otherwise changes to the
block group descriptors end up getting ignored.
problem.c, problemP.h: If e2fsck is run -n, make def_yn variable be 0
for "no". Add support for a new flag, PR_NO_NOMSG, which supresses
the problem message if e2fsck is run with the -n option.
problem.c, problem.h (PR_2_SET_FILETYPE, PR_2_BAD_FILETYPE): Add new
problem codes.
message.c (expand_dirent_expression): Add support for %dt which prints
the dirent type information.
e2fsck.c (e2fsck_reset_context): Free new bitmaps (inode_reg_map and
inode_imagic_map).
e2fsck.h (e2fsck_t): Add new inode_reg_map and inode_magic_map to the
context structure.
ChangeLog, nt_io.c:
nt_io.c: New file which supports I/O under Windows NT.
ChangeLog, gen_uuid_nt.c:
gen_uuid_nt.c: New file which creates a UUID under Windows NT.
Many files:
Add support for non-Unix compiles
unix.c: Add support for calculating a progress bar if the -C0 option
is given. The function e2fsck_clear_progbar() clears the progress bar
and must be called before any message is issued. SIGUSR1 will enable
the progress bar, and SIGUSR2 will disable the progress bar. This is
used by fsck to handle parallel filesystem checks. Also, set the
device_name from the filesystem label if it is available.
e2fsck.h: Add new flags E2F_FLAG_PROG_BAR and E2F_FLAG_PROG_SUPRESS.
Add new field in the e2fsck structure which contains the last tenth of
a percent printed for the user.
message.c (print_e2fsck_message): Add call to e2fsck_clear_progbar().
pass1.c (e2fsck_pass1):
pass2.c (e2fsck_pass2):
pass3.c (e2fsck_pass3):
pass4.c (e2fsck_pass4):
pass5.c (e2fsck_pass5): Add call to e2fsck_clear_progbar when printing
the resource tracking information.
pass5.c (check_block_bitmaps, check_inode_bitmaps): If there is an
error in the bitmaps, suppress printing the progress bar using the
suppression flag for the remainder of the check, in order to clean up
the display.
e2fsck.h:
pass1.c (pass1_get_blocks, pass1_read_inode, pass1_write_inode,
pass1_check_directory, e2fsck_use_inode_shortcuts): Make pass1_* be
private static functions, and create new function
e2fsck_use_inode_shortcuts which sets and clears the inode shortcut
functions in the fs structure.
e2fsck.h:
pass2.c (e2fsck_process_bad_inode): Make process_bad_inode() an
exported function.
pass4.c (e2fsck_pass4): Call e2fsck_process_bad_inode to check if a
disconnected inode has any problems before connecting it to
/lost+found. Bug and suggested fix by Pavel Machek <pavel@bug.ucw.cz>
ChangeLog, swapfs.c:
swapfs.c (ext2fs_swap_inode): Add compatibility for Linux 2.3 kernels
that use i_generation instead of i_version. Patch supplied by Jon
Bright <sircus@sircus.demon.co.uk>.
ChangeLog, mke2fs.8.in:
mke2fs.8.in: Fix typo in man page which caused the badblocks command
to not show up in the "SEE ALSO" section.
ChangeLog, expect.1, expect.2, image.gz, name:
f_recnect_bad: New test which checks the case where a disconnect inode
also bad inode fields; we need to make sure e2fsck offers to fix the
inode (or clear the inode, as necessary).
tune2fs.8.in: Fix minor display bug in the nroff.
mke2fs.c (show_stats, write_inode_tables): Use the log10 function to
calculate the display of block numbers so that things look nice on an
80 character display.
mke2fs.c (usage): Add the sparse-super-flag to the usage message.
ChangeLog, e2fsck.c, pass1.c, pass2.c, problem.c, problem.h, unix.c:
unix.c (main): Move ext2fs_close() after e2fsck_free_context() since
e2fsck_free_context may reference data in ctx->fs.
e2fsck.c (e2fsck_reset_context): Make sure ctx->fs is non-NULL before
checking ctx->fs->dblist.
pass1.c (e2fsck_pass1): Use the device check subroutine on FIFO's and
Socket's, so that we catch bogus immutable inodes.
pass2.c (process_bad_inode): Process bad socket and fifo's.
problem.h, problem.c: Define new problem codes PR_2_BAD_FIFO and
PR_2_BAD_SOCKET.
pass2.c (e2fsck_pass2): Fix the progress accounting so that we get to
100%.
pass3.c (e2fsck_pass3): Change progress accounting to be consistent
with the other e2fsck passes.
e2fsck.c (e2fsck_run): At the end of each pass, call the progress
function with the pass number set to zero.
unix.c (e2fsck_update_progress): If the pass number is zero, ignore
the call, since that indicates that we just want to deallocate any
progress structures.
emptydir.c:
Commit partially done file.
ChangeLog, badblocks.c:
badblocks.c (ext2fs_badblocks_list_add): Use a bigger increment than
10 blocks when we need to expand the size of the badblocks list.
unix.c: Fix bug in check of feature set, to make sure we can really
fix this filesystem.
problem.h: Make blkcount type to be of type blkcnt_t. Make the num
field be a 64 bit type. Add the problem code PR_1_FEATURE_LARGE_FILES
problem.c: Add table entry for the problem code PR_1_FEATURE_LARGE_FILES.
pass1.c (e2fsck_pass1): A non-zero i_dir_acl field is only a problem
for directory inodes. (Since it is also i_size_high now.) If there
are no large_files, then clear the LARGE_FLAG feature flag. If there
are large_files, but the LARGE_FLAG feature flag is not set, complain
and offer to fix it.
(check_blocks): Add support to deal with non-directory inodes that
have i_size_high set (i.e., large_files). Don't give an error if a
directory has preallocated blocks, to support the DIR_PREALLOC
feature.
(process_block, process_bad_block): The blockcnt variable is a type of
blkcnt_t, for conversion to the new block_iterate2.
pass2.c (process_bad_inode): A non-zero i_dir_acl field is only a
problem for directory inodes. (Since it is also i_size_high now.)
message.c (expand_inode_expression): Print a 64-bits of the inode size
for non-directory inodes. (Directory inodes can only use a 32-bit
directory acl size, since i_size_high is shared with i_dir_acl.) Add
sanity check so that trying to print out the directory acl on a
non-directory inode will print zero. (expand_percent_expression): %B
and %N, which print pctx->blkcount and pctx->num, can now be 64 bit
variables. Print them using the "%lld" format if EXT2_NO_64_TYPE is
not defined.
e2fsck.h: Add the large_flagsfield to the e2fsck context.
e2fsck.c (e2fsck_reset_context): Clear the large_flags field.
ChangeLog, expect.1:
f_messy_inode: Modify test to deal with changes to support 64-bit size
files. (/MAKEDEV had i_dir_acl, now i_size_high, set.)
Change the progress function to return an integer; if returns 1, then
the progress function is expected to have set the e2fsck context flag
signalling a user abort, and the caller should also initiate a user
abort.
e2fsck.h: If EXT2_FLAT_INCLUDES is defined, then assume all of
the ext2-specific header files are in a flat directory.
dirinfo.c, ehandler.c, pass1.c, pass1b.c, pass2.c, pass5.c,
super.c, swapfs.c, unix.c: Explicitly cast all assignments
from void * to be compatible with C++.
unix.c (sync_disk): Remove sync_disk and calls to that function,
since ext2fs_close() now takes care of this.
pass1.c, pass1b.c, pass2.c, pass3.c, swapfs, badblocks.c,
ehandler.c, unix.c: Change use of private to be priv_data, to
avoid C++ reserved name clash.
pass1.c, pass2.c, pass3.c, pass4.c, pass5.c: Add calls to the progress
indicator function.
pass1.c (scan_callback): Add call to the progress feedback function
(if it exists).
super.c (check_super_block): Skip the device size check if the
get_device_size returns EXT2_EXT_UNIMPLEMENTED.
iscan.c (main): Don't use fatal_error() anymore.
pass1b.c, swapfs.c, badblocks.c: Set E2F_FLAG_ABORT instead of calling
fatal_error(0).
problem.c, pass3.c (PR_3_ROOT_NOT_DIR_ABORT,
PR_3_NO_ROOT_INODE_ABORT): New problem codes.
problem.c, pass2.c (PR_2_SPLIT_DOT): New problem code.
problem.c, pass1.c (PR_1_SUPPRESS_MESSAGES): New problem code.
problemP.h: New file which separates out the private fix_problem data
structures.
util.c, dirinfo.c, pass1.c, pass1b.c, pass2.c, pass5.c, super.c,
swapfs.c util.c: allocate_memory() now takes a e2fsck context as its
first argument, and rename it to be e2fsck_allocate_memory().
problemP.h:
New file which contains the private problem abstraction definitions.
Makefile.pq:
Remove include of MAKEFILE.STD, which doesn't exist at this point.
Change ext2fs_read_inode, ext2fs_write_inode to take the e2fsck
context as their first argument.
Change dir_info.c routines to take an e2fsck_context, renamed them to
start with e2fsck_ to avoid namespace issues, and changed them to
store the directory information inside the e2fsck context.
Added e2fsck_run() which calls all of the e2fsck passes in the correct
order, and which handles the return of abort codes.
Added abort processing, both via setjmp/longjmp and via flags in the
e2fsck context.
Use a flag in the e2fsck context instead of the restart_e2fsck global
variable.
Change uses of free and malloc to ext2fs_free_mem and ext2fs_get_mem.
pass*.c, super.c: Massive changes to avoid using printf and com_err
routines. All diagnostic messages are now routed through the
fix_problem interface.
pass2.c (check_dir_block): Check for duplicate '.' and '..' entries.
problem.c, problem.h: Add new problem codes PR_2_DUP_DOT and
PR_2_DUP_DOT_DOT.
problem.c: Added new problem codes for some of the superblock
corruption checks, and for the pass header messages. ("Pass
1: xxxxx")
util.c (print_resource_track): Now takes a description argument.
super.c, unix.c, e2fsck.c: New files to separate out the
operating-specific operations out from e2fsck.c. e2fsck.c now
contains the global e2fsck context management routines, and
super.c contains the "pass 0" initial validation of the
superblock and global block group descriptors.
pass1.c, pass2.c, pass3.c, pass4.c, pass5.c, util.c: Eliminate
(nearly) all global variables and moved them to the e2fsck
context structure.
problem.c, problem.h: Added new problem codes PR_0_SB_CORRUPT,
PR_0_FS_SIZE_WRONG, PR_0_NO_FRAGMENTS, PR_0_BLOCKS_PER_GROUP,
PR_0_FIRST_DATA_BLOCK
expect.1, expect.2:
Updated tests to align with e2fsck problem.c changes.
message.c: Add compression for the word "Illegal"
problem.c: Added entries for PR_2_BAD_CHAR_DEV and PR_2_BAD_BLOCK_DEV
pass1.c (pass1, check_device_inode), pass2.c (process_bad_inode): Use
a more stringent test for a valid device.
ChangeLog, Makefile.in:
Makefile.in (install): Fix rm command to use $(DESTDIR) appropriately.
Theodore Ts'o