The existing undo file format (which is based on tdb) has many
problems. First, its comparison of superblock fields is ineffective,
since the last mount time is only written by the kernel, not the tools
(which means that undo files can be applied out of order, thus
corrupting the filesystem); block numbers are written in CPU byte
order, which will cause silent failures if an undo file is moved from
one type of system to another; using the tdb database costs us an
enormous amount of CPU overhead to maintain the key data structure,
and finally, the tdb database is unable to deal with databases larger
than 2GB. (Upstream tdb 1.2.12 can handle 4GB, but upgrading a 2TB FS
to 64bit,metadata_csum easily produces 2.9GB of undo files, so we
might as well move off of tdb now.)
The last problem is fatal if you want to use tune2fs to turn on
metadata checksumming, since that rewrites every block on the
filesystem, which can easily produce a many-gigabyte undo file, which
of course is unreadable and therefore the operation cannot be undone.
Therefore, rip all of that out in favor of writing to a flat file.
Old blocks are appended to a file and the index is written to the end
when we're done. This implementation is much faster than wasting a
considerable amount of time trying to maintain a hash index, which
drops the runtime overhead of tune2fs -O metadata_csum from ~45min
to ~20 seconds on a 2TB filesystem.
I have a few reasons that factored in my decision not to repurpose the
jbd2 file format for undo files. First, undo files are limited to
2^32 blocks (16TB) which some day might not serve us well. Second,
the journal block size is tied to the file system block size, but
mke2fs wants to be able to back up big chunks of old device contents.
This would require large changes to the e2fsck journal replay code,
which itself is derived from the kernel jbd2 driver, which I'd rather
not destabilize. Third, I want to require undo files to store the FS
superblock at the end of undo file creation so that e2undo can be
reasonably sure that an undo file is supposed to apply against the
given block device, and doing so would require changes to the jbd2
format. Fourth, it didn't seem like a good idea that external
journals should resemble undo files so closely.
v2: Provide a state bit that is only set when the undo channel is
closed correctly so we can warn the user about potentially incomplete
undo files. Straighten out the superblock handling so that undo files
won't be confused for real ext* FS images. Record multi-block runs in
each block key to reduce overhead even further. Support reopening an
undo file so that we can combine multiple FS operations into one
(overall smaller) transaction file, which will be easier to manage.
Flush the undo index data if the program should terminate
unexpectedly. Update the ext4 superblock bits if errors or -f is
found to encourage fsck to do a full run the next time it's invoked.
Enable undoing the undo.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Create a journal.c with routines adapted from e2fsck/journal.c to
handle opening and closing the journal, and setting up the
descriptors, and all that. Unlike e2fsck's versions which try to
identify and fix problems, the routines here have no way to repair
anything.
[ Modified by tytso to fold debugfs/jfs_user.h into e2fsck/jfs_user.h,
so we don't have to copy recovery.c and revoke.c into debugfs. --tytso ]
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Perform a little more sanity checking of EA value offsets so that we
don't crash while trying to load things from the filesystem.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Add a new behavior flag to the inode scan functions; when specified,
this flag will do some simple sanity checking of entire inode table
blocks. If all the checksums are ok, we can skip checksum
verification on individual inodes later on. If more than half of the
inodes look "insane" (bad extent tree root or checksum failure) then
ext2fs_get_next_inode_full() can return a special status code
indicating that what's in the buffer is probably garbage.
When e2fsck' inode scan encounters the 'inode is garbage' return code
it'll offer to zap the inode straightaway instead of trying to recover
anything. This replaces the previous behavior of asking to zap
anything with a checksum error (strict_csum).
Signed-off-by: Darrick J. Wong <darrick.wong@orale.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Add magic number checking to the extended attribute editing handle;
move inline data to the head of the attribute list when writing so
that inline data ends up in the inode area; and always zero the
attribute space before writing to ensure that we can delete the last
xattr.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Currently ext2fs_file_read/write are used to copy data from/to a file.
But they manipulate data by blocksize. For supporting inline data, we
handle it in two new fucntions called ext2fs_file_read/write_inline_data.
In read path the implementation is straightforward. But in write path
things get more complicated because if the size of data is greater than
the maximum size of inline data we will expand this file. So now we
will check this in ext2fs_inline_data_set. If this inode doesn't have
enough space, it will return EXT2_ET_INLINE_DATA_NO_SPACE error. Then
the caller will check this error and tries to expand the file.
The following commands in debugfs can handle inline_data feature after
applying this patch:
- dump
- cat
- rdump
- write
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Signed-off-by: Zheng Liu <wenqing.lz@taobao.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Inline_data is handled in dir iterator because a lot of commands use
this function to traverse directory entries in debugfs. We need to
handle inline_data individually because inline_data is saved in two
places. One is in i_block, and another is in ibody extended attribute.
After applied this commit, the following commands in debugfs can
support the inline_data feature:
- cd
- chroot
- link*
- ls
- ncheck
- pwd
- unlink
* TODO: Inline_data doesn't expand to ibody extended attribute because
link command doesn't handle DIR_NO_SPACE error until now. But if we
have already expanded inline data to ibody ea area, link command can
occupy this space.
Signed-off-by: Zheng Liu <wenqing.lz@taobao.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Add functions to allow clients to get, set, and remove extended
attributes from any file. It also supports modifying EAs living in
i_file_acl.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Tweak the wording to be a little less ambiguous, since 'block' can be
a noun or a verb.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
An inode with inline data has no data blocks, so we can not iterate
over such an inode. Return an error code which indicates this fact;
callers can use this to determine whether or not the inode has inline
data, and then call some routine to iterate over the directory intries
in the line data or read the inline data, as appropriate.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Apparently libext2fs didn't have an error code defined for block
bitmap checksum errors, so add one.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Reviewed-by: Lukas Czerner <lczerner@redhat.com>
When we define an error in lib/ext2fs/ext2_err.et.in, we will always use
EXT2_ET_* prefix for a new error. But EXT2_NO_MTAB_FILE doesn't obey
this rule. So fix it.
Signed-off-by: Zheng Liu <wenqing.lz@taobao.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Creating symlinks is a complex affair when accounting for slowlinks.
Create a new function, ext2fs_symlink(), modeled after ext2fs_mkdir().
Like ext2fs_mkdir(), ext2fs_symlink() takes on the task of allocating a
new inode and block (for slowlinks), setting up sane default values in
the inode, copying the target path to either the inode (for fastlinks)
or to the first block (for slowlinks), and accounting for the inode and
block stats. Disallow link targets longer than blocksize as the Linux
kernel prevents this.
It does not attempt to expand the parent directory, instead returning
EXT2_ET_DIR_NO_SPACE and leaving it to the caller to expand just as
ext2fs_mkdir() does. Ideally, I think both of these functions should
make a single attempt to expand the directory.
[ Fixed a few bugs discovered when creating a test case for ext2fs_symlink() ]
Signed-off-by: Darren Hart <dvhart@infradead.org>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Cc: "Darrick J. Wong" <darrick.wong@oracle.com>
Cc: Andreas Dilger <adilger@dilger.ca>
To maintain the error codes numbering, we need to pull in the changes
from the 1.43.x development branch for the libext2's error table.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Multi-mount protection is feature that allows mke2fs, e2fsck, and
others to detect if the filesystem is mounted on a remote node (on
SAN disks) and avoid corrupting the filesystem. For e2fsprogs this
means that it checks the MMP block to see if the filesystem is in use,
and marks the filesystem busy while e2fsck is running on the system.
This is useful on SAN disks that are shared between high-availability
servers, or accessible by multiple nodes that aren't in HA pairs. MMP
isn't intended to serve as a primary HA exclusion mechanism, but as a
failsafe to protect against user, software, or hardware errors.
There is no requirement that e2fsck updates the MMP block at regular
intervals, but e2fsck does this occasionally to provide useful
information to the sysadmin in case of a detected conflict.
For the kernel (since Linux 3.0) MMP adds a "heartbeat" mechanism to
periodically write to disk (every few seconds by default) to notify
other nodes that the filesystem is still in use and unsafe to modify.
Originally-by: Kalpak Shah <kalpak@clusterfs.com>
Signed-off-by: Johann Lombardi <johann@whamcloud.com>
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Initial design was done by Theodore Ts'o; implementation was fleshed
out by Valerie Aurora Henson. Also includes some fixes from Nick Dokos.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Signed-off-by: Valerie Aurora Henson <vaurora@redhat.com>
Signed-off-by: Nick Dokos <nicholas.dokos@hp.com>
To reduce user confusion, if the /etc/mtab file is missing
ext2fs_check_mount_point and ext2fs_check_if_mounted will return a
new, explicit error code to indicate this case.
Addresses-Debian-Bug: #527859
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Provide a C language wrapper function for io_channel_read_blk64() and
io_channel_write_blk64() instead of using a C preprocessor macro, with
an fallback to the old 32-bit functions if an application-provided I/O
channel manager doesn't supply 64-bit method functions and the block
numbers can fit in 32-bit integer.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
When called for a given handle, the new function extent_node_split()
will split the current node such that half of the node's entries will
be moved to a new tree block. The parent will then be updated to
point to the (now smaller) original node as well as the new node.
If the root node is requested to be split, it will move all
entries out to a new node, and leave a single entry in the
root pointing to that new node.
If the reqested split node's parent is full it will recursively
split up to the root to make room for the new node's insertion.
If you ask to split a non-root node with only one entry,
it will refuse (we'd have an empty node otherwise).
It also updates the i_blocks count when a new block has
successfully been connected to the tree.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Add two new functions which allows the caller to examine the last
directory block entry added to the list, and to drop if it necessary.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
This flag allows the caller to promise that it will not try to modify
the block numbers returned by the iterator.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
any debugging statements from within library code (always a bad idea), and
ext2fs_create_resize_inode() will return a proper error code if the
resize inode is corrupt, instead of returning -1.
large number of inodes, then reduce the number of blocks
per group until we find a workable set of filesystem
parameters.
ext2_err.et.in (EXT2_ET_TOO_MANY_INODES): Add new error code.
an I/O object.
Export ext2_file_flush as a public interface.
Also minor cleanups to tighten code in other I/O abstractions, and to
mark a void * pointer as const in the ext2_file_write interface.
mkjournal.c (ext2fs_create_journal_superblock): Add safety check;
return an error if there's an attempt to create a journal less than
1024 filesystem blocks.
ext2_err.et.in, mkjournal.c: Change EXT2_JOURNAL_NOT_BLOCK and
EXT2_NO_JOURNAL_SB to be EXT2_ET_*.
initialize.c (ext2fs_initialize): Add support for initializing the
ext2 superblock for external journal devices. This basically means we
don't bother to allocate any block group descriptors.
openfs.c (ext2fs_open): Only open external journal devices if the new
flag EXT2_FLAG_JOURNAL_DEV_OK is passed to ext2fs_open. When opening
such devices, don't try to read the block group descriptors, since
they're not there.
ext2_err.et.in (EXT2_NO_JOURNAL_SB): Add new error code
mkjournal.c: Export a new function,
ext2fs_create_journal_superblock(), which allocates and returns a
buffer containing a journal superblock. This is needed by mke2fs to
create an external journal. Rewrote ext2fs_add_journal_device() so
that it no longer creates the external journal, but rather adds a
filesystem to an existing external journal. It handles all of the
UUID manipulation.
ext2fs.h: List the EXT3_FEATURE_JOURNAL_DEV as a flag supported by the
library. Define the EXT2_FLAG_JOURNAL_DEV_OK. Changed function
prototype for ext2fs_add_journal_device().
imager.c (ext2fs_image_{inode,super,bitmap}_{read,write}, ext2_fs.h,
Makefile.in: New file that has routines that save ext2fs metadata to a
file.
ext2_err.et.in (EXT2_ET_MAGIC_E2IMAGE): New error code assigned.
e2image.h: New file which defines the file format for the ext2 image
file. (Saved copy of ext2 metadata to a file as a saving throw
against worst-case damage.)
ChangeLog, Makefile.in, e2image.c:
e2image.c, Makefile.in: New program which saves ext2 metadata to a
file for people who need a last-ditch saving throw.
ext2fs.h: Add new superblock fields (s_algorithm_usage_bitmap,
s_prealloc_blocks, s_prealloc_dir_blocks). Added conditional defines
of new features COMPAT_DIR_PREALLOC, RO_COMPAT_LARGE_FILE
RO_COMPAT_BTREE_DIR, INCOMPAT_COMPRESSION, INCOMPAT_DIRNAME_SIZE.
Changed the library to declare that we support COMPAT_DIR_PREALLOC,
INCOMPAT_DIRNAME_SIZE, RO_COMPAT_LARGE_FILE.
fileio.c: Rename function ext2fs_file_llseek to be ext2fs_file_lseek,
which is more accurate.
block.c: Add new function ext2fs_block_iterate3 which calls the
iterator function with the blockcount argument of type blkcnt_t. This
version of the function is allowed to handle large files; the other
fucntions are not.
ext2fs.h: Add new type blkcnt_t
ext2_err.et.in: Add error code EXT2_ET_FILE_TOO_BIG
block.c (ext2fs_block_iterate2): Fix bug where the block count field
wasn't getting correctly incremented for sparse files when the
indirect or doubly-indirect block specified in the inode was zero.
unlink.c (unlink_proc):
lookup.c (lookup_proc):
link.c (link_proc):
get_pathname.c (get_pathname_proc):
dir_iterate.c (ext2fs_process_dir_block): Mask off high 8 bits from
dirent->name_len, so it can be used for other purposes.
ext2fs.h: Add definition of EXT2_FEATURE_INCOMPAT_DIRNAME_SIZE, and
indicate that we have support for this incompatible option.
Makefile.in, tst_getsize.c: Added new file which is used to test the
ext2fs_get_device_size function.
ext2_err.et.in (EXT2_ET_UNIMPLEMENTED): Added new error code.
Rename new error codes to _ET_ in them for consistency.
ChangeLog, et_c.awk, et_h.awk:
Remove support for non STDC compilers, since the workarounds caused
problems with the header file.
Makefile.pq:
Checkpoint of powerquest work.
Darrick J. Wong