-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 07:23:42 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg302-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:23:48 (1787829828) === oleg302-client.virtnet: executing check_config_client /mnt/lustre oleg302-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg302-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a5650666800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a5650666800.idle_timeout=debug disable quota as required oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:23:59 (1787829839) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8226] == sanity-flr test 0a: lfs mirror create with -N option == 07:23:59 (1787829839) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:24:03 (1787829843) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:24:04 (1787829844) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:24:05 (1787829845) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:24:11 (1787829851) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:24:12 (1787829852) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 07:24:13 (1787829853) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9a5650666800.stats=clear osc.lustre-OST0001-osc-ffff9a5650666800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:24:15 (1787829855) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:24:18 (1787829858) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:24:21 (1787829861) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:24:22 (1787829862) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:24:24 (1787829864) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 07:24:27 (1787829867) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:24:29 (1787829869) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:24:31 (1787829871) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:24:33 (1787829873) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:24:37 (1787829877) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:24:39 (1787829879) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.218698 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248521 s, 42.2 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:24:41 (1787829881) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0419826 s, 99.9 MB/s PASS 22 (3s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:24:44 (1787829884) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:24:47 (1787829887) Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a5650666800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a5650666800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a5650666800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a5650666800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg302-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (23s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:25:10 (1787829910) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.249022 s, 46.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a5650666800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a5650666800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (13s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:25:23 (1787829923) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a5650666800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a5650666800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a5650666800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a5650666800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg302-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:25:46 (1787829946) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.679537 s, 77.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 07:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.217s user 0m0.000s sys 0m0.132s Stopping /mnt/lustre-ost1 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a5650666800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a5650666800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.240s user 0m0.002s sys 0m0.137s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg302-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg302-server oleg302-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a5650666800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a5650666800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.358s user 0m0.000s sys 0m0.165s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg302-server: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg302-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg302-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a5650666800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a5650666800.ost_server_uuid in FULL state after 0 sec PASS 33b (35s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:26:21 (1787829981) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:26:22 (1787829982) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:26:24 (1787829984) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 07:26:25 (1787829985) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 07:26:28 (1787829988) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.210232 s, 59.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:26:34 (1787829994) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100642 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00976835 s, 107 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:26:38 (1787829998) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:26:42 (1787830002) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000755477 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000756318 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000771649 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000582189 s, 1.8 MB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 07:26:47 (1787830007) create test files with size 15741557 .. 8f30ef4fc67101803bf4ee4cd9afe076 - cded0af237f6813c063fb1c4453637b0 - 00ee1a910338d64538e6768138f86b27 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg302-client.virtnet /mnt/lustre (opts:) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre PASS 37 (9s) == sanity-flr test 38: resync ============================ 07:26:56 (1787830016) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:27:02 (1787830022) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:27:06 (1787830026) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378734 s, 27.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152853 s, 68.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 07:27:10 (1787830030) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.031557 s, 66.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.112245 s, 37.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0500447 s, 41.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 07:27:15 (1787830035) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 07:27:17 (1787830037) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:27:18 (1787830038) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156669 s, 66.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 43c: read prefer uncompressed mirror == 07:27:22 (1787830042) llite.lustre-ffff9a5643a6a000.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (4s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 07:27:26 (1787830046) llite.lustre-ffff9a5643a6a000.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136108 s, 77.0 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013952 s, 75.2 MB/s **verify components PASS 43d (3s) == sanity-flr test 44a: lfs mirror split check =========== 07:27:29 (1787830049) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.102149 s, 30.8 MB/s PASS 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:27:34 (1787830054) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:27:39 (1787830059) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.128174 s, 81.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0xe:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg302-client.virtnet /mnt/lustre (opts:) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (6s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:27:45 (1787830065) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9a5650376000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.182095 s, 57.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg302-client.virtnet /mnt/lustre (opts:) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9a5660bc4000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:27:58 (1787830078) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (25s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:28:23 (1787830103) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (17s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:28:40 (1787830120) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 07:28:41 (1787830121) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.114038 s, 27.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0839097 s, 37.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6a6f6dfc (nosync,stale) Mirror 2: 0x6a6f6dfc (nosync,stale) ** verify mirror 2 doesn't change original checksum: 8ae6a1eff474654a8e019b1585915a3d - mirror 1 checksum: 29c28e16366284814e1647293611645f - mirror 2 checksum: 8ae6a1eff474654a8e019b1585915a3d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T07:28:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T07:28:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8ae6a1eff474654a8e019b1585915a3d - mirror 1 checksum: 29c28e16366284814e1647293611645f - mirror 2 checksum: 29c28e16366284814e1647293611645f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T07:28:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T07:28:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 07:28:46 (1787830126) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016305 s, 64.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:28:49 (1787830129) Failing mds1 on oleg302-server Stopping /mnt/lustre-mds1 (opts:) on oleg302-server 07:28:57 (1787830137) shut down facet: mds1 facet_host: oleg302-server facet_failover_host: oleg302-server Failover mds1 to oleg302-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0000 07:29:14 (1787830154) targets are mounted 07:29:14 (1787830154) facet_failover done oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260805 s, 40.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 50A (34s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:29:24 (1787830164) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0183669 s, 1.1 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00755723 s, 2.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:29:28 (1787830168) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0287794 s, 712 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00501225 s, 4.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:29:32 (1787830172) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00374004 s, 4.4 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:29:36 (1787830176) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0350726 s, 584 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:29:40 (1787830180) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00291473 s, 1.4 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0141503 s, 289 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:29:44 (1787830184) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:29:55 (1787830195) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:30:16 (1787830216) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:30:35 (1787830235) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:30:54 (1787830254) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 07:30:58 (1787830258) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 58732 mirror split pid 58733 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58732 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58733 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 07:32:01 (1787830321) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 322 fallocating to largest ever: 0x20ef0 000001 1787830322.684118 fallocate 0x0152ac thru 0x020eef (0xbc44 bytes) skipping zero size read 000004 1787830322.695356 MIRROR_EXTEND to 2 mirrors skipping zero size read skipping zero size read skipping zero length punch hole skipping zero size read 000009 1787830322.991349 mapwrite 0x004fb7 thru 0x012dbb (0xde05 bytes) 000010 1787830323.106738 mapread 0x0122a1 thru 0x012dbb (0xb1b bytes) 000011 1787830323.107665 mapread 0x005aae thru 0x00dce7 (0x823a bytes) 000012 1787830323.108499 trunc from 0x012dbc to 0x0028e2 (0x104d9 bytes) 000014 1787830323.128686 punch from 0x001304 to 0x0028e2 (0x15df bytes) 000015 1787830323.132955 read 0x0028a1 thru 0x0028e2 (0x42 bytes) 000016 1787830323.138705 mapread 0x000554 thru 0x0028e2 (0x238f bytes) 000017 1787830323.145358 mapwrite 0x0102e9 thru 0x0117a2 (0x14ba bytes) 000018 1787830323.174210 punch from 0x0057b8 to 0x00d643 (0x7e8c bytes) 000019 1787830323.178658 trunc from 0x006fa5 to 0x00cbe8 (0x5c44 bytes) 000020 1787830323.185131 mapread 0x0055d4 thru 0x00dd29 (0x8756 bytes) 000021 1787830323.194346 trunc from 0x00528a to 0x00573c (0x4b3 bytes) 000022 1787830323.198868 mapread 0x00dbeb thru 0x0117a2 (0x3bb8 bytes) 000023 1787830323.200809 trunc from 0x008140 to 0x00814b (0xc bytes) 000024 1787830323.204399 fallocate 0x00a8e1 thru 0x01885f (0xdf7f bytes) 000025 1787830323.207920 trunc from 0x0117a3 to 0x0098c0 (0x7ee2 bytes) 000026 1787830323.221077 write 0x023ad7 thru 0x0245ae (0xad8 bytes) 000027 1787830323.226636 MIRROR_RESYNC 2 mirrors 000028 1787830323.606732 MIRROR_EXTEND to 3 mirrors 000029 1787830324.030632 write 0x007139 thru 0x01630c (0xf1d4 bytes) 000030 1787830324.066694 read 0x013fb8 thru 0x019039 (0x5082 bytes) 000031 1787830324.070875 read 0x002302 thru 0x011d6a (0xfa69 bytes) 000032 1787830324.078865 trunc from 0x000b8a to 0x0091b7 (0x862e bytes) 000033 1787830324.089471 trunc from 0x003012 to 0x011da2 (0xed91 bytes) 000034 1787830324.100648 MIRROR_RESYNC 3 mirrors 000036 1787830324.664030 punch from 0x020792 to 0x0245ae (0x3e1d bytes) 000037 1787830324.687048 write 0x007533 thru 0x00ca81 (0x554f bytes) 000038 1787830324.698245 mapwrite 0x024c9d thru 0x0284ec (0x3850 bytes) 000039 1787830324.784757 trunc from 0x0284ed to 0x00aa22 (0x1daca bytes) 000040 1787830324.805576 write 0x03f1c6 thru 0x03ffff (0xe3a bytes) 000041 1787830324.808111 read 0x036f1a thru 0x03fab4 (0x8b9b bytes) 000042 1787830324.825629 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x229c7 000043 1787830325.249417 trunc from 0x040000 to 0x0229c6 (0x1d639 bytes) 000044 1787830325.277637 read 0x00bbc5 thru 0x016455 (0xa891 bytes) 000045 1787830325.288466 read 0x005922 thru 0x008614 (0x2cf3 bytes) 000048 1787830325.298544 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x229c7): Mirror 2: 0x9dd73e06 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x229c7): Mirror 2: 0x9dd73e06 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x229c7: skipped truncating to largest ever: 0x33083 000049 1787830325.474284 trunc from 0x0229c7 to 0x033082 (0x106bc bytes) 000050 1787830325.486811 trunc from 0x033083 to 0x021dd5 (0x112ad bytes) 000051 1787830325.504948 write 0x00e718 thru 0x01b328 (0xcc11 bytes) 000052 1787830325.513978 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000053 1787830325.616305 trunc from 0x021dd6 to 0x020292 (0x1b43 bytes) 000055 1787830325.632802 trunc from 0x020293 to 0x01e1d1 (0x20c1 bytes) 000056 1787830325.649351 fallocate 0x00dde6 thru 0x01866c (0xa887 bytes) 000057 1787830325.652626 fallocate 0x02465c thru 0x026147 (0x1aec bytes) 000059 1787830325.657766 mapwrite 0x0133db thru 0x017f8e (0x4bb4 bytes) 000061 1787830325.678729 read 0x00346a thru 0x005c7a (0x2811 bytes) 000062 1787830325.682876 mapwrite 0x030c5a thru 0x03ffff (0xf3a6 bytes) 000063 1787830325.720860 punch from 0x02eb35 to 0x032839 (0x3d05 bytes) 000064 1787830325.723925 punch from 0x031ba2 to 0x03bfde (0xa43d bytes) truncating to largest ever: 0x36e00 000065 1787830325.727014 trunc from 0x040000 to 0x036dff (0x9200 bytes) 000066 1787830325.736088 trunc from 0x016a13 to 0x01f28e (0x887c bytes) 000067 1787830325.739737 mapwrite 0x01608d thru 0x018f88 (0x2efc bytes) 000068 1787830325.760059 mapread 0x00030c thru 0x00bdb6 (0xbaab bytes) 000069 1787830325.768029 trunc from 0x02534a to 0x032458 (0xd10f bytes) 000070 1787830325.770932 trunc from 0x015b1e to 0x018420 (0x2903 bytes) 000071 1787830325.774281 write 0x018928 thru 0x01c62f (0x3d08 bytes) 000072 1787830325.778193 mapread 0x001397 thru 0x007e07 (0x6a71 bytes) 000073 1787830325.779035 write 0x008039 thru 0x0157eb (0xd7b3 bytes) 000074 1787830325.782851 read 0x01c913 thru 0x029a46 (0xd134 bytes) 000075 1787830325.790989 trunc from 0x02ed7e to 0x036dff (0x8082 bytes) 000076 1787830325.794062 punch from 0x00fc2a to 0x01e4a2 (0xe879 bytes) 000077 1787830325.802721 mapwrite 0x007ec0 thru 0x00f559 (0x769a bytes) 000078 1787830325.817943 write 0x014e52 thru 0x01e39b (0x954a bytes) 000079 1787830325.828345 mapread 0x01d461 thru 0x02030e (0x2eae bytes) 000080 1787830325.829224 fallocate 0x00811f thru 0x01435a (0xc23c bytes) 000081 1787830325.842227 MIRROR_EXTEND to 2 mirrors 000082 1787830326.234344 trunc from 0x036e00 to 0x024665 (0x1279a bytes) 000083 1787830326.257912 read 0x005f3e thru 0x006de9 (0xeac bytes) 000084 1787830326.262446 trunc from 0x0125f7 to 0x014e66 (0x2870 bytes) 000086 1787830326.266118 punch from 0x00b65b to 0x017a75 (0xc41b bytes) 000087 1787830326.268919 mapwrite 0x027834 thru 0x0372ec (0xfab9 bytes) 000089 1787830326.314038 trunc from 0x0372ed to 0x0336a1 (0x3c4b bytes) 000090 1787830326.325942 trunc from 0x0336a2 to 0x01ad93 (0x1890e bytes) 000091 1787830326.335837 trunc from 0x01614c to 0x01ad93 (0x4c48 bytes) fallocating to largest ever: 0x38501 000092 1787830326.343093 fallocate 0x033ebb thru 0x038500 (0x4646 bytes) 000093 1787830326.346463 trunc from 0x00c386 to 0x017e9e (0xbb19 bytes) 000094 1787830326.349414 write 0x00ae8d thru 0x01130a (0x647e bytes) 000095 1787830326.354753 punch from 0x01fbb5 to 0x024653 (0x4a9f bytes) 000096 1787830326.358503 trunc from 0x038501 to 0x0203fe (0x18102 bytes) 000097 1787830326.369245 write 0x0085a4 thru 0x014b25 (0xc582 bytes) 000098 1787830326.376040 mapread 0x015110 thru 0x0203fe (0xb2ef bytes) 000100 1787830326.397674 trunc from 0x0203ff to 0x02291a (0x251c bytes) 000101 1787830326.409650 fallocate 0x02c5af thru 0x030383 (0x3dd5 bytes) 000102 1787830326.414397 punch from 0x014296 to 0x0153ca (0x1135 bytes) 000104 1787830326.423042 read 0x010228 thru 0x01882b (0x8604 bytes) 000105 1787830326.428347 write 0x0366e5 thru 0x03ffff (0x991b bytes) 000106 1787830326.429609 fallocate 0x02cbbc thru 0x0329f4 (0x5e39 bytes) 000107 1787830326.433022 read 0x01714d thru 0x01ce13 (0x5cc7 bytes) 000108 1787830326.435992 MIRROR_RESYNC 2 mirrors 000109 1787830326.734619 fallocate 0x02278c thru 0x027afc (0x5371 bytes) 000110 1787830326.764283 read 0x03a66d thru 0x03ffff (0x5993 bytes) 000111 1787830326.769515 punch from 0x01c1be to 0x028412 (0xc255 bytes) 000112 1787830326.774769 MIRROR_RESYNC 2 mirrors 000113 1787830327.083750 punch from 0x01d887 to 0x02ba62 (0xe1dc bytes) 000114 1787830327.101507 write 0x0290f2 thru 0x02bb3f (0x2a4e bytes) 000115 1787830327.105488 punch from 0x03c08a to 0x03ffff (0x3f76 bytes) 000116 1787830327.109434 punch from 0x01868f to 0x01dc41 (0x55b3 bytes) 000117 1787830327.112345 write 0x02ac88 thru 0x0309b0 (0x5d29 bytes) 000118 1787830327.116097 trunc from 0x040000 to 0x017eb4 (0x2814b bytes) 000119 1787830327.134898 MIRROR_RESYNC 2 mirrors 000120 1787830327.372302 trunc from 0x017eb5 to 0x00a5e7 (0xd8cd bytes) 000121 1787830327.391256 mapread 0x005751 thru 0x00583e (0xee bytes) 000122 1787830327.394683 trunc from 0x0032ac to 0x00a5e7 (0x733c bytes) 000123 1787830327.402341 punch from 0x006784 to 0x00a5e7 (0x3e64 bytes) 000124 1787830327.406101 trunc from 0x00a5e8 to 0x00e913 (0x432c bytes) 000125 1787830327.416096 trunc from 0x00e914 to 0x030b98 (0x22285 bytes) 000126 1787830327.425721 read 0x010084 thru 0x01f957 (0xf8d4 bytes) 000127 1787830327.437274 read 0x021d3d thru 0x0304c1 (0xe785 bytes) 000128 1787830327.444962 write 0x01c166 thru 0x026dee (0xac89 bytes) 000129 1787830327.446347 trunc from 0x003615 to 0x00d85b (0xa247 bytes) 000130 1787830327.449634 read 0x014c01 thru 0x01b7a1 (0x6ba1 bytes) 000131 1787830327.450254 read 0x00c04b thru 0x01a3cd (0xe383 bytes) 000132 1787830327.456406 fallocate 0x032866 thru 0x033235 (0x9d0 bytes) 000133 1787830327.460271 mapread 0x02b8ae thru 0x03315a (0x78ad bytes) 000136 1787830327.464720 mapwrite 0x016618 thru 0x024a48 (0xe431 bytes) 000138 1787830327.476695 read 0x001ebe thru 0x00f423 (0xd566 bytes) 000139 1787830327.477297 trunc from 0x009038 to 0x00b70b (0x26d4 bytes) fallocating to largest ever: 0x40000 000141 1787830327.480811 fallocate 0x033d49 thru 0x03ffff (0xc2b7 bytes) 000142 1787830327.483801 mapread 0x00ac9e thru 0x00d8e3 (0x2c46 bytes) 000143 1787830327.486883 trunc from 0x0164af to 0x0243ae (0xdf00 bytes) 000144 1787830327.489927 mapread 0x00d278 thru 0x018a3b (0xb7c4 bytes) 000145 1787830327.495661 trunc from 0x033236 to 0x01377a (0x1fabb bytes) 000147 1787830327.503245 mapread 0x004b60 thru 0x01377a (0xec1b bytes) 000148 1787830327.504113 read 0x0041f9 thru 0x00994f (0x5757 bytes) 000149 1787830327.504542 punch from 0x0066cd to 0x008a66 (0x239a bytes) 000150 1787830327.508967 MIRROR_RESYNC 2 mirrors 000151 1787830327.720063 fallocate 0x03484b thru 0x034fa6 (0x75c bytes) 000152 1787830327.738571 fallocate 0x034b8a thru 0x035ccc (0x1143 bytes) 000153 1787830327.741736 fallocate 0x03b504 thru 0x03c132 (0xc2f bytes) 000154 1787830327.746992 fallocate 0x01f14c thru 0x020930 (0x17e5 bytes) 000155 1787830327.749866 fallocate 0x01c722 thru 0x02b677 (0xef56 bytes) 000156 1787830327.752552 write 0x004652 thru 0x00959a (0x4f49 bytes) 000158 1787830327.761710 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000159 1787830327.840823 mapread 0x00a89d thru 0x00dd64 (0x34c8 bytes) 000160 1787830327.850412 mapwrite 0x03a938 thru 0x03ffff (0x56c8 bytes) 000161 1787830327.887573 fallocate 0x02694b thru 0x0270aa (0x760 bytes) 000162 1787830327.891806 fallocate 0x01d80c thru 0x026753 (0x8f48 bytes) 000164 1787830327.895435 mapwrite 0x0129d4 thru 0x013a5d (0x108a bytes) 000165 1787830327.903532 read 0x001d75 thru 0x009c50 (0x7edc bytes) 000167 1787830327.907406 trunc from 0x01262e to 0x014f79 (0x294c bytes) 000169 1787830327.910589 trunc from 0x040000 to 0x02a386 (0x15c79 bytes) 000170 1787830327.918954 trunc from 0x00e0e4 to 0x0161da (0x80f7 bytes) 000171 1787830327.925325 MIRROR_EXTEND to 2 mirrors 000172 1787830328.214607 write 0x011d37 thru 0x01a17e (0x8448 bytes) 000173 1787830328.243518 fallocate 0x009b54 thru 0x015e14 (0xc2c1 bytes) 000174 1787830328.249346 read 0x004bee thru 0x012308 (0xd71b bytes) 000175 1787830328.255952 fallocate 0x032a2f thru 0x03cd3e (0xa310 bytes) 000176 1787830328.262204 trunc from 0x03b81d to 0x03c266 (0xa4a bytes) 000177 1787830328.265970 trunc from 0x02781a to 0x02c4be (0x4ca5 bytes) 000178 1787830328.270876 trunc from 0x03cd3f to 0x000782 (0x3c5bc bytes) 000179 1787830328.289551 punch from 0x0001e9 to 0x000782 (0x59a bytes) 000182 1787830328.293139 punch from 0x0005e9 to 0x000782 (0x19a bytes) 000183 1787830328.296909 write 0x02cd60 thru 0x030f04 (0x41a5 bytes) 000184 1787830328.298059 read 0x00517c thru 0x00884e (0x36d3 bytes) 000186 1787830328.304213 mapread 0x0303df thru 0x030f04 (0xb26 bytes) 000187 1787830328.304947 trunc from 0x01da97 to 0x026426 (0x8990 bytes) 000188 1787830328.307809 punch from 0x007ac1 to 0x00d82a (0x5d6a bytes) 000189 1787830328.315919 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30f05): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f05): Mirror 2: 0x56c3cd83 (stale) CRC-32 checksum value for chunk [0, 0x30f05): Mirror 1: 0xc9a8d342 Mirror 3: 0xc9a8d342 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f05: skipped 000190 1787830328.632810 mapwrite 0x00ea99 thru 0x012b7e (0x40e6 bytes) 000191 1787830328.686521 mapread 0x01f4a7 thru 0x022ced (0x3847 bytes) 000192 1787830328.692483 trunc from 0x030f05 to 0x0139f8 (0x1d50c bytes) 000193 1787830328.728644 punch from 0x013979 to 0x0139f8 (0x80 bytes) 000194 1787830328.734723 trunc from 0x01003d to 0x0139f8 (0x39bc bytes) 000195 1787830328.741778 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x139f9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x139f9): Mirror 2: 0x53457cd1 (stale) Mirror 3: 0xa824c622 (stale) CRC-32 checksum value for chunk [0, 0x139f9): Mirror 1: 0x573b821a Mirror 4: 0x573b821a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x139f9: skipped 000196 1787830329.118624 write 0x02ea06 thru 0x03a415 (0xba10 bytes) 000197 1787830329.137080 mapread 0x022c8b thru 0x031677 (0xe9ed bytes) 000198 1787830329.151356 trunc from 0x03a416 to 0x021922 (0x18af3 bytes) 000199 1787830329.173536 mapwrite 0x002c8f thru 0x01212b (0xf49d bytes) 000200 1787830329.196796 mapwrite 0x000469 thru 0x004b6f (0x4707 bytes) 000202 1787830329.208562 mapread 0x01f022 thru 0x021922 (0x2901 bytes) 000203 1787830329.209407 mapread 0x013fdb thru 0x01e074 (0xa09a bytes) 000204 1787830329.213483 trunc from 0x021923 to 0x013661 (0xe2c1 bytes) 000205 1787830329.225315 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x13662): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13662): Mirror 2: 0x4f4f2b69 (stale) Mirror 3: 0xbc57aeea (stale) Mirror 4: 0x316769d (stale) CRC-32 checksum value for chunk [0, 0x13662): Mirror 1: 0xbffc261b Mirror 5: 0xbffc261b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13662: skipped 000206 1787830329.683785 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x13662): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13662): Mirror 2: 0x4f4f2b69 (stale) Mirror 3: 0xbc57aeea (stale) Mirror 4: 0x316769d (stale) CRC-32 checksum value for chunk [0, 0x13662): Mirror 1: 0xbffc261b Mirror 5: 0xbffc261b Mirror 6: 0xbffc261b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13662: skipped 000207 1787830330.125859 write 0x03142a thru 0x033fab (0x2b82 bytes) 000208 1787830330.156839 read 0x032fd5 thru 0x033fab (0xfd7 bytes) 000209 1787830330.157506 punch from 0x01d82a to 0x01d839 (0x10 bytes) 000210 1787830330.164715 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33fac): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fac): Mirror 2: 0x3dc7e6fe (stale) Mirror 3: 0xa5c9d6af (stale) Mirror 4: 0xbf333592 (stale) Mirror 5: 0xd45e829c (stale) Mirror 6: 0xd45e829c (stale) CRC-32 checksum value for chunk [0, 0x33fac): Mirror 1: 0xbef88225 Mirror 7: 0xbef88225 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fac: skipped 000211 1787830330.715449 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33fac): Mirror 2: 0x3dc7e6fe (stale) Mirror 4: 0xbf333592 (stale) Mirror 5: 0xd45e829c (stale) Mirror 6: 0xd45e829c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fac): Mirror 2: 0x3dc7e6fe (stale) Mirror 4: 0xbf333592 (stale) Mirror 5: 0xd45e829c (stale) Mirror 6: 0xd45e829c (stale) CRC-32 checksum value for chunk [0, 0x33fac): Mirror 1: 0xbef88225 Mirror 7: 0xbef88225 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fac: skipped 000212 1787830331.023431 trunc from 0x033fac to 0x032263 (0x1d48 bytes) 000213 1787830331.055717 trunc from 0x00c850 to 0x017c0e (0xb3bf bytes) 000214 1787830331.058391 mapwrite 0x020b77 thru 0x026077 (0x5501 bytes) 000215 1787830331.083108 trunc from 0x032264 to 0x028993 (0x98d0 bytes) 000216 1787830331.095714 punch from 0x011aab to 0x018146 (0x669c bytes) 000217 1787830331.100980 MIRROR_RESYNC 6 mirrors 000219 1787830331.788978 mapread 0x022be5 thru 0x028993 (0x5daf bytes) 000220 1787830331.803326 trunc from 0x01fca2 to 0x0235af (0x390e bytes) 000221 1787830331.832713 read 0x00098a thru 0x00b01b (0xa692 bytes) 000222 1787830331.841010 fallocate 0x0342ac thru 0x03ffff (0xbd54 bytes) 000223 1787830331.847004 fallocate 0x0120a2 thru 0x01f960 (0xd8bf bytes) 000224 1787830331.850163 fallocate 0x03abe9 thru 0x03ffff (0x5417 bytes) 000225 1787830331.854986 punch from 0x012e22 to 0x01db1e (0xacfd bytes) 000226 1787830331.861613 mapread 0x01f10a thru 0x027089 (0x7f80 bytes) 000227 1787830331.874322 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9722fb88 (stale) Mirror 4: 0x9722fb88 (stale) Mirror 5: 0x9722fb88 (stale) Mirror 7: 0x9722fb88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9722fb88 (stale) Mirror 4: 0x9722fb88 (stale) Mirror 5: 0x9722fb88 (stale) Mirror 7: 0x9722fb88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000228 1787830332.117656 fallocate 0x006e5d thru 0x010122 (0x92c6 bytes) 000229 1787830332.121345 mapwrite 0x02a17b thru 0x0374cf (0xd355 bytes) 000230 1787830332.153028 fallocate 0x010ee7 thru 0x018972 (0x7a8c bytes) 000231 1787830332.157469 read 0x00c426 thru 0x01759a (0xb175 bytes) 000232 1787830332.165260 write 0x02b785 thru 0x0300d4 (0x4950 bytes) 000234 1787830332.169940 mapread 0x016769 thru 0x022e7c (0xc714 bytes) 000235 1787830332.181098 read 0x023a48 thru 0x02cfb5 (0x956e bytes) 000237 1787830332.182216 read 0x01ac66 thru 0x01ad77 (0x112 bytes) 000238 1787830332.182690 mapwrite 0x015f98 thru 0x023871 (0xd8da bytes) 000239 1787830332.199362 mapwrite 0x01cc26 thru 0x02c452 (0xf82d bytes) 000240 1787830332.225656 trunc from 0x0168e1 to 0x01c567 (0x5c87 bytes) 000241 1787830332.230467 fallocate 0x014792 thru 0x01d1de (0x8a4d bytes) 000242 1787830332.234891 fallocate 0x008e8b thru 0x00d06c (0x41e2 bytes) 000243 1787830332.238995 mapwrite 0x016d3c thru 0x020526 (0x97eb bytes) 000244 1787830332.259919 read 0x001364 thru 0x002d15 (0x19b2 bytes) 000245 1787830332.260434 trunc from 0x040000 to 0x016db6 (0x29249 bytes) 000246 1787830332.270655 read 0x007899 thru 0x00a1b2 (0x291a bytes) 000248 1787830332.271855 trunc from 0x016db7 to 0x00536c (0x11a4a bytes) 000249 1787830332.286870 punch from 0x0051db to 0x00536c (0x192 bytes) 000250 1787830332.295520 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x536d): Mirror 4: 0xb2c0c004 (stale) Mirror 5: 0xb2c0c004 (stale) Mirror 7: 0xb2c0c004 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x536d): Mirror 4: 0xb2c0c004 (stale) Mirror 5: 0xb2c0c004 (stale) Mirror 7: 0xb2c0c004 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x536d: skipped 000251 1787830332.527887 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x536d): Mirror 5: 0xb2c0c004 (stale) Mirror 7: 0xb2c0c004 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x536d): Mirror 5: 0xb2c0c004 (stale) Mirror 7: 0xb2c0c004 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x536d: skipped 000252 1787830332.764058 punch from 0x00004a to 0x00536c (0x5323 bytes) 000253 1787830332.775100 mapread 0x004430 thru 0x00536c (0xf3d bytes) 000254 1787830332.786881 write 0x00f9b1 thru 0x0180c6 (0x8716 bytes) 000255 1787830332.791965 read 0x016240 thru 0x0180c6 (0x1e87 bytes) 000256 1787830332.792572 fallocate 0x02413a thru 0x0323b7 (0xe27e bytes) 000257 1787830332.796525 fallocate 0x019356 thru 0x023e20 (0xaacb bytes) 000258 1787830332.801602 mapwrite 0x0085e1 thru 0x00b1f1 (0x2c11 bytes) 000259 1787830332.820470 trunc from 0x025f5c to 0x02edb3 (0x8e58 bytes) 000260 1787830332.823647 fallocate 0x011c1a thru 0x0191d6 (0x75bd bytes) 000261 1787830332.826869 fallocate 0x021892 thru 0x02c721 (0xae90 bytes) 000262 1787830332.831324 punch from 0x02f3c4 to 0x0323b7 (0x2ff4 bytes) 000263 1787830332.835834 read 0x00462c thru 0x00aa31 (0x6406 bytes) 000264 1787830332.836765 write 0x02b4ba thru 0x03074c (0x5293 bytes) 000265 1787830332.846778 read 0x011a5c thru 0x017a19 (0x5fbe bytes) 000266 1787830332.847558 mapwrite 0x022fc4 thru 0x023b3e (0xb7b bytes) 000267 1787830332.865524 read 0x00997a thru 0x0157f4 (0xbe7b bytes) 000268 1787830332.867419 read 0x026922 thru 0x027f77 (0x1656 bytes) 000269 1787830332.868771 write 0x03632d thru 0x03cf1d (0x6bf1 bytes) 000270 1787830332.870420 punch from 0x01e3d6 to 0x0255a5 (0x71d0 bytes) 000271 1787830332.874955 trunc from 0x03cf1e to 0x0321b4 (0xad69 bytes) 000272 1787830332.888430 write 0x0314c9 thru 0x038eb8 (0x79f0 bytes) 000273 1787830332.889993 trunc from 0x004f7b to 0x00b301 (0x6387 bytes) 000274 1787830332.893928 trunc from 0x038eb9 to 0x006f19 (0x31f9f bytes) 000275 1787830332.905697 mapwrite 0x0110f7 thru 0x013665 (0x256f bytes) 000276 1787830332.935625 read 0x005b38 thru 0x012798 (0xcc61 bytes) 000277 1787830332.936268 fallocate 0x02776d thru 0x032210 (0xaaa4 bytes) 000278 1787830332.939798 write 0x0335ce thru 0x03d229 (0x9c5c bytes) 000279 1787830332.944227 MIRROR_RESYNC 3 mirrors 000280 1787830333.458992 mapread 0x0127e8 thru 0x016705 (0x3f1e bytes) 000281 1787830333.472516 trunc from 0x007fcb to 0x00b5e8 (0x361e bytes) 000282 1787830333.504959 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d22a): Mirror 7: 0x4bcd86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d22a): Mirror 7: 0x4bcd86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d22a: skipped 000283 1787830333.686290 trunc from 0x01a6ee to 0x01ffc7 (0x58da bytes) 000284 1787830333.691577 write 0x033dba thru 0x03edef (0xb036 bytes) 000285 1787830333.701299 MIRROR_RESYNC 2 mirrors 000287 1787830334.054300 punch from 0x01b480 to 0x01dc6f (0x27f0 bytes) 000288 1787830334.075326 trunc from 0x03edf0 to 0x00280b (0x3c5e4 bytes) 000289 1787830334.091970 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000290 1787830334.183482 trunc from 0x001b36 to 0x00280b (0xcd6 bytes) 000292 1787830334.187494 mapread 0x002781 thru 0x00280b (0x8b bytes) 000293 1787830334.192529 trunc from 0x000648 to 0x00280b (0x21c4 bytes) 000294 1787830334.199402 punch from 0x000321 to 0x00280b (0x24eb bytes) 000295 1787830334.204233 read 0x00185d thru 0x00280b (0xfaf bytes) 000298 1787830334.212958 write 0x01474e thru 0x01f629 (0xaedc bytes) 000299 1787830334.222118 write 0x0284b6 thru 0x033188 (0xacd3 bytes) 000300 1787830334.223699 fallocate 0x03e345 thru 0x03ffff (0x1cbb bytes) 000301 1787830334.229338 mapwrite 0x037f25 thru 0x038db3 (0xe8f bytes) 000302 1787830334.274669 write 0x023178 thru 0x0260a4 (0x2f2d bytes) 000303 1787830334.284942 read 0x02afd7 thru 0x02f38d (0x43b7 bytes) 000305 1787830334.286358 trunc from 0x035ea9 to 0x038db3 (0x2f0b bytes) 000307 1787830334.291996 trunc from 0x038db4 to 0x024e2e (0x13f85 bytes) 000310 1787830334.309036 write 0x020a64 thru 0x026baa (0x6147 bytes) 000312 1787830334.319548 trunc from 0x026bab to 0x004c1a (0x21f90 bytes) 000313 1787830334.337959 read 0x004531 thru 0x004c1a (0x6ea bytes) 000314 1787830334.338445 trunc from 0x00354e to 0x004c1a (0x16cd bytes) 000315 1787830334.342442 mapread 0x002407 thru 0x004c1a (0x2814 bytes) 000316 1787830334.348636 mapread 0x002f06 thru 0x004c1a (0x1d15 bytes) 000319 1787830334.350469 read 0x0014c4 thru 0x004c1a (0x3757 bytes) 000320 1787830334.350993 fallocate 0x0251c6 thru 0x02a36b (0x51a6 bytes) 000321 1787830334.359259 read 0x029976 thru 0x02a36b (0x9f6 bytes) 000322 1787830334.368760 trunc from 0x01f0dc to 0x021a9d (0x29c2 bytes) 000323 1787830334.374927 write 0x02a060 thru 0x036b7f (0xcb20 bytes) 000324 1787830334.379499 punch from 0x0089aa to 0x00d6bc (0x4d13 bytes) 000325 1787830334.384788 fallocate 0x01b9a1 thru 0x02763a (0xbc9a bytes) 000327 1787830334.389551 write 0x03d1f9 thru 0x03ffff (0x2e07 bytes) 000328 1787830334.390510 fallocate 0x038a10 thru 0x03e08b (0x567c bytes) 000329 1787830334.397348 MIRROR_EXTEND to 2 mirrors 000330 1787830334.741213 MIRROR_RESYNC 2 mirrors 000331 1787830334.919438 mapread 0x006925 thru 0x00bea5 (0x5581 bytes) 000333 1787830334.939833 write 0x022eff thru 0x02f0f2 (0xc1f4 bytes) 000334 1787830334.983325 mapread 0x02496f thru 0x030376 (0xba08 bytes) 000335 1787830334.990899 trunc from 0x040000 to 0x01238e (0x2dc71 bytes) 000336 1787830335.018961 mapwrite 0x02c18d thru 0x0341b2 (0x8026 bytes) 000337 1787830335.053727 mapread 0x002acb thru 0x00dc70 (0xb1a6 bytes) 000338 1787830335.060944 write 0x023681 thru 0x027a99 (0x4419 bytes) 000339 1787830335.065131 write 0x023855 thru 0x02b4ac (0x7c58 bytes) 000340 1787830335.066422 fallocate 0x013b1e thru 0x014ed0 (0x13b3 bytes) 000341 1787830335.070336 trunc from 0x0241a8 to 0x029cc4 (0x5b1d bytes) 000343 1787830335.083091 read 0x00b885 thru 0x01a154 (0xe8d0 bytes) 000344 1787830335.094456 trunc from 0x0341b3 to 0x0000a6 (0x3410c bytes) 000345 1787830335.104374 mapwrite 0x000e2a thru 0x00c881 (0xba58 bytes) 000346 1787830335.140557 read 0x002a37 thru 0x00c881 (0x9e4b bytes) 000347 1787830335.141363 trunc from 0x00c882 to 0x039578 (0x2ccf7 bytes) 000348 1787830335.157196 punch from 0x0335cc to 0x039578 (0x5fad bytes) 000349 1787830335.163932 MIRROR_RESYNC 2 mirrors 000350 1787830335.393357 trunc from 0x039579 to 0x011fe9 (0x2758f bytes) 000351 1787830335.421505 fallocate 0x006fc3 thru 0x00d3b6 (0x63f4 bytes) 000352 1787830335.424546 read 0x00eeb8 thru 0x011fe9 (0x3132 bytes) 000353 1787830335.427925 trunc from 0x011fea to 0x01ba51 (0x9a68 bytes) 000354 1787830335.440312 trunc from 0x0024ee to 0x011b9a (0xf6ad bytes) 000355 1787830335.443987 mapread 0x004481 thru 0x00a95c (0x64dc bytes) 000356 1787830335.461848 mapread 0x007ebf thru 0x00f1c5 (0x7307 bytes) 000357 1787830335.463119 mapread 0x011cb9 thru 0x017aee (0x5e36 bytes) 000358 1787830335.468211 mapwrite 0x02e129 thru 0x03ccf2 (0xebca bytes) 000359 1787830335.496810 punch from 0x01f7bb to 0x026059 (0x689f bytes) 000360 1787830335.499973 write 0x03c1e5 thru 0x03ffff (0x3e1b bytes) 000361 1787830335.501042 mapwrite 0x01b631 thru 0x0289e6 (0xd3b6 bytes) 000362 1787830335.525065 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000364 1787830335.585102 read 0x014fed thru 0x016b73 (0x1b87 bytes) 000365 1787830335.587939 trunc from 0x040000 to 0x0393ef (0x6c10 bytes) 000366 1787830335.600109 trunc from 0x0393f0 to 0x02a4de (0xef11 bytes) 000368 1787830335.607593 mapread 0x01f195 thru 0x028957 (0x97c3 bytes) 000369 1787830335.619068 mapread 0x01608f thru 0x021bae (0xbb20 bytes) 000371 1787830335.624794 fallocate 0x03c544 thru 0x03f3fc (0x2eb9 bytes) 000372 1787830335.627277 fallocate 0x0214c9 thru 0x029eab (0x89e3 bytes) 000373 1787830335.630471 write 0x00181f thru 0x001f46 (0x728 bytes) 000375 1787830335.635122 read 0x00fe35 thru 0x0149a7 (0x4b73 bytes) 000376 1787830335.640908 write 0x00cdda thru 0x01356a (0x6791 bytes) 000377 1787830335.642320 fallocate 0x0155dc thru 0x01df95 (0x89ba bytes) 000378 1787830335.645602 punch from 0x003c4c to 0x012402 (0xe7b7 bytes) 000379 1787830335.655301 write 0x032cec thru 0x03ffff (0xd314 bytes) 000380 1787830335.656939 fallocate 0x02cbfa thru 0x0327e4 (0x5beb bytes) 000381 1787830335.660633 write 0x0354a9 thru 0x03ffff (0xab57 bytes) 000382 1787830335.661994 fallocate 0x03c36a thru 0x03ffff (0x3c96 bytes) 000384 1787830335.666015 fallocate 0x038e2b thru 0x03ffff (0x71d5 bytes) 000385 1787830335.669467 mapwrite 0x02e4d3 thru 0x03760a (0x9138 bytes) 000386 1787830335.691206 fallocate 0x02c083 thru 0x0389b2 (0xc930 bytes) 000387 1787830335.694108 trunc from 0x0227db to 0x028259 (0x5a7f bytes) 000389 1787830335.698970 punch from 0x00c9af to 0x00e49f (0x1af1 bytes) 000390 1787830335.701821 fallocate 0x002b9a thru 0x00bcfb (0x9162 bytes) 000391 1787830335.707224 MIRROR_EXTEND to 2 mirrors 000392 1787830336.004297 fallocate 0x035d39 thru 0x036b63 (0xe2b bytes) 000393 1787830336.027326 read 0x01ac0f thru 0x02a862 (0xfc54 bytes) 000394 1787830336.034342 write 0x01ee77 thru 0x02ddaf (0xef39 bytes) 000395 1787830336.042932 trunc from 0x040000 to 0x029318 (0x16ce7 bytes) 000396 1787830336.066277 punch from 0x00daba to 0x013f96 (0x64dd bytes) 000397 1787830336.069201 punch from 0x020656 to 0x022ed9 (0x2884 bytes) 000398 1787830336.078297 trunc from 0x029319 to 0x014bf4 (0x14724 bytes) 000399 1787830336.085653 punch from 0x011404 to 0x014bf4 (0x37f1 bytes) 000400 1787830336.088592 trunc from 0x014bf5 to 0x000402 (0x147f2 bytes) 000401 1787830336.101469 read 0x0001f2 thru 0x000402 (0x211 bytes) 000402 1787830336.104394 fallocate 0x00a380 thru 0x017795 (0xd416 bytes) 000403 1787830336.110845 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000404 1787830336.176421 fallocate 0x01b3a9 thru 0x021f49 (0x6ba1 bytes) 000405 1787830336.179672 trunc from 0x017796 to 0x0016c2 (0x160d3 bytes) 000407 1787830336.189302 write 0x020f97 thru 0x023b11 (0x2b7b bytes) 000408 1787830336.190293 write 0x026447 thru 0x031e37 (0xb9f1 bytes) 000409 1787830336.191541 trunc from 0x031e38 to 0x00fde0 (0x22057 bytes) 000410 1787830336.198932 read 0x00773a thru 0x00d795 (0x605c bytes) 000411 1787830336.203295 punch from 0x00ad75 to 0x00ba2d (0xcb9 bytes) 000412 1787830336.205828 trunc from 0x00fde1 to 0x01d837 (0xda57 bytes) 000413 1787830336.212679 trunc from 0x01d838 to 0x02cd9c (0xf565 bytes) 000414 1787830336.220435 MIRROR_EXTEND to 2 mirrors 000415 1787830336.500155 punch from 0x0047be to 0x00a138 (0x597b bytes) 000416 1787830336.525288 mapread 0x00111f thru 0x00ec30 (0xdb12 bytes) 000417 1787830336.539891 trunc from 0x0013bb to 0x007c67 (0x68ad bytes) 000418 1787830336.546392 punch from 0x017264 to 0x024373 (0xd110 bytes) 000419 1787830336.551043 mapread 0x010f51 thru 0x01bb60 (0xac10 bytes) 000421 1787830336.566089 fallocate 0x03431d thru 0x03fcbe (0xb9a2 bytes) 000422 1787830336.569744 write 0x02b27e thru 0x02f868 (0x45eb bytes) 000423 1787830336.580422 read 0x02f448 thru 0x036419 (0x6fd2 bytes) 000424 1787830336.590825 trunc from 0x00f4e4 to 0x0135ad (0x40ca bytes) 000425 1787830336.594164 mapwrite 0x03a22f thru 0x03ffff (0x5dd1 bytes) 000426 1787830336.636785 mapwrite 0x02c39f thru 0x0361c7 (0x9e29 bytes) 000427 1787830336.651542 mapwrite 0x020893 thru 0x025bb4 (0x5322 bytes) 000429 1787830336.675951 mapread 0x023b8f thru 0x0300f7 (0xc569 bytes) 000430 1787830336.677416 fallocate 0x016360 thru 0x01e974 (0x8615 bytes) 000432 1787830336.682366 punch from 0x015fa2 to 0x01aa38 (0x4a97 bytes) 000433 1787830336.686914 fallocate 0x033f36 thru 0x03a4c3 (0x658e bytes) 000434 1787830336.693689 mapwrite 0x00f4ed thru 0x015616 (0x612a bytes) 000435 1787830336.711232 fallocate 0x030d38 thru 0x03d01e (0xc2e7 bytes) 000436 1787830336.716736 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000437 1787830336.781375 read 0x01f584 thru 0x025b65 (0x65e2 bytes) 000438 1787830336.785262 fallocate 0x032437 thru 0x03a505 (0x80cf bytes) 000439 1787830336.789422 read 0x02b8e3 thru 0x039996 (0xe0b4 bytes) 000440 1787830336.795568 punch from 0x03adfc to 0x03ffff (0x5204 bytes) 000441 1787830336.799459 mapwrite 0x00c4a3 thru 0x016c45 (0xa7a3 bytes) 000442 1787830336.813930 trunc from 0x040000 to 0x01c5af (0x23a50 bytes) 000443 1787830336.825630 trunc from 0x016c16 to 0x01c5af (0x599a bytes) 000444 1787830336.829558 fallocate 0x03b7a6 thru 0x03ffff (0x485a bytes) 000445 1787830336.834242 trunc from 0x023e75 to 0x02d291 (0x941d bytes) 000446 1787830336.838327 trunc from 0x040000 to 0x034422 (0xbbdd bytes) 000448 1787830336.847724 fallocate 0x0177be thru 0x01eaf0 (0x7333 bytes) 000449 1787830336.850888 write 0x034309 thru 0x03cff4 (0x8cec bytes) 000451 1787830336.855726 mapwrite 0x007b90 thru 0x010f2d (0x939e bytes) 000452 1787830336.866884 punch from 0x000a29 to 0x0098db (0x8eb3 bytes) 000453 1787830336.871364 punch from 0x03873b to 0x03cff4 (0x48ba bytes) 000455 1787830336.876163 fallocate 0x01f25e thru 0x028b85 (0x9928 bytes) 000456 1787830336.879136 fallocate 0x02cbd2 thru 0x039875 (0xcca4 bytes) 000457 1787830336.881814 punch from 0x027432 to 0x0304ea (0x90b9 bytes) 000458 1787830336.887063 mapwrite 0x01d701 thru 0x01e970 (0x1270 bytes) 000459 1787830336.909411 mapread 0x029e55 thru 0x02c5fc (0x27a8 bytes) 000460 1787830336.923443 mapwrite 0x023e35 thru 0x02bbeb (0x7db7 bytes) 000461 1787830336.948365 read 0x01da29 thru 0x020225 (0x27fd bytes) 000462 1787830336.954165 fallocate 0x0065ef thru 0x010df3 (0xa805 bytes) 000463 1787830336.958095 trunc from 0x02f3a3 to 0x037862 (0x84c0 bytes) 000464 1787830336.962282 fallocate 0x0024a8 thru 0x0076e2 (0x523b bytes) 000465 1787830336.965608 punch from 0x02e3d8 to 0x03acb8 (0xc8e1 bytes) 000466 1787830336.974843 MIRROR_EXTEND to 2 mirrors 000467 1787830337.330758 read 0x02d1b3 thru 0x031537 (0x4385 bytes) 000468 1787830337.336746 mapread 0x020784 thru 0x0275b5 (0x6e32 bytes) 000469 1787830337.349387 mapwrite 0x037a87 thru 0x03c6b5 (0x4c2f bytes) 000470 1787830337.414992 mapread 0x011b0e thru 0x015180 (0x3673 bytes) 000471 1787830337.419867 fallocate 0x017ac4 thru 0x0186dd (0xc1a bytes) 000472 1787830337.422868 punch from 0x02ff7d to 0x031a4f (0x1ad3 bytes) 000473 1787830337.425524 write 0x01e401 thru 0x02944b (0xb04b bytes) 000474 1787830337.436554 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000475 1787830337.529100 fallocate 0x033512 thru 0x03d4b1 (0x9fa0 bytes) 000476 1787830337.542966 punch from 0x0310b3 to 0x03268b (0x15d9 bytes) 000477 1787830337.548607 write 0x03e18b thru 0x03ffff (0x1e75 bytes) 000478 1787830337.553267 fallocate 0x011f81 thru 0x01604b (0x40cb bytes) 000479 1787830337.556505 fallocate 0x007c7c thru 0x00cc28 (0x4fad bytes) 000480 1787830337.560755 read 0x017dad thru 0x01dd6f (0x5fc3 bytes) 000481 1787830337.566911 mapwrite 0x004adf thru 0x011352 (0xc874 bytes) 000482 1787830337.595321 mapread 0x02e3cb thru 0x03bcfb (0xd931 bytes) 000484 1787830337.611684 trunc from 0x040000 to 0x0228d5 (0x1d72a bytes) 000486 1787830337.622579 read 0x000a1f thru 0x00be7b (0xb45d bytes) 000488 1787830337.628906 punch from 0x00742d to 0x0107ce (0x93a2 bytes) 000489 1787830337.632202 mapread 0x013b0e thru 0x01d150 (0x9643 bytes) 000491 1787830337.635641 mapread 0x018109 thru 0x0228d5 (0xa7cd bytes) 000492 1787830337.636917 read 0x00bffc thru 0x01698c (0xa991 bytes) 000493 1787830337.643234 trunc from 0x0228d6 to 0x01c294 (0x6641 bytes) 000494 1787830337.654378 mapwrite 0x009d8e thru 0x015dda (0xc04d bytes) 000496 1787830337.669966 read 0x006067 thru 0x012934 (0xc8ce bytes) 000497 1787830337.670579 read 0x0164c9 thru 0x01c294 (0x5dcc bytes) 000499 1787830337.672057 trunc from 0x0075af to 0x013d40 (0xc792 bytes) 000500 1787830337.678329 fallocate 0x03e3e7 thru 0x03ffff (0x1c19 bytes) 000501 1787830337.682327 trunc from 0x018ca1 to 0x01c294 (0x35f4 bytes) 000502 1787830337.689945 fallocate 0x009333 thru 0x00bf1e (0x2bec bytes) 000504 1787830337.693334 read 0x00912b thru 0x013ec2 (0xad98 bytes) 000505 1787830337.699118 write 0x006742 thru 0x008a87 (0x2346 bytes) 000506 1787830337.700669 read 0x017497 thru 0x017adc (0x646 bytes) 000507 1787830337.701341 mapread 0x000b93 thru 0x009e70 (0x92de bytes) 000508 1787830337.702222 mapread 0x006b2c thru 0x0137ad (0xcc82 bytes) 000509 1787830337.702967 fallocate 0x03d5fa thru 0x03ffff (0x2a06 bytes) 000511 1787830337.706879 read 0x00047f thru 0x00fd76 (0xf8f8 bytes) 000512 1787830337.707421 fallocate 0x009821 thru 0x00b4c1 (0x1ca1 bytes) 000513 1787830337.710773 mapread 0x003c60 thru 0x008e02 (0x51a3 bytes) 000514 1787830337.711653 punch from 0x000428 to 0x0048e9 (0x44c2 bytes) 000515 1787830337.715615 write 0x01cf11 thru 0x026fa1 (0xa091 bytes) 000516 1787830337.716967 mapread 0x012915 thru 0x020ae9 (0xe1d5 bytes) 000517 1787830337.717853 read 0x0026c4 thru 0x006820 (0x415d bytes) 000518 1787830337.723081 mapread 0x01e2d6 thru 0x023e3f (0x5b6a bytes) 000519 1787830337.724088 write 0x0141c4 thru 0x016121 (0x1f5e bytes) 000520 1787830337.725020 trunc from 0x019809 to 0x020c67 (0x745f bytes) 000521 1787830337.735339 trunc from 0x026fa2 to 0x0163ab (0x10bf6 bytes) 000522 1787830337.746775 fallocate 0x00d938 thru 0x016c52 (0x931b bytes) 000523 1787830337.750949 mapwrite 0x0316d6 thru 0x03d0f4 (0xba1f bytes) 000524 1787830337.792418 fallocate 0x029d5f thru 0x03607d (0xc31f bytes) 000525 1787830337.795399 mapwrite 0x0268d9 thru 0x02cb29 (0x6251 bytes) 000526 1787830337.813819 punch from 0x017fee to 0x025103 (0xd116 bytes) 000527 1787830337.816533 write 0x026f8f thru 0x028cb9 (0x1d2b bytes) 000528 1787830337.817607 trunc from 0x03d0f5 to 0x02f58e (0xdb66 bytes) 000529 1787830337.826081 punch from 0x01567a to 0x01ae39 (0x57c0 bytes) 000530 1787830337.831051 mapwrite 0x01f0a2 thru 0x02c529 (0xd488 bytes) 000531 1787830337.859536 trunc from 0x011e64 to 0x016ab3 (0x4c50 bytes) 000532 1787830337.862041 punch from 0x028367 to 0x02f58e (0x7228 bytes) 000533 1787830337.864459 mapread 0x009b23 thru 0x00ee1c (0x52fa bytes) 000536 1787830337.866544 trunc from 0x02136c to 0x02ee85 (0xdb1a bytes) 000537 1787830337.869371 mapwrite 0x03f3dc thru 0x03ffff (0xc24 bytes) 000538 1787830337.888376 mapwrite 0x00c0ae thru 0x012c6d (0x6bc0 bytes) 000539 1787830337.904580 mapread 0x00f965 thru 0x01284e (0x2eea bytes) 000540 1787830337.905465 trunc from 0x040000 to 0x0397d6 (0x6829 bytes) 000541 1787830337.912719 fallocate 0x034020 thru 0x039a0d (0x59ee bytes) 000542 1787830337.915535 trunc from 0x039a0e to 0x00b1d7 (0x2e836 bytes) 000543 1787830337.924541 mapwrite 0x00e236 thru 0x0107bd (0x2588 bytes) 000544 1787830337.947334 write 0x02c13e thru 0x02c648 (0x50b bytes) 000545 1787830337.948327 fallocate 0x00b103 thru 0x017236 (0xc134 bytes) 000546 1787830337.952914 mapread 0x00e992 thru 0x0103ba (0x1a29 bytes) 000547 1787830337.953725 mapread 0x00fb77 thru 0x018cd6 (0x9160 bytes) 000548 1787830337.974280 trunc from 0x0078cc to 0x00c25b (0x4990 bytes) 000549 1787830337.977332 mapread 0x02b6ab thru 0x02c648 (0xf9e bytes) 000550 1787830337.981386 trunc from 0x021758 to 0x027925 (0x61ce bytes) 000551 1787830337.983994 read 0x00c6d1 thru 0x00f88e (0x31be bytes) 000553 1787830337.989039 trunc from 0x01bbc6 to 0x02add5 (0xf210 bytes) 000555 1787830337.992122 write 0x021b4f thru 0x02b0d8 (0x958a bytes) 000557 1787830337.996457 read 0x0172e8 thru 0x024e37 (0xdb50 bytes) 000559 1787830338.002276 mapread 0x012a1b thru 0x022350 (0xf936 bytes) 000560 1787830338.003086 trunc from 0x00e816 to 0x017bcd (0x93b8 bytes) 000561 1787830338.005879 fallocate 0x01d415 thru 0x021d7f (0x496b bytes) 000564 1787830338.009708 trunc from 0x02c649 to 0x0054c5 (0x27183 bytes) 000565 1787830338.016249 write 0x001cf8 thru 0x005859 (0x3b62 bytes) 000566 1787830338.017207 fallocate 0x032b08 thru 0x03cee8 (0xa3e1 bytes) 000567 1787830338.020537 fallocate 0x01a2f6 thru 0x021d9d (0x7aa8 bytes) 000569 1787830338.024027 write 0x024ec1 thru 0x02cd0c (0x7e4c bytes) 000570 1787830338.035141 write 0x004b7e thru 0x00b5d3 (0x6a56 bytes) 000571 1787830338.039621 write 0x019dd7 thru 0x022c5b (0x8e85 bytes) 000572 1787830338.048988 mapread 0x025eae thru 0x02ab02 (0x4c55 bytes) 000573 1787830338.049780 trunc from 0x03cee9 to 0x01c524 (0x209c4 bytes) 000574 1787830338.063208 punch from 0x00eb6d to 0x01c0e1 (0xd575 bytes) 000575 1787830338.075051 write 0x00096a thru 0x00fbfc (0xf293 bytes) 000576 1787830338.082908 MIRROR_EXTEND to 2 mirrors 000578 1787830338.346710 fallocate 0x0370b4 thru 0x037c1c (0xb69 bytes) 000579 1787830338.370057 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000580 1787830338.436082 punch from 0x00fd46 to 0x0146a8 (0x4963 bytes) 000581 1787830338.438788 mapwrite 0x0063b4 thru 0x00a12e (0x3d7b bytes) 000582 1787830338.454982 read 0x00a20c thru 0x01374d (0x9542 bytes) 000583 1787830338.458199 mapwrite 0x00d35a thru 0x011d85 (0x4a2c bytes) 000584 1787830338.467615 write 0x020a18 thru 0x022a4f (0x2038 bytes) 000585 1787830338.473832 MIRROR_EXTEND to 2 mirrors 000586 1787830338.743914 write 0x03e718 thru 0x03ffff (0x18e8 bytes) 000587 1787830338.776516 punch from 0x004910 to 0x009386 (0x4a77 bytes) 000588 1787830338.782779 punch from 0x038aeb to 0x03c22d (0x3743 bytes) 000589 1787830338.788027 write 0x0213d3 thru 0x027f7a (0x6ba8 bytes) 000590 1787830338.799117 punch from 0x028d25 to 0x02d6cb (0x49a7 bytes) 000591 1787830338.805508 write 0x011bed thru 0x016e26 (0x523a bytes) 000592 1787830338.814854 trunc from 0x040000 to 0x02d0d0 (0x12f2f bytes) 000593 1787830338.840597 write 0x00a0e4 thru 0x0138a7 (0x97c4 bytes) 000594 1787830338.845546 read 0x008802 thru 0x01773d (0xef3c bytes) 000595 1787830338.852933 mapread 0x00a104 thru 0x0132c6 (0x91c3 bytes) 000596 1787830338.854133 read 0x00d3db thru 0x00e37b (0xfa1 bytes) 000597 1787830338.854716 punch from 0x02029c to 0x0241bb (0x3f20 bytes) 000598 1787830338.865500 mapread 0x01bdcc thru 0x029f0f (0xe144 bytes) 000599 1787830338.872571 mapread 0x0076ab thru 0x00a241 (0x2b97 bytes) 000601 1787830338.873814 trunc from 0x02d0d1 to 0x02e5b8 (0x14e8 bytes) 000604 1787830338.882375 trunc from 0x00e159 to 0x01648c (0x8334 bytes) 000605 1787830338.885523 read 0x015ba5 thru 0x020942 (0xad9e bytes) 000606 1787830338.892100 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e5b9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e5b9): Mirror 2: 0x9a41b346 (stale) CRC-32 checksum value for chunk [0, 0x2e5b9): Mirror 1: 0xb8596d7a Mirror 3: 0xb8596d7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e5b9: skipped 000607 1787830339.338274 fallocate 0x01cf91 thru 0x02c665 (0xf6d5 bytes) 000608 1787830339.365917 MIRROR_RESYNC 3 mirrors 000609 1787830339.914999 trunc from 0x018432 to 0x01f180 (0x6d4f bytes) 000610 1787830339.943149 mapwrite 0x02a045 thru 0x032ca9 (0x8c65 bytes) 000611 1787830339.983798 mapread 0x011de2 thru 0x020320 (0xe53f bytes) 000612 1787830339.994571 punch from 0x00bf82 to 0x00f100 (0x317f bytes) 000614 1787830339.998898 write 0x0064e0 thru 0x015d92 (0xf8b3 bytes) 000615 1787830340.004314 read 0x01ad3e thru 0x02767d (0xc940 bytes) 000616 1787830340.013603 MIRROR_RESYNC 3 mirrors 000617 1787830340.577665 read 0x02627a thru 0x028482 (0x2209 bytes) 000618 1787830340.583780 trunc from 0x01d636 to 0x026e9c (0x9867 bytes) 000619 1787830340.607531 fallocate 0x02dac7 thru 0x0339b3 (0x5eed bytes) 000620 1787830340.611159 trunc from 0x02a9e7 to 0x0339b3 (0x8fcd bytes) 000621 1787830340.614381 trunc from 0x0339b4 to 0x03f162 (0xb7af bytes) 000622 1787830340.635760 trunc from 0x03f163 to 0x023a94 (0x1b6ce bytes) 000623 1787830340.647670 read 0x00bd07 thru 0x01b8eb (0xfbe5 bytes) 000624 1787830340.657109 read 0x00992b thru 0x012898 (0x8f6e bytes) 000625 1787830340.659073 write 0x032504 thru 0x03ffff (0xdafc bytes) 000626 1787830340.660986 mapread 0x00d736 thru 0x0132fe (0x5bc9 bytes) 000627 1787830340.662283 write 0x004d61 thru 0x011cae (0xcf4e bytes) 000628 1787830340.668817 MIRROR_RESYNC 3 mirrors 000629 1787830341.082487 read 0x00573c thru 0x012198 (0xca5d bytes) 000630 1787830341.092491 mapread 0x01c583 thru 0x027a30 (0xb4ae bytes) 000631 1787830341.109277 mapwrite 0x00efe4 thru 0x01cf61 (0xdf7e bytes) 000632 1787830341.188644 trunc from 0x02235c to 0x025b2c (0x37d1 bytes) 000633 1787830341.193703 fallocate 0x00cf63 thru 0x012013 (0x50b1 bytes) 000635 1787830341.200982 mapread 0x006102 thru 0x00ecaf (0x8bae bytes) 000636 1787830341.208500 fallocate 0x034628 thru 0x03ffff (0xb9d8 bytes) 000637 1787830341.211876 trunc from 0x040000 to 0x01a499 (0x25b66 bytes) 000638 1787830341.236794 trunc from 0x01a49a to 0x0340c2 (0x19c29 bytes) 000639 1787830341.252856 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x340c3): Mirror 3: 0xbc7a4e7a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340c3): Mirror 3: 0xbc7a4e7a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340c3: skipped 000640 1787830341.434490 fallocate 0x00529d thru 0x01081c (0xb580 bytes) 000641 1787830341.439318 punch from 0x01f779 to 0x022334 (0x2bbc bytes) 000642 1787830341.443004 read 0x005f47 thru 0x00ef64 (0x901e bytes) 000643 1787830341.454946 punch from 0x00997f to 0x00c244 (0x28c6 bytes) 000645 1787830341.460024 write 0x03742a thru 0x038e0d (0x19e4 bytes) 000646 1787830341.464129 trunc from 0x00d9a6 to 0x010311 (0x296c bytes) 000647 1787830341.467948 fallocate 0x03fa33 thru 0x03ffff (0x5cd bytes) 000648 1787830341.472708 mapread 0x006f81 thru 0x008b1e (0x1b9e bytes) 000649 1787830341.473560 trunc from 0x038e0e to 0x003c5b (0x351b2 bytes) 000650 1787830341.494548 mapwrite 0x020516 thru 0x02b10c (0xabf7 bytes) 000651 1787830341.564917 punch from 0x02693a to 0x02b10c (0x47d3 bytes) 000652 1787830341.569520 mapwrite 0x035df2 thru 0x038009 (0x2218 bytes) 000654 1787830341.601409 trunc from 0x03800a to 0x01699b (0x2166e bytes) 000655 1787830341.610851 punch from 0x0164b9 to 0x01699b (0x4e3 bytes) 000656 1787830341.614513 write 0x003b6b thru 0x01174f (0xdbe5 bytes) 000657 1787830341.621541 trunc from 0x01699c to 0x02c737 (0x15d9c bytes) 000658 1787830341.631767 fallocate 0x01473f thru 0x01f8b1 (0xb173 bytes) 000660 1787830341.635201 read 0x012f2f thru 0x01d579 (0xa64b bytes) 000662 1787830341.641579 mapread 0x007a01 thru 0x00cad2 (0x50d2 bytes) 000663 1787830341.645689 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c738): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c738): Mirror 3: 0xb2abe7c6 (stale) CRC-32 checksum value for chunk [0, 0x2c738): Mirror 1: 0x16d5c204 Mirror 4: 0x16d5c204 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c738: skipped 000664 1787830342.114599 read 0x020a25 thru 0x021424 (0xa00 bytes) 000665 1787830342.119420 mapwrite 0x02069a thru 0x02eece (0xe835 bytes) 000666 1787830342.201495 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2eecf): Mirror 3: 0xae9e1cc8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eecf): Mirror 3: 0xae9e1cc8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eecf: skipped 000667 1787830342.388912 fallocate 0x02003d thru 0x0216d9 (0x169d bytes) 000668 1787830342.391420 trunc from 0x0055cd to 0x0151e3 (0xfc17 bytes) 000669 1787830342.399392 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000670 1787830342.464415 trunc from 0x02eecf to 0x00f62f (0x1f89f bytes) 000671 1787830342.475887 write 0x024e2b thru 0x02cf96 (0x816c bytes) 000673 1787830342.477685 trunc from 0x018b4d to 0x026775 (0xdc29 bytes) 000675 1787830342.489047 trunc from 0x0088d5 to 0x01664a (0xdd76 bytes) 000676 1787830342.494143 mapwrite 0x0295e5 thru 0x02e1d0 (0x4bec bytes) 000677 1787830342.538644 fallocate 0x01d3ff thru 0x023022 (0x5c24 bytes) 000678 1787830342.542491 mapwrite 0x00967a thru 0x01421b (0xaba2 bytes) 000679 1787830342.570588 mapread 0x01497d thru 0x01e3d2 (0x9a56 bytes) 000680 1787830342.579675 mapread 0x014b27 thru 0x01b6d5 (0x6baf bytes) 000681 1787830342.580546 mapread 0x01c013 thru 0x02a4d8 (0xe4c6 bytes) 000682 1787830342.590318 trunc from 0x02e1d1 to 0x0190c2 (0x1510e bytes) 000683 1787830342.604220 fallocate 0x011a25 thru 0x01c4b7 (0xaa93 bytes) 000684 1787830342.608801 trunc from 0x0190c3 to 0x013f6e (0x5154 bytes) 000685 1787830342.621867 punch from 0x00bd6a to 0x00c579 (0x810 bytes) 000686 1787830342.628116 fallocate 0x037cad thru 0x03cdff (0x5153 bytes) 000687 1787830342.632788 trunc from 0x03ce00 to 0x0295ce (0x13831 bytes) 000688 1787830342.653435 MIRROR_EXTEND to 2 mirrors 000689 1787830343.019895 trunc from 0x0295cf to 0x0188d4 (0x10cfa bytes) 000690 1787830343.043924 mapwrite 0x00071f thru 0x0054e0 (0x4dc2 bytes) 000691 1787830343.068240 mapread 0x00375f thru 0x00f724 (0xbfc6 bytes) 000692 1787830343.069202 punch from 0x00f314 to 0x010305 (0xff2 bytes) 000693 1787830343.074820 read 0x0184e9 thru 0x0188d4 (0x3ec bytes) 000694 1787830343.077931 read 0x008753 thru 0x011065 (0x8913 bytes) 000695 1787830343.087821 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000696 1787830343.154061 trunc from 0x00e0fc to 0x0188d4 (0xa7d9 bytes) 000697 1787830343.157714 trunc from 0x011263 to 0x0146e7 (0x3485 bytes) 000698 1787830343.160009 fallocate 0x02cdbd thru 0x0390ba (0xc2fe bytes) 000699 1787830343.163128 write 0x013ea6 thru 0x01c178 (0x82d3 bytes) 000700 1787830343.170348 trunc from 0x0043f7 to 0x00926d (0x4e77 bytes) 000702 1787830343.173793 trunc from 0x01c179 to 0x0378d2 (0x1b75a bytes) 000703 1787830343.191820 read 0x01924f thru 0x021146 (0x7ef8 bytes) 000705 1787830343.199997 read 0x035917 thru 0x0378d2 (0x1fbc bytes) 000708 1787830343.208902 fallocate 0x003063 thru 0x00b628 (0x85c6 bytes) 000709 1787830343.212693 punch from 0x025bfd to 0x0296ca (0x3ace bytes) 000710 1787830343.224272 punch from 0x010083 to 0x01023c (0x1ba bytes) 000711 1787830343.227386 mapwrite 0x028bd7 thru 0x0352e7 (0xc711 bytes) 000712 1787830343.245931 trunc from 0x0378d3 to 0x03ceac (0x55da bytes) 000713 1787830343.254644 punch from 0x01ca2f to 0x025f42 (0x9514 bytes) 000716 1787830343.258789 trunc from 0x03cead to 0x01cf72 (0x1ff3a bytes) 000718 1787830343.270418 MIRROR_EXTEND to 2 mirrors 000719 1787830343.583026 mapwrite 0x0374fb thru 0x03ffff (0x8b05 bytes) 000720 1787830343.682761 trunc from 0x02fa13 to 0x030157 (0x745 bytes) 000721 1787830343.689827 read 0x00a2c3 thru 0x015746 (0xb484 bytes) 000722 1787830343.701424 mapwrite 0x019f4e thru 0x01cac9 (0x2b7c bytes) 000723 1787830343.724269 punch from 0x00d942 to 0x019924 (0xbfe3 bytes) 000724 1787830343.728584 trunc from 0x02fe6e to 0x037fd2 (0x8165 bytes) 000725 1787830343.732073 punch from 0x00afed to 0x012602 (0x7616 bytes) 000726 1787830343.736402 trunc from 0x040000 to 0x034279 (0xbd86 bytes) 000727 1787830343.754117 trunc from 0x03427a to 0x032342 (0x1f37 bytes) 000728 1787830343.762863 mapread 0x013df6 thru 0x01c5ea (0x87f5 bytes) 000729 1787830343.768943 mapread 0x004d4b thru 0x00aac3 (0x5d79 bytes) 000730 1787830343.779588 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000731 1787830343.857527 write 0x03f1d7 thru 0x03ffff (0xe29 bytes) 000732 1787830343.861009 write 0x025ff0 thru 0x035593 (0xf5a4 bytes) 000734 1787830343.867725 trunc from 0x0126b4 to 0x01af02 (0x884f bytes) 000735 1787830343.870714 fallocate 0x00148e thru 0x005efd (0x4a70 bytes) 000736 1787830343.874270 trunc from 0x040000 to 0x039b9f (0x6460 bytes) 000737 1787830343.882573 trunc from 0x01b08a to 0x01df0b (0x2e82 bytes) 000738 1787830343.886471 mapwrite 0x038ba3 thru 0x039d06 (0x1164 bytes) 000739 1787830343.925129 mapread 0x01d1a6 thru 0x01d1e4 (0x3f bytes) 000740 1787830343.932313 read 0x002573 thru 0x00cd60 (0xa7ee bytes) 000741 1787830343.939223 mapread 0x030fb7 thru 0x035eb0 (0x4efa bytes) 000742 1787830343.940114 trunc from 0x005687 to 0x00d7f0 (0x816a bytes) 000743 1787830343.947920 trunc from 0x01aa58 to 0x027bc7 (0xd170 bytes) 000744 1787830343.951414 read 0x01d356 thru 0x0246c4 (0x736f bytes) 000745 1787830343.957274 trunc from 0x0329d8 to 0x039d06 (0x732f bytes) 000746 1787830343.960335 read 0x001b90 thru 0x00ee6a (0xd2db bytes) 000748 1787830343.961397 read 0x01e438 thru 0x02938d (0xaf56 bytes) 000749 1787830343.961934 punch from 0x016c53 to 0x017b7d (0xf2b bytes) 000750 1787830343.965463 read 0x01a26e thru 0x0250c2 (0xae55 bytes) 000751 1787830343.966062 read 0x013444 thru 0x021a49 (0xe606 bytes) 000752 1787830343.972880 trunc from 0x039d07 to 0x022f33 (0x16dd3 bytes) 000753 1787830343.981495 trunc from 0x022f34 to 0x02ab34 (0x7c01 bytes) 000754 1787830343.991225 mapread 0x01dafb thru 0x0222a3 (0x47a9 bytes) 000755 1787830343.992128 trunc from 0x02ab35 to 0x007ed1 (0x22c63 bytes) 000756 1787830344.014463 mapread 0x007c10 thru 0x007ed1 (0x2c2 bytes) 000757 1787830344.015295 fallocate 0x00ee46 thru 0x01c8c7 (0xda82 bytes) 000759 1787830344.022899 MIRROR_EXTEND to 2 mirrors 000761 1787830344.369513 mapread 0x003981 thru 0x0064b2 (0x2b32 bytes) 000762 1787830344.381151 trunc from 0x003626 to 0x007ed1 (0x48ac bytes) 000763 1787830344.400549 fallocate 0x03700d thru 0x03d754 (0x6748 bytes) 000764 1787830344.403533 trunc from 0x03d755 to 0x0227c1 (0x1af93 bytes) 000765 1787830344.413734 read 0x00024e thru 0x0073a7 (0x715a bytes) 000766 1787830344.418368 punch from 0x0074c8 to 0x014b7e (0xd6b7 bytes) 000767 1787830344.421766 mapread 0x01a1c4 thru 0x01aa38 (0x875 bytes) 000769 1787830344.429398 mapread 0x00e33c thru 0x013c29 (0x58ee bytes) 000770 1787830344.435620 fallocate 0x0012d3 thru 0x0072e7 (0x6015 bytes) 000771 1787830344.438318 mapread 0x016b6f thru 0x0227c1 (0xbc53 bytes) 000772 1787830344.442246 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x227c2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x227c2): Mirror 2: 0x82d806ea (stale) CRC-32 checksum value for chunk [0, 0x227c2): Mirror 1: 0xd224bdbe Mirror 3: 0xd224bdbe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x227c2: skipped 000773 1787830344.770890 fallocate 0x016bdd thru 0x022dea (0xc20e bytes) 000774 1787830344.788335 mapread 0x00cc1c thru 0x013f83 (0x7368 bytes) 000775 1787830344.799744 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x22deb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22deb): Mirror 2: 0x45c37332 (stale) Mirror 3: 0xbf59ce9f (stale) CRC-32 checksum value for chunk [0, 0x22deb): Mirror 1: 0xbf59ce9f Mirror 4: 0xbf59ce9f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22deb: skipped 000776 1787830345.173473 mapread 0x00f2a4 thru 0x010a6b (0x17c8 bytes) 000777 1787830345.181736 fallocate 0x013c5d thru 0x0169bd (0x2d61 bytes) 000778 1787830345.205494 write 0x0244ff thru 0x03233f (0xde41 bytes) 000779 1787830345.211754 write 0x035b2f thru 0x03ffff (0xa4d1 bytes) 000780 1787830345.213383 fallocate 0x027eb2 thru 0x02cd3a (0x4e89 bytes) 000781 1787830345.224530 trunc from 0x040000 to 0x03ab1a (0x54e5 bytes) 000782 1787830345.253127 trunc from 0x023540 to 0x02a4d4 (0x6f95 bytes) 000783 1787830345.263298 read 0x033309 thru 0x0356c5 (0x23bd bytes) 000784 1787830345.267742 trunc from 0x030d8b to 0x03ab1a (0x9d90 bytes) 000785 1787830345.274696 trunc from 0x030ff1 to 0x03ab1a (0x9b2a bytes) 000786 1787830345.278724 write 0x03e872 thru 0x03e924 (0xb3 bytes) 000787 1787830345.279520 read 0x004216 thru 0x005078 (0xe63 bytes) 000788 1787830345.288078 write 0x030e9d thru 0x03569a (0x47fe bytes) 000789 1787830345.289220 mapread 0x00ce65 thru 0x011865 (0x4a01 bytes) 000790 1787830345.290352 fallocate 0x027930 thru 0x034e01 (0xd4d2 bytes) 000791 1787830345.295877 fallocate 0x02aa47 thru 0x0318e0 (0x6e9a bytes) 000792 1787830345.300022 punch from 0x00d59e to 0x018f4a (0xb9ad bytes) 000793 1787830345.306637 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e925): Mirror 3: 0x3233e1fc (stale) Mirror 4: 0x3233e1fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e925): Mirror 3: 0x3233e1fc (stale) Mirror 4: 0x3233e1fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e925: skipped 000794 1787830345.504560 mapwrite 0x0249c9 thru 0x031b64 (0xd19c bytes) 000795 1787830345.561564 mapread 0x0141f2 thru 0x022527 (0xe336 bytes) 000796 1787830345.580756 mapread 0x01615e thru 0x020941 (0xa7e4 bytes) 000797 1787830345.581954 punch from 0x012ec6 to 0x014f39 (0x2074 bytes) 000798 1787830345.588411 fallocate 0x004775 thru 0x00d9de (0x926a bytes) 000799 1787830345.599462 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e925): Mirror 4: 0x3233e1fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e925): Mirror 4: 0x3233e1fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e925: skipped 000800 1787830345.721837 mapwrite 0x00264e thru 0x00b3af (0x8d62 bytes) 000801 1787830345.745800 trunc from 0x019610 to 0x01a34b (0xd3c bytes) 000802 1787830345.748952 mapwrite 0x024adf thru 0x02ddc7 (0x92e9 bytes) 000803 1787830345.768531 punch from 0x03320b to 0x03ccf4 (0x9aea bytes) 000804 1787830345.771336 trunc from 0x03e925 to 0x02a5bc (0x14368 bytes) 000805 1787830345.781431 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000806 1787830345.840894 MIRROR_EXTEND to 2 mirrors 000807 1787830346.141196 fallocate 0x03c905 thru 0x03ffff (0x36fb bytes) 000808 1787830346.161624 punch from 0x0307fb to 0x03d700 (0xcf06 bytes) 000809 1787830346.168804 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x566d5609 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x566d5609 Mirror 3: 0x566d5609 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000810 1787830346.539420 read 0x01854c thru 0x027c53 (0xf708 bytes) 000811 1787830346.543810 trunc from 0x040000 to 0x007dc9 (0x38236 bytes) 000812 1787830346.577867 fallocate 0x01b9ad thru 0x01f2d7 (0x392b bytes) 000813 1787830346.581279 mapread 0x01d089 thru 0x01f2d7 (0x224f bytes) 000814 1787830346.591606 mapwrite 0x030981 thru 0x03f1cf (0xe84f bytes) 000816 1787830346.624454 trunc from 0x02bc14 to 0x036e9a (0xb287 bytes) 000817 1787830346.627848 fallocate 0x006380 thru 0x00ea2d (0x86ae bytes) 000818 1787830346.631405 mapwrite 0x00bdee thru 0x012f22 (0x7135 bytes) 000820 1787830346.651465 write 0x027fc2 thru 0x037e23 (0xfe62 bytes) 000821 1787830346.656000 mapwrite 0x0071c0 thru 0x01587c (0xe6bd bytes) 000822 1787830346.672762 trunc from 0x03e244 to 0x03f1cf (0xf8c bytes) 000823 1787830346.675157 punch from 0x000581 to 0x00ef24 (0xe9a4 bytes) 000824 1787830346.680004 read 0x037185 thru 0x03f1cf (0x804b bytes) 000825 1787830346.687415 trunc from 0x0018c9 to 0x00d727 (0xbe5f bytes) 000826 1787830346.691290 trunc from 0x03f1d0 to 0x01e5a2 (0x20c2d bytes) 000827 1787830346.701251 write 0x025c75 thru 0x02cce1 (0x706d bytes) 000828 1787830346.702773 trunc from 0x0132c0 to 0x019a0a (0x674b bytes) 000829 1787830346.706858 mapread 0x0217c8 thru 0x02cce1 (0xb51a bytes) 000830 1787830346.724576 trunc from 0x00e357 to 0x01cb88 (0xe832 bytes) 000831 1787830346.728855 fallocate 0x025271 thru 0x025d6a (0xafa bytes) 000832 1787830346.732502 mapread 0x0096e2 thru 0x014066 (0xa985 bytes) 000834 1787830346.768336 trunc from 0x0256ba to 0x02965d (0x3fa4 bytes) 000835 1787830346.778969 fallocate 0x033a02 thru 0x03c1a4 (0x87a3 bytes) 000836 1787830346.782427 mapread 0x0057ce thru 0x00a4ba (0x4ced bytes) 000837 1787830346.799260 fallocate 0x024705 thru 0x02a45b (0x5d57 bytes) 000838 1787830346.803541 read 0x002327 thru 0x002b9f (0x879 bytes) 000839 1787830346.812033 fallocate 0x03e3d3 thru 0x03ffff (0x1c2d bytes) 000840 1787830346.816603 punch from 0x0059d1 to 0x00c1f1 (0x6821 bytes) 000841 1787830346.820740 fallocate 0x031f01 thru 0x034a35 (0x2b35 bytes) 000842 1787830346.826768 mapread 0x01cedf thru 0x02b990 (0xeab2 bytes) 000843 1787830346.828152 trunc from 0x040000 to 0x0377bb (0x8844 bytes) 000844 1787830346.838229 fallocate 0x00d559 thru 0x015967 (0x840f bytes) 000845 1787830346.842703 fallocate 0x01913e thru 0x01cd4a (0x3c0d bytes) 000846 1787830346.854608 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x377bc): Mirror 3: 0x120ab0d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x377bc): Mirror 3: 0x120ab0d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x377bc: skipped 000847 1787830347.036858 trunc from 0x0377bc to 0x01719f (0x2061c bytes) 000848 1787830347.049716 write 0x033a78 thru 0x03b4ca (0x7a53 bytes) 000849 1787830347.051566 read 0x008632 thru 0x01070a (0x80d9 bytes) 000850 1787830347.055857 fallocate 0x00fcf7 thru 0x01ec4e (0xef58 bytes) 000851 1787830347.059025 trunc from 0x028627 to 0x0349c4 (0xc39e bytes) 000852 1787830347.067383 mapwrite 0x019330 thru 0x01a4c8 (0x1199 bytes) 000853 1787830347.083992 read 0x030875 thru 0x0390a8 (0x8834 bytes) 000854 1787830347.094232 fallocate 0x015af4 thru 0x01aac9 (0x4fd6 bytes) 000855 1787830347.096997 mapread 0x00ca0c thru 0x00e851 (0x1e46 bytes) 000856 1787830347.100309 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b4cb): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4cb): Mirror 3: 0xafadaceb (stale) CRC-32 checksum value for chunk [0, 0x3b4cb): Mirror 1: 0xc9e68306 Mirror 4: 0xc9e68306 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4cb: skipped 000857 1787830347.580278 trunc from 0x03b4cb to 0x03e895 (0x33cb bytes) 000858 1787830347.640922 trunc from 0x03e896 to 0x0164c4 (0x283d1 bytes) 000859 1787830347.667913 mapwrite 0x03f0c3 thru 0x03ffff (0xf3d bytes) 000860 1787830347.691035 mapread 0x036629 thru 0x03a751 (0x4129 bytes) 000861 1787830347.704291 punch from 0x00eca9 to 0x01a68e (0xb9e6 bytes) 000862 1787830347.709390 read 0x020e56 thru 0x028fee (0x8199 bytes) 000863 1787830347.718162 punch from 0x001909 to 0x00c482 (0xab7a bytes) 000864 1787830347.724706 fallocate 0x024047 thru 0x02f0bd (0xb077 bytes) 000865 1787830347.729739 punch from 0x002f07 to 0x008a0a (0x5b04 bytes) 000866 1787830347.733944 mapwrite 0x016bc1 thru 0x017576 (0x9b6 bytes) 000867 1787830347.753982 trunc from 0x034ef9 to 0x0394a8 (0x45b0 bytes) 000869 1787830347.761895 write 0x028027 thru 0x02db30 (0x5b0a bytes) 000870 1787830347.769544 trunc from 0x040000 to 0x00454d (0x3bab2 bytes) 000871 1787830347.784660 trunc from 0x00454e to 0x00179e (0x2daf bytes) 000872 1787830347.802607 fallocate 0x032fd6 thru 0x0361c7 (0x31f2 bytes) 000873 1787830347.806694 punch from 0x0009f8 to 0x001735 (0xd3e bytes) 000874 1787830347.811049 punch from 0x000323 to 0x00179e (0x147c bytes) 000875 1787830347.814464 mapwrite 0x00a180 thru 0x013ca7 (0x9b28 bytes) 000876 1787830347.859160 write 0x00a14c thru 0x00f2ea (0x519f bytes) 000877 1787830347.865023 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13ca8): Mirror 3: 0x73354ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13ca8): Mirror 3: 0x73354ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13ca8: skipped 000878 1787830348.071297 mapwrite 0x03dfd1 thru 0x03ffff (0x202f bytes) 000879 1787830348.125431 write 0x01a6c1 thru 0x01ebe2 (0x4522 bytes) 000880 1787830348.133805 trunc from 0x034a66 to 0x03ffff (0xb59a bytes) 000881 1787830348.140016 fallocate 0x035f3f thru 0x0365e7 (0x6a9 bytes) 000882 1787830348.143703 fallocate 0x03787f thru 0x039724 (0x1ea6 bytes) 000883 1787830348.149485 punch from 0x038abd to 0x03ffff (0x7543 bytes) 000885 1787830348.153864 trunc from 0x019ac0 to 0x0225cf (0x8b10 bytes) 000886 1787830348.164482 punch from 0x02902a to 0x034224 (0xb1fb bytes) 000887 1787830348.168527 fallocate 0x021696 thru 0x022283 (0xbee bytes) 000888 1787830348.172561 punch from 0x00cb53 to 0x01742f (0xa8dd bytes) 000889 1787830348.181812 write 0x01f24f thru 0x0244db (0x528d bytes) 000890 1787830348.200720 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000891 1787830348.315459 fallocate 0x0093bf thru 0x00f8df (0x6521 bytes) 000893 1787830348.319799 trunc from 0x009a1a to 0x01869f (0xec86 bytes) 000895 1787830348.324524 write 0x00ca64 thru 0x010180 (0x371d bytes) 000896 1787830348.336561 mapwrite 0x012109 thru 0x0160eb (0x3fe3 bytes) 000897 1787830348.361040 read 0x018d0a thru 0x025df0 (0xd0e7 bytes) 000898 1787830348.367508 mapread 0x02273a thru 0x02e95d (0xc224 bytes) 000901 1787830348.375245 trunc from 0x040000 to 0x03b036 (0x4fc9 bytes) 000902 1787830348.385982 fallocate 0x0317f0 thru 0x039fd6 (0x87e7 bytes) 000903 1787830348.389397 punch from 0x0242f4 to 0x025d32 (0x1a3f bytes) 000904 1787830348.394532 trunc from 0x03b037 to 0x03979b (0x189b bytes) 000905 1787830348.409258 trunc from 0x03979c to 0x038c29 (0xb72 bytes) 000906 1787830348.424697 fallocate 0x01681e thru 0x019722 (0x2f05 bytes) 000907 1787830348.429957 fallocate 0x01c5b0 thru 0x020ccb (0x471c bytes) 000908 1787830348.434926 fallocate 0x03bd7e thru 0x03bd85 (0x8 bytes) 000909 1787830348.439009 mapwrite 0x00e0b1 thru 0x019149 (0xb099 bytes) 000910 1787830348.454848 trunc from 0x030db9 to 0x034bf3 (0x3e3b bytes) 000911 1787830348.459284 trunc from 0x03a377 to 0x03bd85 (0x1a0f bytes) 000913 1787830348.465123 read 0x0247ab thru 0x02a053 (0x58a9 bytes) 000914 1787830348.472472 mapwrite 0x03ed00 thru 0x03ffff (0x1300 bytes) 000915 1787830348.495916 trunc from 0x040000 to 0x0207d5 (0x1f82a bytes) 000916 1787830348.506361 fallocate 0x01d3a4 thru 0x02c59c (0xf1f9 bytes) 000918 1787830348.511526 punch from 0x0043a6 to 0x00ff44 (0xbb9f bytes) 000922 1787830348.527533 MIRROR_EXTEND to 2 mirrors 000923 1787830348.902571 MIRROR_EXTEND to 3 mirrors 000924 1787830349.239366 mapwrite 0x00fb48 thru 0x0142e7 (0x47a0 bytes) 000925 1787830349.288016 trunc from 0x0207d6 to 0x01ddbd (0x2a18 bytes) 000926 1787830349.303567 trunc from 0x00fb16 to 0x012238 (0x2723 bytes) 000927 1787830349.307451 trunc from 0x000301 to 0x008cd2 (0x89d2 bytes) 000928 1787830349.311629 write 0x00fc52 thru 0x0166d1 (0x6a80 bytes) 000929 1787830349.323253 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ddbe): Mirror 3: 0x3e6ee37f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ddbe): Mirror 3: 0x3e6ee37f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ddbe: skipped 000930 1787830349.470461 trunc from 0x01ddbe to 0x01f7e6 (0x1a29 bytes) 000931 1787830349.486081 punch from 0x0067c1 to 0x008235 (0x1a75 bytes) 000932 1787830349.492070 mapread 0x013b2c thru 0x01a733 (0x6c08 bytes) 000933 1787830349.503574 trunc from 0x01f7e7 to 0x03e42c (0x1ec46 bytes) 000934 1787830349.514629 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e42d): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e42d): Mirror 3: 0x7dfafc25 (stale) CRC-32 checksum value for chunk [0, 0x3e42d): Mirror 1: 0xc5abb67 Mirror 4: 0xc5abb67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e42d: skipped 000935 1787830349.948431 write 0x01d0c9 thru 0x02a082 (0xcfba bytes) 000936 1787830349.983796 mapwrite 0x034b71 thru 0x03facb (0xaf5b bytes) 000937 1787830350.034868 punch from 0x037fd2 to 0x03facb (0x7afa bytes) 000938 1787830350.039139 fallocate 0x029b14 thru 0x02e500 (0x49ed bytes) 000939 1787830350.046368 write 0x02fc22 thru 0x0315fe (0x19dd bytes) 000940 1787830350.052601 trunc from 0x023be0 to 0x0314ee (0xd90f bytes) 000941 1787830350.063494 trunc from 0x03f733 to 0x03facb (0x399 bytes) 000943 1787830350.074009 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3facc): Mirror 3: 0x5fbab62a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3facc): Mirror 3: 0x5fbab62a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3facc: skipped 000944 1787830350.243121 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000945 1787830350.313785 mapwrite 0x0251d1 thru 0x02a140 (0x4f70 bytes) 000946 1787830350.352245 fallocate 0x0226d6 thru 0x02fe2d (0xd758 bytes) 000947 1787830350.355490 fallocate 0x01af76 thru 0x0269e8 (0xba73 bytes) 000948 1787830350.360946 MIRROR_EXTEND to 2 mirrors 000949 1787830350.757983 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000950 1787830350.839306 MIRROR_EXTEND to 2 mirrors 000951 1787830351.249043 write 0x0067e1 thru 0x0109d0 (0xa1f0 bytes) 000952 1787830351.276028 trunc from 0x03facc to 0x034db2 (0xad19 bytes) 000953 1787830351.285807 read 0x01401d thru 0x021c8b (0xdc6f bytes) 000954 1787830351.291558 trunc from 0x00c39a to 0x01c1a4 (0xfe0b bytes) 000955 1787830351.308036 mapread 0x0272aa thru 0x02c8c3 (0x561a bytes) 000957 1787830351.323949 trunc from 0x034db3 to 0x0185c3 (0x1c7ef bytes) 000959 1787830351.335559 write 0x02fc01 thru 0x034da8 (0x51a8 bytes) 000960 1787830351.336741 punch from 0x023a64 to 0x02c9da (0x8f77 bytes) 000961 1787830351.341054 trunc from 0x00ae3c to 0x015ec6 (0xb08b bytes) 000962 1787830351.348072 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34da9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34da9): Mirror 2: 0xd72ca1b4 (stale) CRC-32 checksum value for chunk [0, 0x34da9): Mirror 1: 0x3dd2f815 Mirror 3: 0x3dd2f815 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34da9: skipped 000963 1787830351.778972 trunc from 0x034da9 to 0x03da92 (0x8cea bytes) 000964 1787830351.818950 read 0x03241a thru 0x032841 (0x428 bytes) 000965 1787830351.823893 write 0x020bb1 thru 0x026042 (0x5492 bytes) 000966 1787830351.828877 mapwrite 0x002fce thru 0x00912f (0x6162 bytes) 000967 1787830351.852342 read 0x01c7e2 thru 0x02510f (0x892e bytes) 000968 1787830351.861315 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3da93): Mirror 3: 0xf2630076 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da93): Mirror 3: 0xf2630076 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da93: skipped 000969 1787830352.013084 write 0x002f98 thru 0x00c44d (0x94b6 bytes) 000970 1787830352.021594 trunc from 0x01b24a to 0x01b483 (0x23a bytes) 000972 1787830352.026375 fallocate 0x015692 thru 0x01c4ec (0x6e5b bytes) 000973 1787830352.029382 read 0x00de2c thru 0x00e64e (0x823 bytes) 000975 1787830352.033387 trunc from 0x00e415 to 0x01e004 (0xfbf0 bytes) 000976 1787830352.037500 punch from 0x011689 to 0x01581c (0x4194 bytes) 000977 1787830352.040139 fallocate 0x0296b6 thru 0x02d704 (0x404f bytes) 000978 1787830352.044142 trunc from 0x00e973 to 0x01d17b (0xe809 bytes) 000979 1787830352.050527 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3da93): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da93): Mirror 3: 0xf2630076 (stale) CRC-32 checksum value for chunk [0, 0x3da93): Mirror 1: 0x61212433 Mirror 4: 0x61212433 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da93: skipped 000980 1787830352.506083 mapwrite 0x033dfc thru 0x03986c (0x5a71 bytes) 000981 1787830352.568522 trunc from 0x03da93 to 0x01ff99 (0x1daf9 bytes) 000982 1787830352.591942 mapwrite 0x006f47 thru 0x013915 (0xc9cf bytes) 000983 1787830352.614795 punch from 0x012a55 to 0x01ff99 (0xd545 bytes) 000984 1787830352.617767 trunc from 0x01ff9a to 0x02f34b (0xf3b2 bytes) 000985 1787830352.625158 mapread 0x008411 thru 0x0100e7 (0x7cd7 bytes) 000987 1787830352.626681 trunc from 0x02f34c to 0x034a9a (0x574f bytes) 000990 1787830352.635374 mapwrite 0x02ae8d thru 0x034148 (0x92bc bytes) 000991 1787830352.652351 punch from 0x0314f1 to 0x034a9a (0x35aa bytes) 000992 1787830352.655667 trunc from 0x027e52 to 0x031fbb (0xa16a bytes) 000994 1787830352.659043 punch from 0x015f14 to 0x018fa0 (0x308d bytes) 000995 1787830352.661736 read 0x003c9c thru 0x00452e (0x893 bytes) 000996 1787830352.664263 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34a9b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a9b): Mirror 3: 0xb734a852 (stale) Mirror 4: 0xdb3b6808 (stale) CRC-32 checksum value for chunk [0, 0x34a9b): Mirror 1: 0x6ea63bc2 Mirror 5: 0x6ea63bc2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a9b: skipped 000997 1787830353.108546 fallocate 0x01fa19 thru 0x02de55 (0xe43d bytes) 000998 1787830353.136495 trunc from 0x034a9b to 0x0163f3 (0x1e6a7 bytes) 000999 1787830353.163569 mapread 0x00b3d0 thru 0x00b573 (0x1a4 bytes) 001000 1787830353.166871 mapread 0x0001c3 thru 0x009634 (0x9472 bytes) All operations completed A-OK! PASS 70a (34s) == sanity-flr test 71: check mirror extend parent fid ==== 07:32:35 (1787830355) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x3da:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 72: check append on FLR file ========== 07:32:39 (1787830359) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.082499 s, 25.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464705 s, 22.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 07:32:42 (1787830362) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4057344 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8585824 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7318208 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8271104 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8654752 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9022784 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8930208 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9131136 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 64145 64146 64148 64150 64151 done 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) 192.168.203.102@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:33:49 (1787830429) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3789952 .. Extending file size to 687616 .. Extending file size to 2252992 .. Extending file size to 4572672 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6459936 .. Extending file size to 6321504 .. Extending file size to 8689632 .. Extending file size to 7365600 .. Extending file size to 7923744 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8789856 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9006976 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9111264 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9184608 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9298176 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 67465 67466 67467 67468 67472 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) 192.168.203.102@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:34:56 (1787830496) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2756 1268548 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2384 1268920 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23072 3583948 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 20980 3586040 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44052 7169988 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2756 1268548 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2384 1268920 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 23072 3583948 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 20980 3586040 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44052 7169988 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0726554 s, 28.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12719 s, 16.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xd95f559b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) PASS 200c (11s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:35:08 (1787830508) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0523505 s, 40.1 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 07:35:12 (1787830512) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x157:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x157:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x158:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:35:15 (1787830515) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0578493 s, 36.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x560:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x561:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:35:19 (1787830519) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0567899 s, 36.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x159:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x159:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:35:24 (1787830524) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0367497 s, 57.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x562:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x562:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x563:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:35:50 (1787830550) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg302-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517197 s, 20.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:36:25 (1787830585) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg302-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0384275 s, 54.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:37:01 (1787830621) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg302-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg302-server: Pool lustre.test_204f created Adding targets to pool oleg302-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg302-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0658942 s, 31.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg302-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg302-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg302-server: Pool lustre.test_204f destroyed PASS 204f (53s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:37:54 (1787830674) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:37:58 (1787830678) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:38:02 (1787830682) oleg302-server: Pool lustre.test_206 created Adding targets to pool oleg302-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg302-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg302-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg302-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg302-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:38:20 (1787830700) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372494 s, 28.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234596 s, 44.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3c2738e4 Mirror 3: 0x3c2738e4 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:38:25 (1787830705) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:38:26 (1787830706) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:38:28 (1787830708) Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0192725 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2290000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b00000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b00000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b00000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2280000 Read this (4 bytes): BBBB 192.168.203.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg302-client.virtnet /mnt/lustre2 (opts:) PASS 209a (24s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:38:52 (1787830732) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0114873 s, 713 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:38:56 (1787830736) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347363 s, 30.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc5 lmm_fid: [0x200000405:0xc5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc5 lmm_fid: [0x200000405:0xc5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x168:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611333 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 07:38:57.000000000 -0400 Modify: 2026-08-27 07:38:57.000000000 -0400 Change: 2026-08-27 07:38:58.000000000 -0400 Birth: 2026-08-27 07:38:56.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:39:01 (1787830741) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205649 s, 51.0 MB/s oleg302-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg302-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg302-server: O/280000401/d7/167: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:39:15 (1787830755) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.144509 s, 283 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0115316 s, 355 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 07:39 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xca:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 07:39 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:39:20 (1787830760) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:39:25 (1787830765) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 947 sec ============ 07:39:30 (1787830770) === sanity-flr: start cleanup 07:39:30 (1787830770) === === sanity-flr: finish cleanup 07:39:35 (1787830775) ===