-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 04:46:53 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg153-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 04:47:07 (1787734027) === oleg153-client.virtnet: executing check_config_client /mnt/lustre oleg153-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg153-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88e748c59800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88e748c59800.idle_timeout=debug disable quota as required oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 04:47:42 (1787734062) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8225] == sanity-flr test 0a: lfs mirror create with -N option == 04:47:44 (1787734064) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:47:56 (1787734076) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:47:58 (1787734078) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:48:00 (1787734080) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:48:19 (1787734099) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:48:22 (1787734102) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 04:48:25 (1787734105) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88e748c59800.stats=clear osc.lustre-OST0001-osc-ffff88e748c59800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:48:32 (1787734112) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:48:42 (1787734122) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:48:49 (1787734129) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:48:52 (1787734132) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:48:59 (1787734139) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 04:49:08 (1787734148) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:49:15 (1787734155) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:49:25 (1787734165) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:49:32 (1787734172) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:49:41 (1787734181) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:49:48 (1787734188) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.603439 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092017 s, 11.4 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:49:56 (1787734196) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.718845 s, 27.7 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:50:04 (1787734204) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:50:15 (1787734215) Stopping /mnt/lustre-ost1 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88e748c59800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88e748c59800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg153-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg153-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88e748c59800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88e748c59800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg153-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (133s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:52:28 (1787734348) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.567272 s, 20.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88e748c59800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88e748c59800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg153-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (37s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:53:05 (1787734385) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88e748c59800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88e748c59800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg153-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88e748c59800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88e748c59800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg153-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (71s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:54:16 (1787734456) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.80633 s, 18.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 04:54 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.992s user 0m0.002s sys 0m0.187s Stopping /mnt/lustre-ost1 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88e748c59800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88e748c59800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.821s user 0m0.000s sys 0m0.163s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg153-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg153-server oleg153-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88e748c59800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88e748c59800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.057s user 0m0.000s sys 0m0.188s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg153-server: oleg153-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg153-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg153-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88e748c59800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88e748c59800.ost_server_uuid in FULL state after 0 sec PASS 33b (81s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:55:38 (1787734538) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:55:41 (1787734541) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:55:45 (1787734545) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:55:48 (1787734548) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 04:55:55 (1787734555) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.569876 s, 22.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:56:05 (1787734565) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204952 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201944 s, 51.9 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:56:11 (1787734571) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:56:18 (1787734578) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00345643 s, 296 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00122734 s, 834 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00133903 s, 765 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111584 s, 918 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 04:56:28 (1787734588) create test files with size 15750381 .. f8edc39e8050b4fcfc492b5a85b1a00e - 398eab9b58dad8ff9f6f7eef12b2db77 - 1744c66f262a35e004f12d51b9d4fd83 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg153-client.virtnet /mnt/lustre (opts:) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 04:56:48 (1787734608) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:57:00 (1787734620) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:57:06 (1787734626) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484845 s, 21.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259374 s, 40.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 04:57:15 (1787734635) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0680724 s, 30.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.370644 s, 11.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193389 s, 10.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 04:57:33 (1787734653) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 04:57:35 (1787734655) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:57:39 (1787734659) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019407 s, 54.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 04:57:46 (1787734666) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.233679 s, 13.5 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:57:56 (1787734676) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:58:03 (1787734683) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.281848 s, 37.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg153-client.virtnet /mnt/lustre (opts:) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:58:12 (1787734692) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88e750ae6800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.272305 s, 38.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg153-client.virtnet /mnt/lustre (opts:) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88e76c0c3800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:58:26 (1787734706) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:59:02 (1787734742) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:59:36 (1787734776) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 04:59:39 (1787734779) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.272814 s, 11.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.124094 s, 25.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf2ddd032 (nosync,stale) Mirror 2: 0xf2ddd032 (nosync,stale) ** verify mirror 2 doesn't change original checksum: ff80917e4f677a4ac29f2c0188df5372 - mirror 1 checksum: 6aa2986f451985f6ef45de13fcd50810 - mirror 2 checksum: ff80917e4f677a4ac29f2c0188df5372 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T04:59:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T04:59:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ff80917e4f677a4ac29f2c0188df5372 - mirror 1 checksum: 6aa2986f451985f6ef45de13fcd50810 - mirror 2 checksum: 6aa2986f451985f6ef45de13fcd50810 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T04:59:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T04:59:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 04:59:48 (1787734788) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262538 s, 39.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:59:54 (1787734794) Failing mds1 on oleg153-server Stopping /mnt/lustre-mds1 (opts:) on oleg153-server 05:00:05 (1787734805) shut down facet: mds1 facet_host: oleg153-server facet_failover_host: oleg153-server Failover mds1 to oleg153-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg153-server: oleg153-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0000 05:00:39 (1787734839) targets are mounted 05:00:39 (1787734839) facet_failover done oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428955 s, 24.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:00:55 (1787734855) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0428215 s, 478 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.012242 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:01:05 (1787734865) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.051774 s, 396 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0115215 s, 1.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:01:14 (1787734874) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00652998 s, 2.5 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:01:22 (1787734882) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.031215 s, 656 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:01:30 (1787734890) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0174112 s, 235 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.037884 s, 108 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:01:39 (1787734899) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:02:00 (1787734920) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:02:26 (1787734946) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:02:48 (1787734968) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:03:09 (1787734989) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 05:03:15 (1787734995) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57565 mirror split pid 57567 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57565 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57567 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 05:04:23 (1787735063) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 5065 skipping zero size read skipping zero size read truncating to largest ever: 0x229e7 000006 1787735066.050242 trunc from 00000000 to 0x0229e6 (0x229e7 bytes) 000007 1787735066.083159 trunc from 0x01ab79 to 0x01cfcb (0x2453 bytes) 000008 1787735066.090427 mapread 0x0058ad thru 0x00977b (0x3ecf bytes) 000009 1787735066.134081 punch from 0x00edaa to 0x019845 (0xaa9c bytes) 000010 1787735066.144560 read 0x01b0c7 thru 0x0229e6 (0x7920 bytes) 000011 1787735066.162483 fallocate 0x00e600 thru 0x01404a (0x5a4b bytes) 000012 1787735066.172997 trunc from 0x01b1f8 to 0x0229e6 (0x77ef bytes) 000014 1787735066.181247 write 0x018852 thru 0x01e9a1 (0x6150 bytes) 000015 1787735066.194760 mapread 0x0030aa thru 0x00735b (0x42b2 bytes) 000017 1787735066.198332 write 0x02aeeb thru 0x0313d1 (0x64e7 bytes) 000018 1787735066.200169 write 0x030379 thru 0x035e3d (0x5ac5 bytes) 000019 1787735066.202143 read 0x00d7cd thru 0x01861a (0xae4e bytes) truncating to largest ever: 0x3d1e7 000020 1787735066.216291 trunc from 0x035e3e to 0x03d1e6 (0x73a9 bytes) 000021 1787735066.239867 trunc from 0x0155da to 0x0174b7 (0x1ede bytes) 000022 1787735066.244470 mapwrite 0x02ae72 thru 0x033159 (0x82e8 bytes) 000023 1787735066.270264 read 0x01460e thru 0x017261 (0x2c54 bytes) 000024 1787735066.286490 write 0x0324a8 thru 0x03ffff (0xdb58 bytes) 000025 1787735066.288053 write 0x012586 thru 0x01f146 (0xcbc1 bytes) 000026 1787735066.291413 mapwrite 0x025533 thru 0x026ad4 (0x15a2 bytes) 000027 1787735066.319416 trunc from 0x0068af to 0x011714 (0xae66 bytes) 000028 1787735066.340072 trunc from 0x000f3e to 0x002f90 (0x2053 bytes) 000030 1787735066.354466 fallocate 0x00f3de thru 0x012cd9 (0x38fc bytes) 000032 1787735066.362204 read 0x03bca1 thru 0x03ffff (0x435f bytes) 000033 1787735066.366591 punch from 0x01952f to 0x01d3b3 (0x3e85 bytes) 000034 1787735066.376176 write 0x01c841 thru 0x01d991 (0x1151 bytes) 000035 1787735066.393207 trunc from 0x040000 to 0x021ed0 (0x1e12f bytes) 000036 1787735066.435261 read 0x019f8f thru 0x01aa37 (0xaa9 bytes) 000037 1787735066.436053 write 0x02f1a7 thru 0x02fbe2 (0xa3c bytes) 000038 1787735066.452422 MIRROR_EXTEND to 2 mirrors 000039 1787735067.241420 write 0x00a5a8 thru 0x00a8b7 (0x310 bytes) 000040 1787735067.318236 punch from 0x01b033 to 0x029031 (0xdfff bytes) 000041 1787735067.331567 write 0x016561 thru 0x016efb (0x99b bytes) 000042 1787735067.340221 write 0x03dfe8 thru 0x03ffff (0x2018 bytes) 000043 1787735067.345306 read 0x018bee thru 0x025978 (0xcd8b bytes) 000045 1787735067.383435 MIRROR_RESYNC 2 mirrors 000047 1787735068.125419 punch from 0x0361c6 to 0x03fb27 (0x9962 bytes) 000048 1787735068.159934 fallocate 0x0177a6 thru 0x018fed (0x1848 bytes) 000049 1787735068.168160 punch from 0x00804b to 0x00de7e (0x5e34 bytes) zero_range to largest ever: 0x40000 000050 1787735068.173964 trunc from 0x03a912 to 0x03ffff (0x56ee bytes) 000051 1787735068.179872 trunc from 0x040000 to 0x017af7 (0x28508 bytes) 000052 1787735068.218464 mapwrite 0x031d04 thru 0x036629 (0x4926 bytes) 000053 1787735068.272937 write 0x01a55c thru 0x01c1ba (0x1c5f bytes) 000054 1787735068.289882 read 0x00252d thru 0x00ac66 (0x873a bytes) 000055 1787735068.300272 fallocate 0x017ab5 thru 0x0222ba (0xa806 bytes) 000056 1787735068.307114 read 0x02097d thru 0x0286c3 (0x7d47 bytes) 000057 1787735068.319113 trunc from 0x014432 to 0x0167b9 (0x2388 bytes) 000058 1787735068.325716 mapwrite 0x034b0d thru 0x039fe7 (0x54db bytes) 000059 1787735068.373177 trunc from 0x0126d7 to 0x01cb19 (0xa443 bytes) 000060 1787735068.382183 mapwrite 0x000f9e thru 0x00b94e (0xa9b1 bytes) 000061 1787735068.407785 trunc from 0x014ae7 to 0x0195e7 (0x4b01 bytes) 000062 1787735068.414303 trunc from 0x00b231 to 0x018a74 (0xd844 bytes) 000063 1787735068.421011 trunc from 0x039fe8 to 0x0291d4 (0x10e13 bytes) 000064 1787735068.444310 MIRROR_RESYNC 2 mirrors 000065 1787735069.044300 trunc from 0x01ca0f to 0x0291d4 (0xc7c6 bytes) 000066 1787735069.085254 mapread 0x014551 thru 0x018d1b (0x47cb bytes) 000067 1787735069.098945 mapread 0x02891b thru 0x0291d4 (0x8ba bytes) 000068 1787735069.105315 read 0x01e5a7 thru 0x028c67 (0xa6c1 bytes) 000069 1787735069.110438 read 0x021f3d thru 0x026c75 (0x4d39 bytes) 000070 1787735069.113914 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x291d5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x291d5): Mirror 2: 0x5cb7ff2a (stale) CRC-32 checksum value for chunk [0, 0x291d5): Mirror 1: 0x5cb7ff2a Mirror 3: 0x5cb7ff2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x291d5: skipped 000071 1787735070.147899 punch from 0x000dee to 0x008492 (0x76a5 bytes) 000072 1787735070.190117 fallocate 0x017a70 thru 0x021e91 (0xa422 bytes) 000073 1787735070.197236 trunc from 0x0291d5 to 0x0266dd (0x2af7 bytes) 000074 1787735070.235297 mapread 0x022e15 thru 0x02351e (0x70a bytes) 000075 1787735070.256078 trunc from 0x00ad8b to 0x014a6b (0x9ce1 bytes) 000076 1787735070.269071 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x266de): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x266de): Mirror 2: 0x9036a507 (stale) Mirror 3: 0x9036a507 (stale) CRC-32 checksum value for chunk [0, 0x266de): Mirror 1: 0xe4aa6f6c Mirror 4: 0xe4aa6f6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x266de: skipped 000077 1787735071.179224 mapread 0x021388 thru 0x0266dd (0x5356 bytes) 000078 1787735071.211369 read 0x01e949 thru 0x0213ec (0x2aa4 bytes) 000079 1787735071.211856 fallocate 0x0329d9 thru 0x03ffff (0xd627 bytes) 000080 1787735071.300604 mapwrite 0x028ade thru 0x034bfe (0xc121 bytes) 000081 1787735071.387024 mapwrite 0x020268 thru 0x02e3e3 (0xe17c bytes) 000082 1787735071.411299 read 0x022097 thru 0x02b2a5 (0x920f bytes) 000083 1787735071.412242 mapwrite 0x03be1d thru 0x03ffff (0x41e3 bytes) 000085 1787735071.448772 trunc from 0x019242 to 0x027fa1 (0xed60 bytes) 000088 1787735071.457669 trunc from 0x040000 to 0x0059c6 (0x3a639 bytes) 000089 1787735071.533001 trunc from 0x0050f5 to 0x0059c6 (0x8d2 bytes) 000090 1787735071.543925 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x59c7): Mirror 3: 0x664a2fda (stale) Mirror 4: 0xcced7803 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x59c7): Mirror 3: 0x664a2fda (stale) Mirror 4: 0xcced7803 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x59c7: skipped 000091 1787735071.861503 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x59c7): Mirror 4: 0xcced7803 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x59c7): Mirror 4: 0xcced7803 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x59c7: skipped 000092 1787735072.129299 trunc from 0x002bea to 0x0059c6 (0x2ddd bytes) 000093 1787735072.133945 trunc from 0x0059c7 to 0x000db1 (0x4c15 bytes) 000094 1787735072.150071 write 0x014d40 thru 0x01bf7a (0x723b bytes) 000095 1787735072.151496 trunc from 0x01bf7b to 0x03a1ae (0x1e234 bytes) 000097 1787735072.171673 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a1af): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1af): Mirror 4: 0x8563005b (stale) CRC-32 checksum value for chunk [0, 0x3a1af): Mirror 1: 0x8f2c29a7 Mirror 5: 0x8f2c29a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1af: skipped 000098 1787735072.930640 trunc from 0x001eff to 0x01153f (0xf641 bytes) 000099 1787735072.976207 trunc from 0x03a1af to 0x01f662 (0x1ab4c bytes) 000100 1787735072.998830 trunc from 0x01f663 to 0x02328e (0x3c2c bytes) 000101 1787735073.022827 trunc from 0x02328f to 0x006a13 (0x1c87b bytes) 000102 1787735073.038939 trunc from 0x006a14 to 0x007dba (0x13a7 bytes) 000103 1787735073.053526 write 0x024957 thru 0x02ec85 (0xa32f bytes) 000104 1787735073.054760 punch from 0x022197 to 0x02ec85 (0xcaef bytes) 000106 1787735073.089521 MIRROR_RESYNC 3 mirrors 000109 1787735073.822888 write 0x030a09 thru 0x037e36 (0x742e bytes) 000110 1787735073.846013 fallocate 0x00f786 thru 0x01e0cf (0xe94a bytes) 000111 1787735073.850438 fallocate 0x0244a8 thru 0x032fd0 (0xeb29 bytes) 000112 1787735073.857116 trunc from 0x037e37 to 0x01d6f6 (0x1a740 bytes) 000113 1787735073.876918 fallocate 0x01567b thru 0x01a8f0 (0x5276 bytes) 000114 1787735073.883257 mapwrite 0x00fcf4 thru 0x01e6da (0xe9e7 bytes) 000115 1787735073.941992 trunc from 0x01e6db to 0x008316 (0x163c4 bytes) 000116 1787735073.959731 write 0x0335c1 thru 0x03ffff (0xca3f bytes) 000117 1787735073.966292 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x603dea3b Mirror 6: 0x603dea3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000119 1787735074.657057 mapread 0x029b0e thru 0x02d59e (0x3a91 bytes) 000120 1787735074.677020 mapwrite 0x0312f3 thru 0x03ffff (0xed0d bytes) 000121 1787735074.774122 read 0x02ccb3 thru 0x039c81 (0xcfcf bytes) 000122 1787735074.774799 trunc from 0x040000 to 0x037c9b (0x8364 bytes) 000123 1787735074.806927 punch from 0x0364e4 to 0x036673 (0x190 bytes) 000124 1787735074.813326 write 0x02b585 thru 0x030a01 (0x547d bytes) 000125 1787735074.825131 read 0x007243 thru 0x012bdb (0xb999 bytes) 000126 1787735074.842613 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37c9c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c9c): Mirror 4: 0x22be109e (stale) Mirror 5: 0x22be109e (stale) Mirror 6: 0x2d8135dc (stale) CRC-32 checksum value for chunk [0, 0x37c9c): Mirror 1: 0x7a1cdef5 Mirror 7: 0x7a1cdef5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c9c: skipped 000127 1787735075.693475 mapread 0x0135f6 thru 0x01b81f (0x822a bytes) 000128 1787735075.724990 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c9c): Mirror 4: 0x22be109e (stale) Mirror 6: 0x2d8135dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c9c): Mirror 4: 0x22be109e (stale) Mirror 6: 0x2d8135dc (stale) CRC-32 checksum value for chunk [0, 0x37c9c): Mirror 1: 0x7a1cdef5 Mirror 7: 0x7a1cdef5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c9c: skipped 000129 1787735076.249024 read 0x024a8c thru 0x0334bb (0xea30 bytes) 000130 1787735076.268013 trunc from 0x037c9c to 0x03d944 (0x5ca9 bytes) 000131 1787735076.397617 punch from 0x01de00 to 0x02608d (0x828e bytes) 000132 1787735076.415358 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d945): Mirror 4: 0xd53ce195 (stale) Mirror 6: 0x51e0f178 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d945): Mirror 4: 0xd53ce195 (stale) Mirror 6: 0x51e0f178 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d945: skipped 000133 1787735076.772622 mapread 0x02cbc6 thru 0x039797 (0xcbd2 bytes) 000134 1787735076.796923 write 0x00aa68 thru 0x016ded (0xc386 bytes) 000135 1787735076.810997 fallocate 0x03d159 thru 0x03ffff (0x2ea7 bytes) 000136 1787735076.817053 mapread 0x0082cc thru 0x015ebb (0xdbf0 bytes) 000137 1787735076.836367 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 6: 0x603dea3b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5f7efe92 Mirror 7: 0x5f7efe92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000138 1787735077.617506 punch from 0x016c55 to 0x01dc4e (0x6ffa bytes) 000139 1787735077.672775 trunc from 0x01bd6a to 0x0287d9 (0xca70 bytes) 000140 1787735077.685510 mapwrite 0x0200ce thru 0x022816 (0x2749 bytes) 000141 1787735077.743797 trunc from 0x02df1d to 0x03aa33 (0xcb17 bytes) 000142 1787735077.756029 mapread 0x02b205 thru 0x02ca96 (0x1892 bytes) 000143 1787735077.799732 read 0x039bdf thru 0x03ffff (0x6421 bytes) 000145 1787735077.817209 read 0x00df48 thru 0x01398c (0x5a45 bytes) 000146 1787735077.828883 mapread 0x02c1e7 thru 0x039aa5 (0xd8bf bytes) 000147 1787735077.829990 read 0x007eb0 thru 0x012d0d (0xae5e bytes) 000148 1787735077.831406 mapread 0x030c83 thru 0x03b5a4 (0xa922 bytes) 000149 1787735077.832710 read 0x02ca98 thru 0x034b0e (0x8077 bytes) 000150 1787735077.833533 trunc from 0x03b921 to 0x03ffff (0x46df bytes) 000151 1787735077.839154 write 0x00df1c thru 0x016425 (0x850a bytes) 000152 1787735077.859870 MIRROR_RESYNC 4 mirrors 000154 1787735078.817476 MIRROR_RESYNC 4 mirrors 000155 1787735079.162768 read 0x03e7cd thru 0x03ffff (0x1833 bytes) 000156 1787735079.171788 punch from 0x01acb2 to 0x01d27d (0x25cc bytes) 000157 1787735079.214401 punch from 0x03b828 to 0x03ffff (0x47d8 bytes) 000158 1787735079.220660 punch from 0x00b6f3 to 0x00edae (0x36bc bytes) 000159 1787735079.229527 MIRROR_RESYNC 4 mirrors 000160 1787735080.368466 mapread 0x000521 thru 0x009a55 (0x9535 bytes) 000161 1787735080.390056 mapread 0x017357 thru 0x01ef3e (0x7be8 bytes) 000162 1787735080.399173 read 0x038ac8 thru 0x03ffff (0x7538 bytes) 000164 1787735080.408108 mapread 0x030266 thru 0x03dc9e (0xda39 bytes) 000165 1787735080.415137 trunc from 0x00aa49 to 0x01877c (0xdd34 bytes) 000166 1787735080.451942 mapread 0x00d870 thru 0x01cc0b (0xf39c bytes) 000167 1787735080.470452 write 0x004b51 thru 0x013242 (0xe6f2 bytes) 000168 1787735080.475776 read 0x032d6a thru 0x03e876 (0xbb0d bytes) 000169 1787735080.484069 mapread 0x00eff9 thru 0x01682e (0x7836 bytes) 000170 1787735080.485075 trunc from 0x040000 to 0x015d83 (0x2a27c bytes) 000171 1787735080.506044 trunc from 0x00e10e to 0x015d83 (0x7c76 bytes) 000172 1787735080.519882 trunc from 0x01338f to 0x015d83 (0x29f5 bytes) 000173 1787735080.523068 write 0x03b5da thru 0x03ffff (0x4a26 bytes) 000174 1787735080.526391 MIRROR_RESYNC 4 mirrors 000175 1787735081.325784 trunc from 0x029d5b to 0x035b33 (0xbdd9 bytes) 000177 1787735081.372948 read 0x03f350 thru 0x03ffff (0xcb0 bytes) 000178 1787735081.381145 trunc from 0x02c7c8 to 0x02d9b7 (0x11f0 bytes) 000179 1787735081.385172 punch from 0x024027 to 0x02432e (0x308 bytes) 000180 1787735081.389051 trunc from 0x014fb3 to 0x019aee (0x4b3c bytes) 000181 1787735081.392772 trunc from 0x040000 to 0x02a23d (0x15dc2 bytes) 000182 1787735081.419595 read 0x025fcd thru 0x028991 (0x29c5 bytes) 000183 1787735081.435997 write 0x01d6e2 thru 0x029cc6 (0xc5e5 bytes) 000184 1787735081.437456 read 0x026a92 thru 0x02942a (0x2999 bytes) 000185 1787735081.439449 fallocate 0x01036a thru 0x0111f3 (0xe8a bytes) 000186 1787735081.443971 trunc from 0x02a23e to 0x01b1f5 (0xf048 bytes) 000187 1787735081.481414 trunc from 0x01b1f6 to 0x001d9a (0x1945b bytes) 000188 1787735081.503943 fallocate 0x01f987 thru 0x02f331 (0xf9ab bytes) 000189 1787735081.511405 trunc from 0x000eec to 0x001d9a (0xeaf bytes) 000190 1787735081.524661 trunc from 0x0001f0 to 0x001d9a (0x1bab bytes) 000191 1787735081.537047 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1d9b): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d9b): Mirror 4: 0x9e6116e9 (stale) Mirror 6: 0x9e6116e9 (stale) Mirror 7: 0x9e6116e9 (stale) CRC-32 checksum value for chunk [0, 0x1d9b): Mirror 1: 0x9e6116e9 Mirror 8: 0x9e6116e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d9b: skipped 000192 1787735082.081747 trunc from 0x000625 to 0x001d9a (0x1776 bytes) 000193 1787735082.106320 punch from 0x000863 to 0x001d9a (0x1538 bytes) 000194 1787735082.109392 punch from 0x000ffb to 0x001d9a (0xda0 bytes) 000195 1787735082.113194 trunc from 0x00116a to 0x001d9a (0xc31 bytes) 000196 1787735082.117139 read 0x000d56 thru 0x001d9a (0x1045 bytes) 000198 1787735082.122086 read 0x000606 thru 0x001d9a (0x1795 bytes) 000199 1787735082.122762 mapread 0x001ac5 thru 0x001d9a (0x2d6 bytes) 000200 1787735082.127945 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1d9b): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d9b): Mirror 4: 0x9e6116e9 (stale) Mirror 6: 0x9e6116e9 (stale) Mirror 7: 0x9e6116e9 (stale) Mirror 8: 0x9e6116e9 (stale) CRC-32 checksum value for chunk [0, 0x1d9b): Mirror 1: 0x9e6116e9 Mirror 9: 0x9e6116e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d9b: skipped 000201 1787735082.832362 trunc from 0x001d9b to 0x0193f6 (0x1765c bytes) 000202 1787735082.883751 punch from 0x0004f4 to 0x002db0 (0x28bd bytes) 000203 1787735082.890866 punch from 0x007ada to 0x00ad29 (0x3250 bytes) 000204 1787735082.899682 trunc from 0x0193f7 to 0x00e808 (0xabee bytes) 000205 1787735082.917385 trunc from 0x00e809 to 0x01557a (0x6d72 bytes) 000206 1787735082.936316 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1557b): Mirror 4: 0x5b54d55d (stale) Mirror 6: 0x5b54d55d (stale) Mirror 7: 0x5b54d55d (stale) Mirror 8: 0xdbe5f365 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1557b): Mirror 4: 0x5b54d55d (stale) Mirror 6: 0x5b54d55d (stale) Mirror 7: 0x5b54d55d (stale) Mirror 8: 0xdbe5f365 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1557b: skipped 000207 1787735083.220695 trunc from 0x01557b to 0x018d58 (0x37de bytes) 000209 1787735083.240749 mapwrite 0x00f3db thru 0x018fa7 (0x9bcd bytes) 000210 1787735083.286954 read 0x013048 thru 0x018fa7 (0x5f60 bytes) 000211 1787735083.287994 fallocate 0x01f10f thru 0x020972 (0x1864 bytes) 000213 1787735083.305165 mapread 0x008b47 thru 0x014f48 (0xc402 bytes) 000214 1787735083.320596 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18fa8): Mirror 4: 0xa3224555 (stale) Mirror 6: 0xa3224555 (stale) Mirror 7: 0xa3224555 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18fa8): Mirror 4: 0xa3224555 (stale) Mirror 6: 0xa3224555 (stale) Mirror 7: 0xa3224555 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18fa8: skipped 000215 1787735083.690994 trunc from 0x018fa8 to 0x03b040 (0x22099 bytes) 000216 1787735083.712151 trunc from 0x03a9d5 to 0x03b040 (0x66c bytes) 000217 1787735083.717021 mapwrite 0x036e55 thru 0x038a79 (0x1c25 bytes) 000218 1787735083.746771 trunc from 0x03b041 to 0x00d5db (0x2da65 bytes) 000219 1787735083.768365 MIRROR_RESYNC 4 mirrors 000221 1787735084.387876 write 0x03de5c thru 0x03ffff (0x21a4 bytes) 000222 1787735084.417173 mapwrite 0x033f93 thru 0x03e2ff (0xa36d bytes) 000223 1787735084.458601 mapread 0x01ea63 thru 0x02b44c (0xc9ea bytes) 000225 1787735084.475070 mapwrite 0x02c618 thru 0x039bb9 (0xd5a2 bytes) 000228 1787735084.493421 trunc from 0x040000 to 0x0055ba (0x3aa45 bytes) 000231 1787735084.516319 punch from 0x00537e to 0x0055ba (0x23d bytes) 000232 1787735084.521026 mapwrite 0x017815 thru 0x021995 (0xa181 bytes) 000234 1787735084.576323 MIRROR_RESYNC 4 mirrors 000235 1787735085.168570 mapwrite 0x023e20 thru 0x02d585 (0x9766 bytes) 000236 1787735085.247904 punch from 0x01ca30 to 0x02a354 (0xd925 bytes) 000237 1787735085.251010 read 0x010fa8 thru 0x011951 (0x9aa bytes) 000238 1787735085.254755 write 0x02ed58 thru 0x036ec4 (0x816d bytes) 000239 1787735085.256035 trunc from 0x01eb45 to 0x020999 (0x1e55 bytes) 000240 1787735085.261397 trunc from 0x0319b3 to 0x036ec4 (0x5512 bytes) 000241 1787735085.284974 punch from 0x007fab to 0x0169c5 (0xea1b bytes) 000242 1787735085.294054 write 0x0308b5 thru 0x039b86 (0x92d2 bytes) 000243 1787735085.303450 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39b87): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b87): Mirror 4: 0x84a10526 (stale) Mirror 6: 0x84a10526 (stale) Mirror 7: 0x84a10526 (stale) CRC-32 checksum value for chunk [0, 0x39b87): Mirror 1: 0x43bab88c Mirror 8: 0x43bab88c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b87: skipped 000244 1787735085.890143 read 0x015f18 thru 0x019229 (0x3312 bytes) 000245 1787735085.902294 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39b87): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b87): Mirror 4: 0x84a10526 (stale) Mirror 6: 0x84a10526 (stale) Mirror 7: 0x84a10526 (stale) CRC-32 checksum value for chunk [0, 0x39b87): Mirror 1: 0x43bab88c Mirror 8: 0x43bab88c Mirror 9: 0x43bab88c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b87: skipped 000246 1787735086.581910 punch from 0x0204b5 to 0x028da0 (0x88ec bytes) 000247 1787735086.617940 trunc from 0x01ad18 to 0x02754c (0xc835 bytes) 000249 1787735086.624337 fallocate 0x0171cb thru 0x01a03a (0x2e70 bytes) 000250 1787735086.629019 trunc from 0x008253 to 0x00a9f0 (0x279e bytes) 000251 1787735086.633565 mapwrite 0x015028 thru 0x023a79 (0xea52 bytes) 000252 1787735086.724101 read 0x0375c0 thru 0x0395f0 (0x2031 bytes) 000253 1787735086.731460 mapread 0x0381d7 thru 0x039b86 (0x19b0 bytes) 000254 1787735086.734047 mapwrite 0x022f3d thru 0x03007d (0xd141 bytes) 000255 1787735086.759024 mapread 0x035f34 thru 0x039b86 (0x3c53 bytes) 000256 1787735086.766908 punch from 0x00e3a4 to 0x01b0af (0xcd0c bytes) 000257 1787735086.772958 mapwrite 0x00acb5 thru 0x0120c7 (0x7413 bytes) 000258 1787735086.797996 mapread 0x00688a thru 0x00e058 (0x77cf bytes) 000259 1787735086.799420 fallocate 0x03f7a5 thru 0x03ffff (0x85b bytes) 000261 1787735086.804473 trunc from 0x039b87 to 0x015165 (0x24a21 bytes) 000262 1787735086.854365 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x15166): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15166): Mirror 4: 0x59ca392 (stale) Mirror 6: 0x59ca392 (stale) Mirror 7: 0x59ca392 (stale) Mirror 8: 0x59ca392 (stale) Mirror 9: 0x59ca392 (stale) CRC-32 checksum value for chunk [0, 0x15166): Mirror 1: 0x90f756a8 Mirror 10: 0x90f756a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15166: skipped 000263 1787735087.636045 punch from 0x00b2fc to 0x00c1b7 (0xebc bytes) 000264 1787735087.686446 write 0x0334a0 thru 0x03ffff (0xcb60 bytes) 000265 1787735087.698532 MIRROR_RESYNC 7 mirrors 000266 1787735089.006003 trunc from 0x01e388 to 0x01f318 (0xf91 bytes) 000267 1787735089.057501 trunc from 0x005a66 to 0x010b58 (0xb0f3 bytes) 000268 1787735089.061380 trunc from 0x0326d0 to 0x03ffff (0xd930 bytes) 000269 1787735089.064549 trunc from 0x040000 to 0x0293a2 (0x16c5d bytes) 000271 1787735089.084215 mapwrite 0x0087e3 thru 0x011463 (0x8c81 bytes) 000272 1787735089.119222 punch from 0x0077ca to 0x00b1c7 (0x39fe bytes) 000273 1787735089.126225 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x293a3): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x293a3): Mirror 4: 0x4a8115da (stale) Mirror 6: 0x4a8115da (stale) Mirror 7: 0x4a8115da (stale) Mirror 8: 0x4a8115da (stale) Mirror 9: 0x4a8115da (stale) Mirror 10: 0x4a8115da (stale) CRC-32 checksum value for chunk [0, 0x293a3): Mirror 1: 0x1f095e4a Mirror 11: 0x1f095e4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x293a3: skipped 000274 1787735089.823248 read 0x008ee4 thru 0x015b66 (0xcc83 bytes) 000275 1787735089.834466 write 0x01c605 thru 0x0258b6 (0x92b2 bytes) 000276 1787735089.884492 punch from 0x00a27e to 0x01728b (0xd00e bytes) 000277 1787735089.890063 trunc from 0x002d6f to 0x00a188 (0x741a bytes) 000278 1787735089.898383 fallocate 0x007f4a thru 0x017c1a (0xfcd1 bytes) 000279 1787735089.903867 read 0x0031fd thru 0x00dcfb (0xaaff bytes) 000280 1787735089.915151 mapread 0x0244a7 thru 0x0293a2 (0x4efc bytes) 000281 1787735089.917747 mapwrite 0x036073 thru 0x03ffff (0x9f8d bytes) 000282 1787735090.032360 fallocate 0x02eba6 thru 0x03d2ee (0xe749 bytes) 000283 1787735090.039798 mapwrite 0x0122be thru 0x0169fc (0x473f bytes) 000284 1787735090.060055 write 0x01d7bc thru 0x02730c (0x9b51 bytes) 000286 1787735090.063901 punch from 0x036d3d to 0x03f75a (0x8a1e bytes) 000287 1787735090.070342 punch from 0x02e3d0 to 0x0370b0 (0x8ce1 bytes) 000288 1787735090.074683 punch from 0x026b1b to 0x0316a3 (0xab89 bytes) 000289 1787735090.091653 write 0x019a24 thru 0x028071 (0xe64e bytes) 000290 1787735090.097495 punch from 0x0380b2 to 0x03ffff (0x7f4e bytes) 000291 1787735090.102510 write 0x02223a thru 0x031117 (0xeede bytes) 000292 1787735090.108854 mapwrite 0x014f2e thru 0x024ced (0xfdc0 bytes) 000293 1787735090.130312 fallocate 0x01d932 thru 0x028a3b (0xb10a bytes) 000294 1787735090.134998 trunc from 0x040000 to 0x0378f6 (0x8709 bytes) 000295 1787735090.145036 mapwrite 0x0102cb thru 0x0155fa (0x5330 bytes) 000296 1787735090.155017 punch from 0x02b98b to 0x02c290 (0x906 bytes) 000297 1787735090.158085 punch from 0x027104 to 0x03070e (0x960b bytes) 000298 1787735090.163307 mapwrite 0x03f3d4 thru 0x03ffff (0xc2c bytes) 000299 1787735090.200195 mapread 0x02dd41 thru 0x033345 (0x5605 bytes) 000300 1787735090.233039 trunc from 0x02444f to 0x0333f4 (0xefa6 bytes) 000301 1787735090.238252 fallocate 0x021729 thru 0x023da5 (0x267d bytes) 000303 1787735090.243831 mapread 0x012cb5 thru 0x014604 (0x1950 bytes) 000304 1787735090.244856 fallocate 0x03f627 thru 0x03fdab (0x785 bytes) 000305 1787735090.251076 mapread 0x0123ec thru 0x01465c (0x2271 bytes) 000306 1787735090.252125 trunc from 0x040000 to 0x00f2c8 (0x30d37 bytes) 000307 1787735090.276251 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf2c9): Mirror 4: 0x6a319f35 (stale) Mirror 6: 0x6a319f35 (stale) Mirror 8: 0x6a319f35 (stale) Mirror 9: 0x6a319f35 (stale) Mirror 10: 0x6a319f35 (stale) Mirror 11: 0x7d12e112 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf2c9): Mirror 4: 0x6a319f35 (stale) Mirror 6: 0x6a319f35 (stale) Mirror 8: 0x6a319f35 (stale) Mirror 9: 0x6a319f35 (stale) Mirror 10: 0x6a319f35 (stale) Mirror 11: 0x7d12e112 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf2c9: skipped 000308 1787735090.714479 read 0x00872e thru 0x00f2c8 (0x6b9b bytes) 000309 1787735090.732379 trunc from 0x00f2c9 to 0x02b4cd (0x1c205 bytes) 000310 1787735090.794536 write 0x01aefa thru 0x0255bd (0xa6c4 bytes) 000311 1787735090.802591 mapwrite 0x023341 thru 0x02f4c5 (0xc185 bytes) 000312 1787735090.868231 mapwrite 0x0193d8 thru 0x024850 (0xb479 bytes) 000314 1787735090.904304 write 0x00d6cc thru 0x00f68f (0x1fc4 bytes) 000315 1787735090.905569 mapread 0x01c20f thru 0x0277bd (0xb5af bytes) 000316 1787735090.907156 fallocate 0x0274d5 thru 0x030e8b (0x99b7 bytes) 000317 1787735090.911065 trunc from 0x00289b to 0x0061a6 (0x390c bytes) 000318 1787735090.914207 mapread 0x01cb51 thru 0x026f97 (0xa447 bytes) 000320 1787735090.915734 read 0x02b8e2 thru 0x030e8b (0x55aa bytes) 000321 1787735090.921504 trunc from 0x01168b to 0x014257 (0x2bcd bytes) 000322 1787735090.925163 trunc from 0x01adc6 to 0x02ad41 (0xff7c bytes) 000324 1787735090.931295 punch from 0x01fba9 to 0x02b27f (0xb6d7 bytes) 000325 1787735090.935512 trunc from 0x030e8c to 0x03570c (0x4881 bytes) 000327 1787735090.949212 trunc from 0x03570d to 0x00f0af (0x2665d bytes) 000328 1787735090.967856 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf0b0): Mirror 6: 0xa751ef4c (stale) Mirror 8: 0xa751ef4c (stale) Mirror 9: 0xa751ef4c (stale) Mirror 10: 0xa751ef4c (stale) Mirror 11: 0xaf0c55c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf0b0): Mirror 6: 0xa751ef4c (stale) Mirror 8: 0xa751ef4c (stale) Mirror 9: 0xa751ef4c (stale) Mirror 10: 0xa751ef4c (stale) Mirror 11: 0xaf0c55c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf0b0: skipped 000329 1787735091.348429 mapwrite 0x01b888 thru 0x024332 (0x8aab bytes) 000330 1787735091.444946 MIRROR_RESYNC 6 mirrors 000331 1787735092.581996 mapread 0x00f1f2 thru 0x00fc8f (0xa9e bytes) 000332 1787735092.588738 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) 000333 1787735092.867886 mapwrite 0x01a924 thru 0x02a6dd (0xfdba bytes) 000334 1787735092.954782 mapread 0x004496 thru 0x007653 (0x31be bytes) 000335 1787735092.965252 MIRROR_RESYNC 5 mirrors 000336 1787735093.792355 write 0x01356b thru 0x01a68c (0x7122 bytes) 000337 1787735093.817660 mapwrite 0x00bf63 thru 0x013592 (0x7630 bytes) 000338 1787735093.844702 punch from 0x019567 to 0x01d922 (0x43bc bytes) 000339 1787735093.848523 trunc from 0x0224c4 to 0x023325 (0xe62 bytes) 000340 1787735093.851416 read 0x011f5d thru 0x013896 (0x193a bytes) 000341 1787735093.851785 write 0x01fde3 thru 0x02dd21 (0xdf3f bytes) 000342 1787735093.857075 write 0x034d3c thru 0x036742 (0x1a07 bytes) 000343 1787735093.857717 punch from 0x028d92 to 0x032d0d (0x9f7c bytes) 000344 1787735093.870647 punch from 0x01090a to 0x01c447 (0xbb3e bytes) 000345 1787735093.875188 trunc from 0x01a0a5 to 0x02178f (0x76eb bytes) 000347 1787735093.878777 mapread 0x02dc8f thru 0x0315ee (0x3960 bytes) 000348 1787735093.885992 fallocate 0x039cae thru 0x03ffff (0x6352 bytes) 000350 1787735093.890620 trunc from 0x02ce1d to 0x036742 (0x9926 bytes) 000351 1787735093.894777 trunc from 0x026b0f to 0x0291e9 (0x26db bytes) 000352 1787735093.898799 trunc from 0x036743 to 0x02c25b (0xa4e7 bytes) 000353 1787735093.919202 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2c25c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c25c): Mirror 6: 0xb5d38987 (stale) Mirror 8: 0xb5d38987 (stale) Mirror 9: 0xb5d38987 (stale) Mirror 10: 0xb5d38987 (stale) CRC-32 checksum value for chunk [0, 0x2c25c): Mirror 1: 0x4f7df554 Mirror 11: 0x4f7df554 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c25c: skipped 000354 1787735094.600021 write 0x008b19 thru 0x016b01 (0xdfe9 bytes) 000355 1787735094.645455 punch from 0x0100db to 0x01c856 (0xc77c bytes) 000357 1787735094.678015 write 0x0374ff thru 0x03fef2 (0x89f4 bytes) 000358 1787735094.681693 trunc from 0x002f43 to 0x00fdac (0xce6a bytes) 000359 1787735094.691752 fallocate 0x018de0 thru 0x01f906 (0x6b27 bytes) 000360 1787735094.703910 write 0x030c12 thru 0x031a95 (0xe84 bytes) 000361 1787735094.743212 read 0x012431 thru 0x01354c (0x111c bytes) 000363 1787735094.749950 trunc from 0x03fef3 to 0x03057f (0xf973 bytes) 000364 1787735094.792710 punch from 0x02c9b1 to 0x03057f (0x3bcf bytes) 000366 1787735094.802762 read 0x0251f0 thru 0x02c334 (0x7145 bytes) 000367 1787735094.816628 read 0x01cfab thru 0x028cf3 (0xbd49 bytes) 000368 1787735094.817108 trunc from 0x030234 to 0x03057f (0x34c bytes) 000369 1787735094.823486 trunc from 0x030580 to 0x0051d8 (0x2b3a7 bytes) 000370 1787735094.842633 fallocate 0x007713 thru 0x007906 (0x1f4 bytes) 000371 1787735094.848459 fallocate 0x0166c9 thru 0x0225e9 (0xbf21 bytes) 000372 1787735094.853467 mapread 0x002cd3 thru 0x007906 (0x4c34 bytes) 000373 1787735094.860382 trunc from 0x007907 to 0x00e325 (0x6a1f bytes) 000374 1787735094.873378 punch from 0x0086d2 to 0x00e325 (0x5c54 bytes) 000375 1787735094.878919 mapwrite 0x02e830 thru 0x031eed (0x36be bytes) 000377 1787735094.918279 write 0x00abd1 thru 0x013e17 (0x9247 bytes) 000378 1787735094.925213 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31eee): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31eee): Mirror 6: 0xad243036 (stale) Mirror 8: 0xad243036 (stale) Mirror 9: 0xad243036 (stale) Mirror 10: 0xad243036 (stale) Mirror 11: 0x786fec8d (stale) CRC-32 checksum value for chunk [0, 0x31eee): Mirror 1: 0x7361b786 Mirror 12: 0x7361b786 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31eee: skipped 000379 1787735095.699255 punch from 0x022ae4 to 0x02f8d5 (0xcdf2 bytes) 000380 1787735095.736129 trunc from 0x031eee to 0x033f57 (0x206a bytes) 000381 1787735095.751479 mapread 0x0021e4 thru 0x00aed7 (0x8cf4 bytes) 000382 1787735095.762496 write 0x003fa3 thru 0x00fb07 (0xbb65 bytes) 000383 1787735095.763917 trunc from 0x033332 to 0x033f57 (0xc26 bytes) 000384 1787735095.769922 read 0x01da1b thru 0x025646 (0x7c2c bytes) 000385 1787735095.778030 fallocate 0x02638f thru 0x02c327 (0x5f99 bytes) 000388 1787735095.785166 fallocate 0x02e4bc thru 0x03117d (0x2cc2 bytes) 000389 1787735095.790346 mapread 0x02e325 thru 0x02f073 (0xd4f bytes) 000390 1787735095.800096 mapwrite 0x009ea4 thru 0x011b12 (0x7c6f bytes) 000391 1787735095.830261 trunc from 0x0144c5 to 0x02418b (0xfcc7 bytes) 000392 1787735095.834171 write 0x03ccb9 thru 0x03ffff (0x3347 bytes) 000393 1787735095.835567 read 0x0050c6 thru 0x0085d4 (0x350f bytes) 000394 1787735095.836045 trunc from 0x040000 to 0x0090c8 (0x36f37 bytes) 000395 1787735095.845718 trunc from 0x0090c9 to 0x03cedc (0x33e14 bytes) 000396 1787735095.858276 trunc from 0x03cedd to 0x02fc71 (0xd26b bytes) 000397 1787735095.875744 trunc from 0x02fc72 to 0x00dbb0 (0x220c1 bytes) 000398 1787735095.896543 write 0x009fa1 thru 0x0132bc (0x931c bytes) 000399 1787735095.899222 trunc from 0x00bfc9 to 0x0132bc (0x72f4 bytes) 000400 1787735095.910507 fallocate 0x008ffb thru 0x00a871 (0x1877 bytes) 000401 1787735095.915276 punch from 0x00bd95 to 0x00dec9 (0x2135 bytes) 000402 1787735095.920252 mapwrite 0x03ae70 thru 0x03fac3 (0x4c54 bytes) 000403 1787735095.975158 punch from 0x03e41f to 0x03fac3 (0x16a5 bytes) 000405 1787735095.980769 mapread 0x02b0bc thru 0x02eadc (0x3a21 bytes) 000406 1787735095.981775 punch from 0x009f9b to 0x017a72 (0xdad8 bytes) 000407 1787735095.990505 MIRROR_RESYNC 7 mirrors 000408 1787735097.176844 mapread 0x025061 thru 0x031b93 (0xcb33 bytes) 000409 1787735097.200622 read 0x0313ae thru 0x0373ed (0x6040 bytes) 000410 1787735097.201101 punch from 0x03ab96 to 0x03fac3 (0x4f2e bytes) 000411 1787735097.239436 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fac4): Mirror 6: 0x1244e6e6 (stale) Mirror 8: 0x1244e6e6 (stale) Mirror 9: 0x1244e6e6 (stale) Mirror 11: 0x1244e6e6 (stale) Mirror 12: 0x1244e6e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fac4): Mirror 6: 0x1244e6e6 (stale) Mirror 8: 0x1244e6e6 (stale) Mirror 9: 0x1244e6e6 (stale) Mirror 11: 0x1244e6e6 (stale) Mirror 12: 0x1244e6e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fac4: skipped 000413 1787735097.609739 punch from 0x005dd6 to 0x008116 (0x2341 bytes) 000414 1787735097.617449 mapread 0x0331cd thru 0x03fac3 (0xc8f7 bytes) 000416 1787735097.641478 trunc from 0x001345 to 0x008f47 (0x7c03 bytes) 000418 1787735097.646897 fallocate 0x016c45 thru 0x017f82 (0x133e bytes) 000419 1787735097.651289 write 0x037f34 thru 0x03e30f (0x63dc bytes) 000420 1787735097.660443 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fac4): Mirror 6: 0x1244e6e6 (stale) Mirror 8: 0x1244e6e6 (stale) Mirror 11: 0x1244e6e6 (stale) Mirror 12: 0x1244e6e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fac4): Mirror 6: 0x1244e6e6 (stale) Mirror 8: 0x1244e6e6 (stale) Mirror 11: 0x1244e6e6 (stale) Mirror 12: 0x1244e6e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fac4: skipped 000421 1787735098.023712 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3fac4): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fac4): Mirror 6: 0x1244e6e6 (stale) Mirror 8: 0x1244e6e6 (stale) Mirror 11: 0x1244e6e6 (stale) Mirror 12: 0x1244e6e6 (stale) CRC-32 checksum value for chunk [0, 0x3fac4): Mirror 1: 0xaa8a53e7 Mirror 13: 0xaa8a53e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fac4: skipped 000423 1787735098.793092 fallocate 0x018229 thru 0x01f9c2 (0x779a bytes) 000424 1787735098.830686 write 0x004b12 thru 0x013653 (0xeb42 bytes) 000425 1787735098.841438 write 0x00750b thru 0x009d7e (0x2874 bytes) 000427 1787735098.842961 fallocate 0x01a3da thru 0x022ca1 (0x88c8 bytes) 000428 1787735098.852411 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fac4): Mirror 8: 0x1244e6e6 (stale) Mirror 11: 0x1244e6e6 (stale) Mirror 12: 0x1244e6e6 (stale) Mirror 13: 0xaa8a53e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fac4): Mirror 8: 0x1244e6e6 (stale) Mirror 11: 0x1244e6e6 (stale) Mirror 12: 0x1244e6e6 (stale) Mirror 13: 0xaa8a53e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fac4: skipped 000429 1787735099.141466 trunc from 0x005ed2 to 0x00f3e4 (0x9513 bytes) 000430 1787735099.147163 trunc from 0x004ad4 to 0x005044 (0x571 bytes) 000431 1787735099.159628 MIRROR_RESYNC 5 mirrors 000432 1787735100.389703 trunc from 0x0151d6 to 0x0158f0 (0x71b bytes) 000433 1787735100.417413 mapwrite 0x032e8f thru 0x03e525 (0xb697 bytes) 000434 1787735100.467755 read 0x009f20 thru 0x00b721 (0x1802 bytes) 000435 1787735100.474351 mapwrite 0x01808c thru 0x024ebc (0xce31 bytes) 000436 1787735100.512099 mapwrite 0x029755 thru 0x029ee7 (0x793 bytes) 000437 1787735100.522736 trunc from 0x022810 to 0x0228fb (0xec bytes) 000438 1787735100.527484 trunc from 0x0247e2 to 0x031879 (0xd098 bytes) 000439 1787735100.533488 fallocate 0x02feae thru 0x03c3dc (0xc52f bytes) 000440 1787735100.538060 mapread 0x01fa79 thru 0x0222b9 (0x2841 bytes) 000441 1787735100.547467 punch from 0x03a177 to 0x03fac3 (0x594d bytes) 000442 1787735100.555601 read 0x004a9a thru 0x00977f (0x4ce6 bytes) 000443 1787735100.564358 punch from 0x036f7d to 0x03d336 (0x63ba bytes) 000444 1787735100.568842 mapread 0x02a23a thru 0x02f1bd (0x4f84 bytes) 000445 1787735100.570638 trunc from 0x01a92b to 0x0248bd (0x9f93 bytes) 000446 1787735100.576425 trunc from 0x03fac4 to 0x00e4b8 (0x3160b bytes) 000447 1787735100.604998 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xe4b9): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe4b9): Mirror 8: 0x52301f3b (stale) Mirror 11: 0x52301f3b (stale) Mirror 12: 0x52301f3b (stale) Mirror 13: 0x52301f3b (stale) CRC-32 checksum value for chunk [0, 0xe4b9): Mirror 1: 0x52301f3b Mirror 14: 0x52301f3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe4b9: skipped 000448 1787735101.254282 fallocate 0x027405 thru 0x02f908 (0x8504 bytes) 000449 1787735101.345604 mapwrite 0x00ff2a thru 0x011057 (0x112e bytes) 000450 1787735101.415544 mapwrite 0x029c3a thru 0x02e0e3 (0x44aa bytes) 000451 1787735101.442213 fallocate 0x00195a thru 0x003069 (0x1710 bytes) 000452 1787735101.447499 punch from 0x0002cf to 0x00555b (0x528d bytes) 000453 1787735101.459268 trunc from 0x02cf3a to 0x02f908 (0x29cf bytes) 000454 1787735101.469798 trunc from 0x0036d2 to 0x00c350 (0x8c7f bytes) 000455 1787735101.484501 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f909): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f909): Mirror 8: 0xddf5d974 (stale) Mirror 11: 0xddf5d974 (stale) Mirror 12: 0xddf5d974 (stale) Mirror 13: 0xddf5d974 (stale) Mirror 14: 0xf79b4e16 (stale) CRC-32 checksum value for chunk [0, 0x2f909): Mirror 1: 0x89c755e0 Mirror 15: 0x89c755e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f909: skipped 000456 1787735102.145013 mapwrite 0x03fec4 thru 0x03ffff (0x13c bytes) 000457 1787735102.222084 MIRROR_RESYNC 7 mirrors 000458 1787735103.762273 punch from 0x03be6f to 0x03ffff (0x4191 bytes) 000459 1787735103.791411 mapread 0x025ec2 thru 0x029da5 (0x3ee4 bytes) 000460 1787735103.810952 mapwrite 0x01b89b thru 0x028d3a (0xd4a0 bytes) 000461 1787735103.838959 read 0x01aa01 thru 0x01dc76 (0x3276 bytes) 000462 1787735103.839549 trunc from 0x020b2b to 0x02b37f (0xa855 bytes) 000463 1787735103.844283 fallocate 0x005502 thru 0x0066d7 (0x11d6 bytes) 000464 1787735103.848532 write 0x01ed94 thru 0x024072 (0x52df bytes) 000465 1787735103.853456 read 0x007afe thru 0x00c82d (0x4d30 bytes) 000466 1787735103.861606 punch from 0x00bfba to 0x01bdb6 (0xfdfd bytes) 000467 1787735103.866475 mapwrite 0x01bb45 thru 0x02b848 (0xfd04 bytes) 000469 1787735103.899568 write 0x024ebc thru 0x033702 (0xe847 bytes) 000471 1787735103.902656 mapread 0x03dfd7 thru 0x03e70b (0x735 bytes) 000472 1787735103.903596 trunc from 0x040000 to 0x02058a (0x1fa75 bytes) 000473 1787735103.931525 mapwrite 0x02ec8b thru 0x0351c1 (0x6537 bytes) 000474 1787735104.001739 punch from 0x0113e8 to 0x0115fc (0x215 bytes) 000475 1787735104.006007 trunc from 0x0351c2 to 0x003105 (0x320bc bytes) 000477 1787735104.021892 trunc from 0x001325 to 0x003105 (0x1de1 bytes) 000478 1787735104.025629 write 0x02e16a thru 0x02e4f4 (0x38b bytes) 000480 1787735104.027167 mapread 0x02deba thru 0x02e4f4 (0x63b bytes) 000481 1787735104.031397 write 0x002af2 thru 0x00eec9 (0xc3d8 bytes) 000484 1787735104.042613 trunc from 0x02e4f5 to 0x01a101 (0x143f3 bytes) 000485 1787735104.060416 punch from 0x002c80 to 0x008408 (0x5789 bytes) 000486 1787735104.073728 mapread 0x01267a thru 0x01a101 (0x7a88 bytes) 000487 1787735104.110972 trunc from 0x01a102 to 0x00f945 (0xa7bc bytes) 000488 1787735104.140149 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf946): Mirror 8: 0x4f07862 (stale) Mirror 12: 0x4f07862 (stale) Mirror 13: 0x4f07862 (stale) Mirror 14: 0x4f07862 (stale) Mirror 15: 0x4f07862 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf946): Mirror 8: 0x4f07862 (stale) Mirror 12: 0x4f07862 (stale) Mirror 13: 0x4f07862 (stale) Mirror 14: 0x4f07862 (stale) Mirror 15: 0x4f07862 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf946: skipped 000490 1787735104.439450 mapread 0x00f323 thru 0x00f4f2 (0x1d0 bytes) 000491 1787735104.447468 read 0x00f2ce thru 0x00f945 (0x678 bytes) 000492 1787735104.448265 read 0x006e6a thru 0x0080ba (0x1251 bytes) 000493 1787735104.456596 trunc from 0x00f946 to 0x023dfd (0x144b8 bytes) 000494 1787735104.473338 mapwrite 0x02b535 thru 0x0360f5 (0xabc1 bytes) 000496 1787735104.513650 trunc from 0x017c94 to 0x0226bc (0xaa29 bytes) 000497 1787735104.517132 trunc from 0x008db3 to 0x009788 (0x9d6 bytes) 000498 1787735104.523067 write 0x009a38 thru 0x009ae3 (0xac bytes) 000499 1787735104.527779 write 0x01bade thru 0x02692f (0xae52 bytes) 000500 1787735104.533353 mapread 0x00febb thru 0x01c816 (0xc95c bytes) 000501 1787735104.550004 fallocate 0x004405 thru 0x0099ac (0x55a8 bytes) 000502 1787735104.554626 punch from 0x02da8e to 0x0331a5 (0x5718 bytes) 000503 1787735104.560412 mapread 0x00030a thru 0x008536 (0x822d bytes) 000504 1787735104.567269 read 0x01e124 thru 0x02d865 (0xf742 bytes) 000505 1787735104.577381 trunc from 0x00b812 to 0x0152b6 (0x9aa5 bytes) 000506 1787735104.582528 mapread 0x019162 thru 0x01c9d1 (0x3870 bytes) 000507 1787735104.583524 fallocate 0x0033aa thru 0x00689f (0x34f6 bytes) 000508 1787735104.590042 fallocate 0x013745 thru 0x01cce6 (0x95a2 bytes) 000509 1787735104.595089 write 0x01c220 thru 0x0257c8 (0x95a9 bytes) 000510 1787735104.597243 trunc from 0x011680 to 0x01f6a0 (0xe021 bytes) 000512 1787735104.611748 punch from 0x0230e0 to 0x0275dc (0x44fd bytes) 000513 1787735104.617245 trunc from 0x0360f6 to 0x00e5c2 (0x27b33 bytes) 000514 1787735104.631964 mapwrite 0x024fa8 thru 0x02dbd3 (0x8c2c bytes) 000515 1787735104.718575 trunc from 0x02a8ef to 0x02dbd3 (0x32e5 bytes) 000516 1787735104.725618 read 0x019f38 thru 0x01a0c3 (0x18c bytes) 000517 1787735104.740090 fallocate 0x02673e thru 0x02e841 (0x8104 bytes) 000518 1787735104.744984 write 0x00ed4f thru 0x017664 (0x8916 bytes) 000520 1787735104.747735 write 0x00769d thru 0x00c525 (0x4e89 bytes) 000522 1787735104.749841 trunc from 0x014cd1 to 0x024735 (0xfa65 bytes) 000523 1787735104.765951 trunc from 0x02dbd4 to 0x003924 (0x2a2af bytes) 000524 1787735104.786689 trunc from 0x000e63 to 0x003924 (0x2ac2 bytes) 000525 1787735104.792254 punch from 0x0024ab to 0x003924 (0x147a bytes) 000526 1787735104.800570 trunc from 0x0017bf to 0x003924 (0x2166 bytes) 000528 1787735104.805311 read 0x000608 thru 0x003924 (0x331d bytes) 000529 1787735104.811327 fallocate 0x03e91c thru 0x03ffff (0x16e4 bytes) 000530 1787735104.816061 mapwrite 0x018a6d thru 0x0192f0 (0x884 bytes) 000531 1787735104.870436 mapread 0x0038fa thru 0x004791 (0xe98 bytes) 000532 1787735104.871361 trunc from 0x007bfd to 0x00af2a (0x332e bytes) 000533 1787735104.876530 punch from 0x00b228 to 0x00ed87 (0x3b60 bytes) 000534 1787735104.881460 mapread 0x00bf09 thru 0x00cb21 (0xc19 bytes) 000535 1787735104.893188 trunc from 0x0192f1 to 0x026a58 (0xd768 bytes) 000536 1787735104.917367 trunc from 0x007870 to 0x00af40 (0x36d1 bytes) 000537 1787735104.922099 mapwrite 0x038049 thru 0x03ffff (0x7fb7 bytes) 000538 1787735104.958503 fallocate 0x01271e thru 0x017b24 (0x5407 bytes) 000540 1787735104.964220 write 0x01def0 thru 0x029f12 (0xc023 bytes) 000541 1787735104.966019 write 0x0321ef thru 0x036eda (0x4cec bytes) 000543 1787735104.968103 mapwrite 0x03c632 thru 0x03ed4c (0x271b bytes) 000544 1787735104.982732 trunc from 0x034b23 to 0x03ffff (0xb4dd bytes) 000546 1787735104.988782 read 0x0299c9 thru 0x02dd58 (0x4390 bytes) 000547 1787735104.989592 fallocate 0x035603 thru 0x039b84 (0x4582 bytes) 000548 1787735104.994061 trunc from 0x040000 to 0x0136ca (0x2c935 bytes) 000549 1787735105.014615 mapread 0x0110e1 thru 0x0136ca (0x25ea bytes) 000550 1787735105.020701 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x136cb): Mirror 8: 0x64b07d84 (stale) Mirror 12: 0x64b07d84 (stale) Mirror 13: 0x64b07d84 (stale) Mirror 15: 0x64b07d84 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x136cb): Mirror 8: 0x64b07d84 (stale) Mirror 12: 0x64b07d84 (stale) Mirror 13: 0x64b07d84 (stale) Mirror 15: 0x64b07d84 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x136cb: skipped 000551 1787735105.452264 fallocate 0x0157c7 thru 0x01ad77 (0x55b1 bytes) 000552 1787735105.461387 MIRROR_RESYNC 5 mirrors 000553 1787735106.153528 write 0x016ec2 thru 0x022eee (0xc02d bytes) 000554 1787735106.181655 fallocate 0x013d9b thru 0x0198e2 (0x5b48 bytes) 000555 1787735106.186138 trunc from 0x022eef to 0x01cce4 (0x620a bytes) 000556 1787735106.208402 write 0x0091bf thru 0x0144bc (0xb2fe bytes) 000557 1787735106.217336 punch from 0x016fdc to 0x018f44 (0x1f69 bytes) 000558 1787735106.228282 trunc from 0x00b8d1 to 0x01733b (0xba6b bytes) 000560 1787735106.233650 mapwrite 0x02199f thru 0x027d3e (0x63a0 bytes) 000561 1787735106.269787 mapread 0x0217f8 thru 0x027d3e (0x6547 bytes) 000562 1787735106.270426 read 0x00c465 thru 0x019028 (0xcbc4 bytes) 000563 1787735106.290485 write 0x00c0e4 thru 0x0196fc (0xd619 bytes) 000564 1787735106.291725 write 0x00009c thru 0x0032d3 (0x3238 bytes) 000565 1787735106.303685 fallocate 0x01f1ca thru 0x025bab (0x69e2 bytes) 000566 1787735106.307789 fallocate 0x0115d3 thru 0x0138a3 (0x22d1 bytes) 000567 1787735106.315333 trunc from 0x027d3f to 0x001a57 (0x262e7 bytes) 000568 1787735106.335521 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1a58): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a58): Mirror 8: 0x5ec19d8f (stale) Mirror 12: 0x5ec19d8f (stale) Mirror 13: 0x5ec19d8f (stale) Mirror 15: 0x5ec19d8f (stale) CRC-32 checksum value for chunk [0, 0x1a58): Mirror 1: 0xb5b9524e Mirror 16: 0xb5b9524e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a58: skipped 000569 1787735106.879002 read 0x00086d thru 0x0014f5 (0xc89 bytes) 000570 1787735106.889613 fallocate 0x024bfc thru 0x026731 (0x1b36 bytes) 000571 1787735106.944311 mapread 0x0017df thru 0x001a57 (0x279 bytes) 000572 1787735106.949399 write 0x032712 thru 0x03ffff (0xd8ee bytes) 000573 1787735106.963993 trunc from 0x030d06 to 0x034428 (0x3723 bytes) 000574 1787735106.999964 mapread 0x03fd79 thru 0x03ffff (0x287 bytes) 000575 1787735107.001586 punch from 0x016f70 to 0x0264c0 (0xf551 bytes) 000576 1787735107.009471 fallocate 0x03b505 thru 0x03ffff (0x4afb bytes) 000577 1787735107.017741 trunc from 0x040000 to 0x01d985 (0x2267a bytes) 000578 1787735107.069750 trunc from 0x003bc5 to 0x012053 (0xe48f bytes) 000579 1787735107.073039 mapread 0x01c0b8 thru 0x01d985 (0x18ce bytes) 000580 1787735107.090487 read 0x01d94f thru 0x01d985 (0x37 bytes) 000581 1787735107.090980 trunc from 0x01d986 to 0x011154 (0xc831 bytes) 000583 1787735107.106954 mapread 0x003478 thru 0x00d177 (0x9d00 bytes) 000584 1787735107.121048 mapwrite 0x036691 thru 0x03f90b (0x927b bytes) 000585 1787735107.160700 read 0x01dac2 thru 0x0257e9 (0x7d28 bytes) 000586 1787735107.161460 trunc from 0x03f90c to 0x03c8fb (0x3010 bytes) 000587 1787735107.176698 trunc from 0x023ba6 to 0x025016 (0x1471 bytes) 000588 1787735107.181214 mapwrite 0x0223a6 thru 0x026bdf (0x483a bytes) 000589 1787735107.200542 trunc from 0x009782 to 0x018028 (0xe8a7 bytes) 000590 1787735107.205576 mapread 0x021c74 thru 0x02c16c (0xa4f9 bytes) 000592 1787735107.209937 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c8fc): Mirror 8: 0x70ddad1d (stale) Mirror 12: 0x70ddad1d (stale) Mirror 13: 0x70ddad1d (stale) Mirror 16: 0xf08ff45f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c8fc): Mirror 8: 0x70ddad1d (stale) Mirror 12: 0x70ddad1d (stale) Mirror 13: 0x70ddad1d (stale) Mirror 16: 0xf08ff45f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c8fc: skipped 000593 1787735107.511266 write 0x036405 thru 0x03c4fa (0x60f6 bytes) 000594 1787735107.530540 punch from 0x03bd08 to 0x03c8fb (0xbf4 bytes) 000596 1787735107.543036 punch from 0x018490 to 0x021bbf (0x9730 bytes) 000597 1787735107.548924 mapwrite 0x013443 thru 0x014265 (0xe23 bytes) 000598 1787735107.567670 fallocate 0x00342d thru 0x006c2a (0x37fe bytes) 000600 1787735107.572466 punch from 0x01e641 to 0x02471e (0x60de bytes) 000601 1787735107.580071 trunc from 0x03c8fc to 0x025c9c (0x16c5f bytes) 000602 1787735107.598942 mapread 0x01b275 thru 0x01d32c (0x20b8 bytes) 000603 1787735107.612444 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x25c9d): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c9d): Mirror 8: 0x14165925 (stale) Mirror 12: 0x14165925 (stale) Mirror 13: 0x14165925 (stale) Mirror 16: 0x6e552013 (stale) CRC-32 checksum value for chunk [0, 0x25c9d): Mirror 1: 0xbd1f506b Mirror 17: 0xbd1f506b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c9d: skipped 000604 1787735108.186122 trunc from 0x005d7b to 0x006a50 (0xcd6 bytes) 000606 1787735108.235225 mapwrite 0x034748 thru 0x03ffff (0xb8b8 bytes) 000607 1787735108.429092 mapwrite 0x00a8de thru 0x00bbeb (0x130e bytes) 000608 1787735108.473396 trunc from 0x033e8b to 0x03841f (0x4595 bytes) 000609 1787735108.478658 punch from 0x025fe1 to 0x033ec9 (0xdee9 bytes) 000610 1787735108.483204 write 0x01233f thru 0x020d22 (0xe9e4 bytes) 000611 1787735108.489386 write 0x0203e7 thru 0x0215a8 (0x11c2 bytes) 000612 1787735108.500023 fallocate 0x01e9b0 thru 0x01fb9b (0x11ec bytes) 000613 1787735108.504323 write 0x032d1f thru 0x03e9da (0xbcbc bytes) 000614 1787735108.509650 mapread 0x01850c thru 0x020c1d (0x8712 bytes) 000615 1787735108.511587 trunc from 0x040000 to 0x02ee29 (0x111d6 bytes) 000616 1787735108.527640 fallocate 0x025252 thru 0x030063 (0xae12 bytes) 000617 1787735108.530932 trunc from 0x0053b3 to 0x00a0c4 (0x4d12 bytes) 000618 1787735108.535467 read 0x01ea4f thru 0x026e3b (0x83ed bytes) 000619 1787735108.548440 punch from 0x00a637 to 0x018030 (0xd9fa bytes) 000620 1787735108.566255 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ee2a): Mirror 8: 0xb51c821 (stale) Mirror 13: 0xb51c821 (stale) Mirror 16: 0x65299293 (stale) Mirror 17: 0x8afb4ba6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee2a): Mirror 8: 0xb51c821 (stale) Mirror 13: 0xb51c821 (stale) Mirror 16: 0x65299293 (stale) Mirror 17: 0x8afb4ba6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee2a: skipped 000621 1787735108.850796 mapread 0x006beb thru 0x008b28 (0x1f3e bytes) 000622 1787735108.869380 mapwrite 0x034268 thru 0x03bd22 (0x7abb bytes) 000623 1787735108.930440 mapread 0x032813 thru 0x03bd22 (0x9510 bytes) 000624 1787735108.933254 punch from 0x020b50 to 0x028900 (0x7db1 bytes) 000626 1787735108.947451 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd23): Mirror 8: 0xa793d640 (stale) Mirror 16: 0xf5ff28cf (stale) Mirror 17: 0x16a6ddb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd23): Mirror 8: 0xa793d640 (stale) Mirror 16: 0xf5ff28cf (stale) Mirror 17: 0x16a6ddb0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd23: skipped 000627 1787735109.314741 mapwrite 0x024117 thru 0x02a1e6 (0x60d0 bytes) 000628 1787735109.337275 punch from 0x0174a5 to 0x0238af (0xc40b bytes) 000629 1787735109.340040 punch from 0x022bfe to 0x02b54b (0x894e bytes) 000630 1787735109.346762 read 0x011eb6 thru 0x021743 (0xf88e bytes) 000631 1787735109.359268 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd23): Mirror 8: 0xa793d640 (stale) Mirror 17: 0x16a6ddb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd23): Mirror 8: 0xa793d640 (stale) Mirror 17: 0x16a6ddb0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd23: skipped 000632 1787735109.612556 punch from 0x033254 to 0x03bd22 (0x8acf bytes) 000633 1787735109.617547 trunc from 0x00325f to 0x00f956 (0xc6f8 bytes) 000635 1787735109.621553 punch from 0x000d4d to 0x0072c7 (0x657b bytes) 000637 1787735109.626528 punch from 0x0196c8 to 0x01fe70 (0x67a9 bytes) 000638 1787735109.635303 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd23): Mirror 17: 0x16a6ddb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd23): Mirror 17: 0x16a6ddb0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd23: skipped 000639 1787735109.929685 write 0x002e22 thru 0x006e90 (0x406f bytes) 000640 1787735109.953207 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) 000641 1787735110.082033 trunc from 0x03bd23 to 0x0216f8 (0x1a62a bytes) 000642 1787735110.120254 punch from 0x009f2a to 0x0175b0 (0xd687 bytes) 000643 1787735110.127779 write 0x032702 thru 0x03ffff (0xd8fe bytes) 000645 1787735110.132468 fallocate 0x0215fe thru 0x0302a7 (0xecaa bytes) 000647 1787735110.141214 mapwrite 0x02f5d2 thru 0x036922 (0x7351 bytes) 000648 1787735110.184459 write 0x019a27 thru 0x027359 (0xd933 bytes) 000649 1787735110.200530 trunc from 0x040000 to 0x020d66 (0x1f299 bytes) 000650 1787735110.220229 trunc from 0x00c7fd to 0x012c31 (0x6435 bytes) 000652 1787735110.227282 fallocate 0x039779 thru 0x03ffff (0x6887 bytes) 000653 1787735110.234040 write 0x03a5da thru 0x03ffff (0x5a26 bytes) 000655 1787735110.238542 read 0x026cea thru 0x026e5f (0x176 bytes) 000656 1787735110.244676 punch from 0x0250ff to 0x0329ab (0xd8ad bytes) 000657 1787735110.249875 read 0x012436 thru 0x01a8b9 (0x8484 bytes) 000658 1787735110.263528 write 0x001e00 thru 0x002d57 (0xf58 bytes) 000659 1787735110.265712 mapwrite 0x022e23 thru 0x02e36c (0xb54a bytes) 000661 1787735110.296204 mapwrite 0x015e38 thru 0x017ffc (0x21c5 bytes) 000662 1787735110.321760 read 0x039fe6 thru 0x03e5d1 (0x45ec bytes) 000663 1787735110.322366 write 0x029af5 thru 0x037076 (0xd582 bytes) 000664 1787735110.323782 punch from 0x00080e to 0x00f39e (0xeb91 bytes) 000666 1787735110.335685 mapread 0x023765 thru 0x025a42 (0x22de bytes) 000668 1787735110.337634 write 0x014433 thru 0x014788 (0x356 bytes) 000669 1787735110.339907 mapread 0x023104 thru 0x02d3ef (0xa2ec bytes) 000670 1787735110.341482 write 0x029906 thru 0x02c474 (0x2b6f bytes) 000671 1787735110.349694 MIRROR_EXTEND to 2 mirrors 000672 1787735111.012093 mapread 0x00a195 thru 0x00e229 (0x4095 bytes) 000673 1787735111.033007 trunc from 0x0022f7 to 0x00c9f1 (0xa6fb bytes) 000674 1787735111.071236 trunc from 0x031335 to 0x037b0b (0x67d7 bytes) 000675 1787735111.077657 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) 000677 1787735111.185509 trunc from 0x01173d to 0x013b31 (0x23f5 bytes) 000678 1787735111.192297 trunc from 0x008fb1 to 0x0146e7 (0xb737 bytes) 000680 1787735111.205583 fallocate 0x03dfd9 thru 0x03eb44 (0xb6c bytes) 000681 1787735111.210340 write 0x03446e thru 0x03ffff (0xbb92 bytes) 000682 1787735111.225253 read 0x0320ac thru 0x037798 (0x56ed bytes) 000683 1787735111.231800 read 0x00b0a0 thru 0x01613f (0xb0a0 bytes) 000685 1787735111.248529 MIRROR_EXTEND to 2 mirrors 000686 1787735111.812628 punch from 0x00346b to 0x003b64 (0x6fa bytes) 000687 1787735111.836308 trunc from 0x03815f to 0x03ffff (0x7ea1 bytes) 000688 1787735111.841487 read 0x02fd93 thru 0x03668f (0x68fd bytes) 000689 1787735111.851975 write 0x0189ac thru 0x0284c5 (0xfb1a bytes) 000690 1787735111.864387 fallocate 0x012303 thru 0x01980a (0x7508 bytes) 000691 1787735111.870184 mapread 0x0153eb thru 0x0167ec (0x1402 bytes) 000692 1787735111.885172 mapwrite 0x01aa80 thru 0x025daf (0xb330 bytes) 000693 1787735111.925553 trunc from 0x02b843 to 0x0336b1 (0x7e6f bytes) 000694 1787735111.943406 MIRROR_RESYNC 2 mirrors 000695 1787735112.464000 MIRROR_EXTEND to 3 mirrors 000696 1787735113.066894 mapwrite 0x03df3c thru 0x03ffff (0x20c4 bytes) 000698 1787735113.225123 punch from 0x020f6b to 0x02d575 (0xc60b bytes) 000699 1787735113.241132 read 0x0182bc thru 0x0262fb (0xe040 bytes) 000700 1787735113.260591 fallocate 0x029730 thru 0x030cf4 (0x75c5 bytes) 000701 1787735113.278568 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf6c386cc (stale) Mirror 3: 0xf6c386cc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8a108cb5 Mirror 4: 0x8a108cb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000702 1787735113.869534 trunc from 0x029452 to 0x02a1a4 (0xd53 bytes) 000703 1787735113.902422 mapread 0x037b01 thru 0x03ffff (0x84ff bytes) 000704 1787735113.920514 mapwrite 0x01a475 thru 0x0204e7 (0x6073 bytes) 000705 1787735113.945124 mapread 0x030a8f thru 0x035536 (0x4aa8 bytes) 000706 1787735113.951592 trunc from 0x0383a9 to 0x03f210 (0x6e68 bytes) 000707 1787735113.955928 mapwrite 0x00249a thru 0x008fda (0x6b41 bytes) 000708 1787735113.972610 read 0x008b2b thru 0x01293c (0x9e12 bytes) 000709 1787735113.977100 mapread 0x011168 thru 0x0162d8 (0x5171 bytes) 000710 1787735113.978233 trunc from 0x040000 to 0x00ea24 (0x315db bytes) 000711 1787735114.022164 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xea25): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xea25): Mirror 2: 0xa6d684b (stale) Mirror 3: 0xa6d684b (stale) Mirror 4: 0xa6d684b (stale) CRC-32 checksum value for chunk [0, 0xea25): Mirror 1: 0xbd384196 Mirror 5: 0xbd384196 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xea25: skipped 000712 1787735114.666268 write 0x001ca7 thru 0x00c579 (0xa8d3 bytes) 000713 1787735114.689713 trunc from 0x00ea25 to 0x02cd82 (0x1e35e bytes) 000714 1787735114.705124 punch from 0x0042f0 to 0x00f02d (0xad3e bytes) 000715 1787735114.714249 mapwrite 0x018cae thru 0x026087 (0xd3da bytes) 000717 1787735114.784494 write 0x01be51 thru 0x01d0c2 (0x1272 bytes) 000718 1787735114.788357 mapwrite 0x02dc46 thru 0x03bbcb (0xdf86 bytes) 000719 1787735114.872525 mapread 0x025654 thru 0x02960d (0x3fba bytes) 000720 1787735114.879263 MIRROR_RESYNC 5 mirrors 000721 1787735116.158198 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) 000722 1787735116.468708 punch from 0x032de1 to 0x03bbcb (0x8deb bytes) 000723 1787735116.512531 mapread 0x00f390 thru 0x01bb72 (0xc7e3 bytes) 000724 1787735116.546322 trunc from 0x03bbcc to 0x02e77a (0xd451 bytes) 000725 1787735116.570908 trunc from 0x02e77b to 0x025c3b (0x8b3f bytes) 000726 1787735116.609087 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25c3c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c3c): Mirror 2: 0x32aa2cb1 (stale) Mirror 3: 0x32aa2cb1 (stale) Mirror 5: 0x32aa2cb1 (stale) CRC-32 checksum value for chunk [0, 0x25c3c): Mirror 1: 0x32aa2cb1 Mirror 6: 0x32aa2cb1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c3c: skipped 000727 1787735117.388435 trunc from 0x025c3c to 0x01f02d (0x6c0e bytes) 000728 1787735117.496410 fallocate 0x013b2f thru 0x02374d (0xfc1f bytes) 000729 1787735117.508456 punch from 0x00f132 to 0x0133ec (0x42bb bytes) 000730 1787735117.519396 trunc from 0x013b02 to 0x01c836 (0x8d35 bytes) 000731 1787735117.533375 trunc from 0x01f02e to 0x02a453 (0xb426 bytes) 000732 1787735117.570776 punch from 0x018867 to 0x01e950 (0x60ea bytes) 000733 1787735117.598672 MIRROR_RESYNC 5 mirrors 000736 1787735118.581123 trunc from 0x023a3d to 0x0293d8 (0x599c bytes) 000737 1787735118.627035 trunc from 0x02a454 to 0x03b675 (0x11222 bytes) 000738 1787735118.662873 write 0x01dbde thru 0x01fe79 (0x229c bytes) 000739 1787735118.674291 trunc from 0x02934a to 0x02a129 (0xde0 bytes) 000740 1787735118.678527 write 0x01147b thru 0x017597 (0x611d bytes) 000743 1787735118.693669 MIRROR_RESYNC 5 mirrors 000744 1787735120.103485 trunc from 0x03b676 to 0x01995e (0x21d17 bytes) 000745 1787735120.151832 fallocate 0x030ec7 thru 0x03cbd3 (0xbd0d bytes) 000746 1787735120.163855 mapwrite 0x025b04 thru 0x025e76 (0x373 bytes) 000747 1787735120.185188 fallocate 0x03892f thru 0x03a150 (0x1822 bytes) 000748 1787735120.190166 fallocate 0x03364e thru 0x034c69 (0x161c bytes) 000749 1787735120.194930 trunc from 0x018419 to 0x01c9b3 (0x459b bytes) 000750 1787735120.199301 write 0x00e22c thru 0x018829 (0xa5fe bytes) 000751 1787735120.209545 trunc from 0x03cbd4 to 0x027e65 (0x14d6e bytes) 000752 1787735120.231724 mapwrite 0x009876 thru 0x01383e (0x9fc9 bytes) 000753 1787735120.271793 write 0x022171 thru 0x022340 (0x1d0 bytes) 000754 1787735120.276471 punch from 0x0130bc to 0x022081 (0xefc6 bytes) 000755 1787735120.284227 punch from 0x0154cc to 0x01558c (0xc1 bytes) 000756 1787735120.289130 fallocate 0x015205 thru 0x025151 (0xff4d bytes) 000758 1787735120.300592 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x27e66): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27e66): Mirror 2: 0x10eea9ec (stale) Mirror 3: 0x10eea9ec (stale) Mirror 5: 0x10eea9ec (stale) Mirror 6: 0x10eea9ec (stale) CRC-32 checksum value for chunk [0, 0x27e66): Mirror 1: 0x1dfd2932 Mirror 7: 0x1dfd2932 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27e66: skipped 000759 1787735121.124786 mapwrite 0x03904d thru 0x03ffff (0x6fb3 bytes) 000760 1787735121.237175 write 0x032ce2 thru 0x03caf7 (0x9e16 bytes) 000761 1787735121.247516 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6e104776 (stale) Mirror 3: 0x6e104776 (stale) Mirror 5: 0x6e104776 (stale) Mirror 6: 0x6e104776 (stale) Mirror 7: 0x1d224874 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xafa69e15 Mirror 8: 0xafa69e15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000762 1787735121.946349 read 0x029507 thru 0x035617 (0xc111 bytes) 000763 1787735121.958311 fallocate 0x00728d thru 0x00da85 (0x67f9 bytes) 000764 1787735121.997949 mapwrite 0x03b078 thru 0x03ffff (0x4f88 bytes) 000765 1787735122.022965 read 0x03bd67 thru 0x03ffff (0x4299 bytes) 000766 1787735122.023508 mapread 0x00503f thru 0x00c38d (0x734f bytes) 000767 1787735122.054469 mapwrite 0x01a03b thru 0x01ce3c (0x2e02 bytes) 000768 1787735122.067294 punch from 0x03be33 to 0x03ffff (0x41cd bytes) 000769 1787735122.072102 read 0x01f804 thru 0x028e09 (0x9606 bytes) 000770 1787735122.073236 read 0x02a27e thru 0x0354f6 (0xb279 bytes) 000771 1787735122.073934 mapwrite 0x00c0dd thru 0x00d7fe (0x1722 bytes) 000772 1787735122.084958 write 0x01b795 thru 0x01e34f (0x2bbb bytes) 000774 1787735122.089442 punch from 0x01cee4 to 0x022ff9 (0x6116 bytes) 000776 1787735122.100703 mapread 0x03f013 thru 0x03ffff (0xfed bytes) 000777 1787735122.109722 fallocate 0x001dc9 thru 0x008b9a (0x6dd2 bytes) 000778 1787735122.114281 punch from 0x031cf9 to 0x037ebc (0x61c4 bytes) 000779 1787735122.119499 trunc from 0x040000 to 0x006880 (0x3977f bytes) 000781 1787735122.162145 trunc from 0x006881 to 0x0226d8 (0x1be58 bytes) 000782 1787735122.175237 write 0x028c2b thru 0x0364ea (0xd8c0 bytes) 000783 1787735122.176933 punch from 0x01662c to 0x01be30 (0x5805 bytes) 000784 1787735122.182268 mapread 0x02d23c thru 0x0364ea (0x92af bytes) 000785 1787735122.183278 mapwrite 0x03d99a thru 0x03ffff (0x2666 bytes) 000786 1787735122.229195 read 0x01f005 thru 0x02ae3e (0xbe3a bytes) 000787 1787735122.239298 punch from 0x00710d to 0x00e73f (0x7633 bytes) 000788 1787735122.243460 punch from 0x012f3d to 0x01b512 (0x85d6 bytes) 000789 1787735122.247183 trunc from 0x040000 to 0x018b25 (0x274da bytes) 000790 1787735122.262642 fallocate 0x025087 thru 0x0266e0 (0x165a bytes) 000791 1787735122.272173 MIRROR_RESYNC 7 mirrors 000792 1787735123.215243 mapwrite 0x036ec2 thru 0x03ffff (0x913e bytes) 000793 1787735123.319455 punch from 0x013b66 to 0x018b58 (0x4ff3 bytes) 000794 1787735123.328152 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x659ea605 (stale) Mirror 3: 0x659ea605 (stale) Mirror 5: 0x659ea605 (stale) Mirror 6: 0x659ea605 (stale) Mirror 8: 0x659ea605 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x659ea605 (stale) Mirror 3: 0x659ea605 (stale) Mirror 5: 0x659ea605 (stale) Mirror 6: 0x659ea605 (stale) Mirror 8: 0x659ea605 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000795 1787735123.715954 trunc from 0x035e5c to 0x03ab66 (0x4d0b bytes) 000796 1787735123.723094 mapwrite 0x02d6f4 thru 0x037543 (0x9e50 bytes) 000798 1787735123.765116 mapwrite 0x02b6bd thru 0x02ca4b (0x138f bytes) 000799 1787735123.783261 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x659ea605 (stale) Mirror 5: 0x659ea605 (stale) Mirror 6: 0x659ea605 (stale) Mirror 8: 0x659ea605 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x659ea605 (stale) Mirror 5: 0x659ea605 (stale) Mirror 6: 0x659ea605 (stale) Mirror 8: 0x659ea605 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000800 1787735124.091189 mapwrite 0x009ff4 thru 0x018784 (0xe791 bytes) 000801 1787735124.132965 trunc from 0x040000 to 0x03de0a (0x21f5 bytes) 000802 1787735124.153445 write 0x035814 thru 0x03ffff (0xa7ec bytes) 000803 1787735124.158993 read 0x02ee9e thru 0x03a980 (0xbae3 bytes) 000805 1787735124.168287 punch from 0x03a5bf to 0x03b158 (0xb9a bytes) 000806 1787735124.180632 trunc from 0x03fefe to 0x03ffff (0x102 bytes) 000807 1787735124.184267 mapwrite 0x028d92 thru 0x0350b2 (0xc321 bytes) 000808 1787735124.206678 trunc from 0x0296e5 to 0x029cf0 (0x60c bytes) 000809 1787735124.211407 mapread 0x000556 thru 0x0009e6 (0x491 bytes) 000810 1787735124.218245 write 0x013ac8 thru 0x017666 (0x3b9f bytes) 000811 1787735124.219353 fallocate 0x02d8bd thru 0x03b020 (0xd764 bytes) 000812 1787735124.229997 punch from 0x03cc29 to 0x03ffff (0x33d7 bytes) 000814 1787735124.236167 mapwrite 0x0258ad thru 0x02f08b (0x97df bytes) 000815 1787735124.262765 trunc from 0x040000 to 0x03caad (0x3552 bytes) 000816 1787735124.287445 MIRROR_RESYNC 5 mirrors 000817 1787735125.476341 trunc from 0x032517 to 0x03a695 (0x817f bytes) 000818 1787735125.518391 mapwrite 0x013eae thru 0x01f4be (0xb611 bytes) 000819 1787735125.602673 mapwrite 0x00eed5 thru 0x01dc0c (0xed38 bytes) 000820 1787735125.636988 punch from 0x006c3c to 0x01397d (0xcd42 bytes) 000821 1787735125.646220 write 0x00081f thru 0x00beb0 (0xb692 bytes) 000822 1787735125.661797 punch from 0x02cbcb to 0x037b2f (0xaf65 bytes) 000823 1787735125.668711 trunc from 0x033ba2 to 0x03a2d5 (0x6734 bytes) 000824 1787735125.673926 write 0x00fd9c thru 0x01682d (0x6a92 bytes) 000825 1787735125.679198 fallocate 0x03633f thru 0x03ffff (0x9cc1 bytes) 000826 1787735125.684249 punch from 0x035bca to 0x03ea56 (0x8e8d bytes) 000827 1787735125.694135 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x80761c4e (stale) Mirror 5: 0x80761c4e (stale) Mirror 8: 0x80761c4e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x80761c4e (stale) Mirror 5: 0x80761c4e (stale) Mirror 8: 0x80761c4e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000828 1787735126.104152 read 0x01dd21 thru 0x02d6b8 (0xf998 bytes) 000829 1787735126.123676 fallocate 0x032db0 thru 0x03ffff (0xd250 bytes) 000830 1787735126.130610 trunc from 0x02dd1a to 0x035e4a (0x8131 bytes) 000831 1787735126.142132 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x80761c4e (stale) Mirror 8: 0x80761c4e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x80761c4e (stale) Mirror 8: 0x80761c4e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000833 1787735126.462607 trunc from 0x040000 to 0x0041b4 (0x3be4b bytes) 000834 1787735126.493189 mapwrite 0x03c855 thru 0x03d1fe (0x9aa bytes) 000835 1787735126.533651 punch from 0x0022b7 to 0x004dff (0x2b49 bytes) 000836 1787735126.542206 trunc from 0x03d1ff to 0x002776 (0x3aa88 bytes) 000837 1787735126.560332 read 0x000d18 thru 0x002776 (0x1a5f bytes) 000838 1787735126.567645 read 0x0026d6 thru 0x002776 (0xa1 bytes) 000839 1787735126.568407 mapwrite 0x0277b6 thru 0x03037f (0x8bca bytes) 000840 1787735126.630164 fallocate 0x00a5a0 thru 0x00d09d (0x2afe bytes) 000841 1787735126.634645 mapread 0x013a49 thru 0x015cda (0x2292 bytes) 000842 1787735126.635454 mapwrite 0x00efa8 thru 0x0176d3 (0x872c bytes) 000843 1787735126.653731 fallocate 0x032837 thru 0x03ed6a (0xc534 bytes) 000844 1787735126.658187 trunc from 0x030380 to 0x01bab8 (0x148c7 bytes) 000845 1787735126.671218 write 0x00bad8 thru 0x013653 (0x7b7c bytes) 000846 1787735126.672522 fallocate 0x036507 thru 0x038e3d (0x2937 bytes) 000847 1787735126.677209 trunc from 0x038e3e to 0x016efd (0x21f40 bytes) 000848 1787735126.692179 punch from 0x00fefb to 0x016efd (0x7003 bytes) 000849 1787735126.704061 trunc from 0x016efe to 0x00adc5 (0xc138 bytes) 000850 1787735126.717259 mapwrite 0x024347 thru 0x029ce3 (0x599d bytes) 000851 1787735126.771458 punch from 0x00286b to 0x0036a2 (0xe38 bytes) 000852 1787735126.774251 trunc from 0x002684 to 0x00be30 (0x97ad bytes) 000854 1787735126.777420 write 0x0082cf thru 0x0121dd (0x9f0f bytes) 000855 1787735126.783036 mapread 0x010b01 thru 0x011741 (0xc41 bytes) 000856 1787735126.783648 mapread 0x016c3a thru 0x01747c (0x843 bytes) 000857 1787735126.792400 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29ce4): Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ce4): Mirror 5: 0xe403eeb5 (stale) Mirror 8: 0xe403eeb5 (stale) CRC-32 checksum value for chunk [0, 0x29ce4): Mirror 1: 0xa1e423d4 Mirror 9: 0xa1e423d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ce4: skipped 000858 1787735127.561493 mapwrite 0x035da8 thru 0x03fa65 (0x9cbe bytes) 000859 1787735127.644456 punch from 0x004082 to 0x00e9b5 (0xa934 bytes) 000860 1787735127.649953 punch from 0x0276c4 to 0x029347 (0x1c84 bytes) 000861 1787735127.655011 trunc from 0x03fa66 to 0x027f9b (0x17aca bytes) 000862 1787735127.674290 punch from 0x004fe5 to 0x01213a (0xd156 bytes) 000863 1787735127.679586 write 0x027136 thru 0x034b6e (0xda39 bytes) 000864 1787735127.688308 MIRROR_RESYNC 4 mirrors 000865 1787735128.538723 write 0x01cbac thru 0x02597e (0x8dd3 bytes) 000866 1787735128.575256 mapread 0x0270de thru 0x02854d (0x1470 bytes) 000867 1787735128.581302 mapread 0x028868 thru 0x034b6e (0xc307 bytes) 000868 1787735128.594127 write 0x024540 thru 0x025da3 (0x1864 bytes) 000869 1787735128.595008 mapwrite 0x02b93d thru 0x035457 (0x9b1b bytes) 000870 1787735128.626328 trunc from 0x027ce2 to 0x030c1f (0x8f3e bytes) 000871 1787735128.634101 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35458): Mirror 8: 0x52e565fc (stale) Mirror 9: 0x52e565fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35458): Mirror 8: 0x52e565fc (stale) Mirror 9: 0x52e565fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35458: skipped 000872 1787735128.881314 trunc from 0x02d567 to 0x035457 (0x7ef1 bytes) 000873 1787735128.887028 write 0x00afed thru 0x016a53 (0xba67 bytes) 000874 1787735128.901132 trunc from 0x035458 to 0x0070eb (0x2e36c bytes) 000875 1787735128.920291 trunc from 0x004d50 to 0x0070eb (0x239c bytes) 000877 1787735128.925946 punch from 0x005983 to 0x0070eb (0x1769 bytes) 000879 1787735128.930963 trunc from 0x0070ec to 0x039862 (0x32777 bytes) 000881 1787735128.945488 write 0x000790 thru 0x0073d8 (0x6c49 bytes) 000882 1787735128.954751 fallocate 0x026c10 thru 0x02ffcb (0x93bc bytes) 000883 1787735128.959673 punch from 0x02be67 to 0x02c76f (0x909 bytes) 000884 1787735128.967320 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39863): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39863): Mirror 8: 0x811469c0 (stale) Mirror 9: 0x811469c0 (stale) CRC-32 checksum value for chunk [0, 0x39863): Mirror 1: 0x1c18e465 Mirror 10: 0x1c18e465 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39863: skipped 000885 1787735129.683476 mapwrite 0x034860 thru 0x036d85 (0x2526 bytes) 000886 1787735129.752003 read 0x00e321 thru 0x00e810 (0x4f0 bytes) 000887 1787735129.758019 punch from 0x00fa0c to 0x016f59 (0x754e bytes) 000888 1787735129.761647 read 0x0033b9 thru 0x01076e (0xd3b6 bytes) 000889 1787735129.771312 write 0x03f763 thru 0x03ffff (0x89d bytes) 000890 1787735129.776240 punch from 0x0179d9 to 0x021496 (0x9abe bytes) 000891 1787735129.780380 read 0x03a4c1 thru 0x03ffff (0x5b3f bytes) 000892 1787735129.789376 fallocate 0x02e0fc thru 0x034491 (0x6396 bytes) 000893 1787735129.793629 read 0x0026cf thru 0x00f7d1 (0xd103 bytes) 000894 1787735129.794502 read 0x014af7 thru 0x0193af (0x48b9 bytes) 000895 1787735129.795408 punch from 0x030a3b to 0x03a3d6 (0x999c bytes) 000897 1787735129.801191 fallocate 0x003465 thru 0x009477 (0x6013 bytes) 000898 1787735129.806052 trunc from 0x00a82d to 0x014ed5 (0xa6a9 bytes) 000899 1787735129.810188 mapread 0x03a26c thru 0x03ffff (0x5d94 bytes) 000900 1787735129.825482 MIRROR_RESYNC 4 mirrors 000901 1787735130.598514 MIRROR_EXTEND to 5 mirrors 000903 1787735131.431992 mapwrite 0x0211d8 thru 0x0296aa (0x84d3 bytes) 000904 1787735131.534578 fallocate 0x011942 thru 0x013a47 (0x2106 bytes) 000905 1787735131.541154 fallocate 0x0396cf thru 0x03bdf2 (0x2724 bytes) 000906 1787735131.550278 mapwrite 0x01379e thru 0x01479e (0x1001 bytes) 000907 1787735131.577511 trunc from 0x040000 to 0x034d77 (0xb288 bytes) 000908 1787735131.615980 read 0x01546c thru 0x01ed61 (0x98f6 bytes) 000909 1787735131.617921 punch from 0x029c0a to 0x030077 (0x646e bytes) 000910 1787735131.624628 mapwrite 0x034e6c thru 0x036fd5 (0x216a bytes) 000911 1787735131.668999 mapwrite 0x022ab9 thru 0x02d8e2 (0xae2a bytes) 000912 1787735131.708011 mapread 0x028bce thru 0x033f23 (0xb356 bytes) 000913 1787735131.716960 MIRROR_RESYNC 5 mirrors 000914 1787735132.980508 fallocate 0x03ae6f thru 0x03ffff (0x5191 bytes) 000915 1787735133.017739 fallocate 0x021e7a thru 0x02a045 (0x81cc bytes) 000916 1787735133.024318 trunc from 0x028244 to 0x028661 (0x41e bytes) 000917 1787735133.031578 write 0x03522f thru 0x0366da (0x14ac bytes) 000918 1787735133.047279 mapread 0x016e75 thru 0x0198f1 (0x2a7d bytes) 000919 1787735133.080934 trunc from 0x040000 to 0x030a91 (0xf56e bytes) 000921 1787735133.101563 trunc from 0x030a92 to 0x005e55 (0x2ac3c bytes) 000922 1787735133.122407 read 0x004614 thru 0x005e55 (0x1842 bytes) 000923 1787735133.129507 mapread 0x0025f5 thru 0x005e55 (0x3861 bytes) 000924 1787735133.136038 read 0x001140 thru 0x001ff0 (0xeb1 bytes) 000925 1787735133.142126 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5e56): Mirror 9: 0xd86664f3 (stale) Mirror 10: 0xd86664f3 (stale) Mirror 11: 0xd86664f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5e56): Mirror 9: 0xd86664f3 (stale) Mirror 10: 0xd86664f3 (stale) Mirror 11: 0xd86664f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5e56: skipped 000926 1787735133.450169 mapwrite 0x005b2b thru 0x00d900 (0x7dd6 bytes) 000927 1787735133.559253 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd901): Mirror 10: 0xab8ccf14 (stale) Mirror 11: 0xab8ccf14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd901): Mirror 10: 0xab8ccf14 (stale) Mirror 11: 0xab8ccf14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd901: skipped 000928 1787735133.897303 read 0x004acd thru 0x00d900 (0x8e34 bytes) 000929 1787735133.905179 read 0x001fbe thru 0x00d900 (0xb943 bytes) 000930 1787735133.912155 punch from 0x0092de to 0x00d900 (0x4623 bytes) 000931 1787735133.923970 fallocate 0x028bf9 thru 0x029162 (0x56a bytes) 000932 1787735133.934566 MIRROR_RESYNC 3 mirrors 000933 1787735134.520418 punch from 0x00b26f to 0x00d900 (0x2692 bytes) 000934 1787735134.553547 fallocate 0x020f5e thru 0x0241e3 (0x3286 bytes) 000935 1787735134.559411 read 0x00d359 thru 0x00d900 (0x5a8 bytes) 000936 1787735134.569192 MIRROR_RESYNC 3 mirrors 000937 1787735135.429060 mapwrite 0x00a91f thru 0x0131c8 (0x88aa bytes) 000938 1787735135.508050 write 0x025cec thru 0x030d60 (0xb075 bytes) 000939 1787735135.512448 punch from 0x0033b0 to 0x01213b (0xed8c bytes) 000940 1787735135.518890 mapwrite 0x00c961 thru 0x019053 (0xc6f3 bytes) 000941 1787735135.554736 mapwrite 0x02c8cb thru 0x033d2d (0x7463 bytes) 000942 1787735135.598775 punch from 0x0335de to 0x033d2d (0x750 bytes) 000943 1787735135.613202 trunc from 0x01350c to 0x018993 (0x5488 bytes) 000944 1787735135.619472 mapread 0x030abd thru 0x033d2d (0x3271 bytes) 000946 1787735135.631732 write 0x0247cc thru 0x02bd1a (0x754f bytes) 000947 1787735135.638111 fallocate 0x03cad3 thru 0x03ffff (0x352d bytes) 000948 1787735135.643269 punch from 0x027f12 to 0x02d07a (0x5169 bytes) 000949 1787735135.659090 read 0x0079f6 thru 0x00c88f (0x4e9a bytes) 000951 1787735135.665257 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xbdb6586d (stale) Mirror 11: 0xbdb6586d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb97453bb Mirror 12: 0xb97453bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000952 1787735136.595889 fallocate 0x037483 thru 0x038a23 (0x15a1 bytes) 000953 1787735136.653786 trunc from 0x040000 to 0x036307 (0x9cf8 bytes) 000954 1787735136.680687 punch from 0x02b89c to 0x0327be (0x6f23 bytes) 000955 1787735136.695018 MIRROR_RESYNC 4 mirrors 000956 1787735138.170125 trunc from 0x036308 to 0x00eaf2 (0x27815 bytes) 000957 1787735138.227086 fallocate 0x0305b7 thru 0x0393b9 (0x8e03 bytes) 000958 1787735138.235793 read 0x000720 thru 0x007c11 (0x74f2 bytes) 000959 1787735138.243054 fallocate 0x021d4a thru 0x0298ae (0x7b65 bytes) 000960 1787735138.247126 fallocate 0x030d6f thru 0x032b3d (0x1dcf bytes) 000962 1787735138.259702 mapread 0x02d729 thru 0x032b3d (0x5415 bytes) 000963 1787735138.293140 write 0x007f37 thru 0x009ef2 (0x1fbc bytes) 000964 1787735138.301697 punch from 0x031dbe to 0x032b3d (0xd80 bytes) 000965 1787735138.311306 mapwrite 0x0290c3 thru 0x033f6f (0xaead bytes) 000966 1787735138.367364 write 0x016a31 thru 0x0250b8 (0xe688 bytes) 000967 1787735138.380047 mapwrite 0x00cf21 thru 0x01aa1b (0xdafb bytes) 000969 1787735138.461519 MIRROR_RESYNC 4 mirrors 000970 1787735139.701180 mapwrite 0x005ff6 thru 0x00f2d4 (0x92df bytes) 000971 1787735139.805272 punch from 0x01cac8 to 0x029313 (0xc84c bytes) 000972 1787735139.820651 trunc from 0x033f70 to 0x019364 (0x1ac0b bytes) 000973 1787735139.843208 trunc from 0x00f230 to 0x019364 (0xa135 bytes) 000974 1787735139.848873 mapread 0x00d8b3 thru 0x0147b4 (0x6f02 bytes) 000976 1787735139.861518 mapread 0x00210c thru 0x01058a (0xe47f bytes) 000977 1787735139.864002 trunc from 0x011a03 to 0x01281a (0xe18 bytes) 000978 1787735139.871319 trunc from 0x019365 to 0x023adf (0xa77b bytes) 000979 1787735139.894672 fallocate 0x018adf thru 0x0239f7 (0xaf19 bytes) 000980 1787735139.901210 fallocate 0x01f217 thru 0x020801 (0x15eb bytes) 000981 1787735139.906513 mapread 0x0075b3 thru 0x00bdad (0x47fb bytes) 000982 1787735139.907896 punch from 0x009bc7 to 0x010d72 (0x71ac bytes) 000983 1787735139.913335 mapread 0x0107ec thru 0x019911 (0x9126 bytes) 000984 1787735139.931356 trunc from 0x023ae0 to 0x015910 (0xe1cf bytes) 000985 1787735139.952016 write 0x029362 thru 0x02aed2 (0x1b71 bytes) 000986 1787735139.953485 trunc from 0x022dd4 to 0x02aed2 (0x80ff bytes) 000987 1787735139.963264 write 0x019a59 thru 0x025c81 (0xc229 bytes) 000988 1787735139.974016 trunc from 0x013f5c to 0x021c82 (0xdd27 bytes) 000989 1787735139.988397 mapwrite 0x028e9e thru 0x02ff58 (0x70bb bytes) 000990 1787735140.018493 fallocate 0x0000ea thru 0x00f6a9 (0xf5c0 bytes) 000991 1787735140.022296 punch from 0x0117af to 0x0154a5 (0x3cf7 bytes) 000992 1787735140.025925 read 0x014b97 thru 0x022168 (0xd5d2 bytes) 000993 1787735140.043880 trunc from 0x00d23b to 0x01c97f (0xf745 bytes) 000994 1787735140.054137 write 0x01e192 thru 0x02832d (0xa19c bytes) 000995 1787735140.055648 write 0x02a31c thru 0x031a3e (0x7723 bytes) 000996 1787735140.064357 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31a3f): Mirror 10: 0x70808e1e (stale) Mirror 11: 0x70808e1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31a3f): Mirror 10: 0x70808e1e (stale) Mirror 11: 0x70808e1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31a3f: skipped 000997 1787735140.395320 mapread 0x0212b1 thru 0x02ef8d (0xdcdd bytes) 000998 1787735140.412724 trunc from 0x00f3bf to 0x0192c4 (0x9f06 bytes) 000999 1787735140.419321 MIRROR_RESYNC 3 mirrors All operations completed A-OK! PASS 70a (83s) == sanity-flr test 71: check mirror extend parent fid ==== 05:05:46 (1787735146) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x45:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 05:05:55 (1787735155) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x379:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x376:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237672 s, 8.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x379:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x376:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868419 s, 12.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x379:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x377:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x376:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x379:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x377:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x376:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37b:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 05:06:03 (1787735163) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6339008 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5733312 .. Extending file size to 8246016 .. Extending file size to 8394752 .. failed Extending file size to 8334080 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8449312 .. Extending file size to 8889952 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8426144 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9251744 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9277472 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 61546 61547 61550 61552 61553 192.168.201.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre2 (opts:) 192.168.201.153@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (76s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:07:19 (1787735239) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5187072 .. Extending file size to 3639136 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 4186688 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6479488 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7755968 .. Extending file size to 7084000 .. Extending file size to 7663104 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8286944 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8442336 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8729184 .. failed Extending file size to 8366240 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9124960 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8798304 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 63206 63207 63210 63211 63214 failed 192.168.201.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre2 (opts:) 192.168.201.153@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (72s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:08:31 (1787735311) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2680 1268624 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17828 3589192 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23624 3583396 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 41452 7172588 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2680 1268624 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17828 3589192 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23624 3583396 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 41452 7172588 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0565203 s, 37.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192559 s, 10.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x63b102e8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:08:47 (1787735327) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0674504 s, 31.1 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 05:08:52 (1787735332) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x235:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x38d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x38d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x38e:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x390:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:09:00 (1787735340) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0955625 s, 21.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x36:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x38:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:09:07 (1787735347) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0638309 s, 32.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x38f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x390:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x391:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x38f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x390:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x391:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x392:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:09:14 (1787735354) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0827555 s, 25.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x37:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x39:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x37:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x39:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3a:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:09:44 (1787735384) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg153-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627281 s, 16.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x38:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x38:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x39:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:10:32 (1787735432) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg153-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x391:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x392:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0971135 s, 21.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x391:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x392:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x393:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x391:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x392:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x393:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:11:18 (1787735478) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg153-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg153-server: Pool lustre.test_204f created Adding targets to pool oleg153-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg153-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122964 s, 17.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg153-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg153-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg153-server: Pool lustre.test_204f destroyed PASS 204f (66s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:12:24 (1787735544) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:12:30 (1787735550) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:12:36 (1787735556) oleg153-server: Pool lustre.test_206 created Adding targets to pool oleg153-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg153-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg153-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg153-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg153-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:13:01 (1787735581) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056677 s, 18.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236804 s, 44.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc097f580 Mirror 3: 0xc097f580 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:13:10 (1787735590) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:13:12 (1787735592) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:13:15 (1787735595) Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0468243 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2080000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x20b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x20b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x20b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x700000 Read this (4 bytes): BBBB 192.168.201.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg153-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:13:48 (1787735628) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0260309 s, 315 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:13:53 (1787735633) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439351 s, 23.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x252 lmm_fid: [0x200000405:0x252:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x252 lmm_fid: [0x200000405:0x252:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611730 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 05:13:54.000000000 -0400 Modify: 2026-08-26 05:13:54.000000000 -0400 Change: 2026-08-26 05:13:55.000000000 -0400 Birth: 2026-08-26 05:13:54.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:14:00 (1787735640) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204902 s, 51.2 MB/s oleg153-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg153-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg153-server: O/280000401/d29/39d: File not found by ext2_lookup PASS 210b (19s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:14:19 (1787735659) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.139572 s, 293 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0110227 s, 372 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 05:14 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x257:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 05:14 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:14:24 (1787735664) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:14:30 (1787735670) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 1660 sec =========== 05:14:35 (1787735675) === sanity-flr: start cleanup 05:14:36 (1787735676) === === sanity-flr: finish cleanup 05:14:42 (1787735682) ===