-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 04:07:29 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg656-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 04:07:47 (1787818067) === oleg656-client.virtnet: executing check_config_client /mnt/lustre oleg656-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg656-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e5049a13000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5049a13000.idle_timeout=debug disable quota as required oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 04:08:29 (1787818109) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8241] == sanity-flr test 0a: lfs mirror create with -N option == 04:08:32 (1787818112) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:08:54 (1787818134) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:08:57 (1787818137) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:09:01 (1787818141) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:09:24 (1787818164) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:09:27 (1787818167) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 04:09:31 (1787818171) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9e5049a13000.stats=clear osc.lustre-OST0001-osc-ffff9e5049a13000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:09:40 (1787818180) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:09:52 (1787818192) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:10:01 (1787818201) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:10:05 (1787818205) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:10:15 (1787818215) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 04:10:24 (1787818224) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:10:31 (1787818231) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:10:38 (1787818238) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:10:46 (1787818246) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:10:56 (1787818256) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:11:03 (1787818263) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 1.13821 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114492 s, 9.2 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:11:12 (1787818272) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.101422 s, 31.0 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:11:20 (1787818280) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:11:31 (1787818291) Stopping /mnt/lustre-ost1 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5049a13000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5049a13000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg656-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg656-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e5049a13000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5049a13000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg656-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (134s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:13:45 (1787818425) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.495623 s, 14.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5049a13000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5049a13000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg656-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:14:32 (1787818472) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5049a13000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5049a13000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg656-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e5049a13000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5049a13000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg656-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (93s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:16:05 (1787818565) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.46833 s, 15.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 04:16 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.007s user 0m0.001s sys 0m0.181s Stopping /mnt/lustre-ost1 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5049a13000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5049a13000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.075s user 0m0.001s sys 0m0.216s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg656-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg656-server oleg656-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e5049a13000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5049a13000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.471s user 0m0.000s sys 0m0.168s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg656-server: oleg656-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg656-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg656-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e5049a13000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5049a13000.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:17:42 (1787818662) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:17:45 (1787818665) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:17:49 (1787818669) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:17:52 (1787818672) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 04:18:00 (1787818680) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.540814 s, 23.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:18:14 (1787818694) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614052 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289881 s, 36.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:18:23 (1787818703) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:18:31 (1787818711) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00475928 s, 215 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00281882 s, 363 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00127071 s, 806 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000885486 s, 1.2 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 04:18:41 (1787818721) create test files with size 15741630 .. 3b8f08bfc0b631c4185e49f82d72aa73 - 09d75bf54ecce14ea5a71633a6603b21 - e9b3b72a98da98066e83076b445d629d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg656-client.virtnet /mnt/lustre (opts:) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 04:19:11 (1787818751) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:19:25 (1787818765) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:19:32 (1787818772) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914393 s, 11.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351742 s, 29.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb3:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb4:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 04:19:41 (1787818781) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122635 s, 17.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.368564 s, 11.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135388 s, 15.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 04:19:57 (1787818797) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 04:20:00 (1787818800) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:20:04 (1787818804) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033481 s, 31.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 04:20:11 (1787818811) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.307431 s, 10.2 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:20:22 (1787818822) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xbe:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:20:34 (1787818834) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.450907 s, 23.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg656-client.virtnet /mnt/lustre (opts:) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:20:49 (1787818849) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9e5059870800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.286777 s, 36.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg656-client.virtnet /mnt/lustre (opts:) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9e50481db800.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:21:08 (1787818868) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:21:45 (1787818905) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:22:17 (1787818937) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 04:22:20 (1787818940) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.298839 s, 10.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.23578 s, 13.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9d3a626e (nosync,stale) Mirror 2: 0x9d3a626e (nosync,stale) ** verify mirror 2 doesn't change original checksum: 20215130b1ee58daf1d85a2be4e5ba9a - mirror 1 checksum: a7de02622f59710cdef2fd7597502b5a - mirror 2 checksum: 20215130b1ee58daf1d85a2be4e5ba9a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T04:22:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T04:22:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 20215130b1ee58daf1d85a2be4e5ba9a - mirror 1 checksum: a7de02622f59710cdef2fd7597502b5a - mirror 2 checksum: a7de02622f59710cdef2fd7597502b5a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T04:22:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T04:22:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 04:22:31 (1787818951) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303672 s, 34.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:22:38 (1787818958) Failing mds1 on oleg656-server Stopping /mnt/lustre-mds1 (opts:) on oleg656-server 04:22:50 (1787818970) shut down facet: mds1 facet_host: oleg656-server facet_failover_host: oleg656-server Failover mds1 to oleg656-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg656-server: oleg656-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-MDT0000 04:23:13 (1787818993) targets are mounted 04:23:13 (1787818993) facet_failover done oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224434 s, 46.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } PASS 50A (53s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:23:31 (1787819011) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0708452 s, 289 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00983341 s, 2.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:23:41 (1787819021) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0419954 s, 488 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0133309 s, 1.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:23:52 (1787819032) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0168571 s, 972 kB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:24:00 (1787819040) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0505746 s, 405 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:24:09 (1787819049) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0187166 s, 219 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0563681 s, 72.7 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:24:16 (1787819056) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:24:37 (1787819077) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:25:03 (1787819103) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:25:25 (1787819125) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:25:49 (1787819149) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 04:25:55 (1787819155) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57585 mirror split pid 57587 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57585 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57587 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 04:27:03 (1787819223) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 9225 fallocating to largest ever: 0x2b234 000001 1787819225.327212 fallocate 0x020570 thru 0x02b233 (0xacc4 bytes) 000003 1787819225.339621 mapwrite 0x03aa66 thru 0x03bd3a (0x12d5 bytes) 000004 1787819225.384976 write 0x00e1f2 thru 0x01d4d2 (0xf2e1 bytes) 000005 1787819225.399238 trunc from 0x03bd3b to 0x00abed (0x3114d bytes) 000006 1787819225.422846 mapread 0x007d96 thru 0x00a789 (0x29f4 bytes) 000007 1787819225.438604 fallocate 0x02388d thru 0x027645 (0x3db9 bytes) 000009 1787819225.446428 read 0x006542 thru 0x009d77 (0x3836 bytes) 000010 1787819225.447747 trunc from 0x007451 to 0x00abed (0x379d bytes) 000011 1787819225.454543 mapread 0x003294 thru 0x0093b7 (0x6124 bytes) 000012 1787819225.467751 mapwrite 0x008524 thru 0x010851 (0x832e bytes) 000013 1787819225.577616 write 0x0111d1 thru 0x020511 (0xf341 bytes) 000014 1787819225.579385 trunc from 0x01f3ed to 0x020511 (0x1125 bytes) fallocating to largest ever: 0x3a724 000015 1787819225.611417 fallocate 0x02f969 thru 0x03a723 (0xadbb bytes) 000016 1787819225.623921 read 0x02090e thru 0x02acff (0xa3f2 bytes) 000017 1787819225.641436 mapread 0x01ff7f thru 0x02295d (0x29df bytes) 000018 1787819225.642394 fallocate 0x01876e thru 0x019a9a (0x132d bytes) 000019 1787819225.650911 fallocate 0x019738 thru 0x0222cd (0x8b96 bytes) 000020 1787819225.659021 read 0x03663b thru 0x03a723 (0x40e9 bytes) 000021 1787819225.660852 punch from 0x01825d to 0x01faba (0x785e bytes) 000022 1787819225.676142 write 0x01f7be thru 0x027528 (0x7d6b bytes) 000023 1787819225.696308 fallocate 0x00ff41 thru 0x01c431 (0xc4f1 bytes) 000024 1787819225.706311 fallocate 0x0115b5 thru 0x0169a1 (0x53ed bytes) 000025 1787819225.712799 punch from 0x037019 to 0x03a723 (0x370b bytes) 000026 1787819225.719715 mapwrite 0x0049aa thru 0x00e947 (0x9f9e bytes) 000027 1787819225.775428 mapwrite 0x023967 thru 0x02c331 (0x89cb bytes) 000028 1787819225.817706 trunc from 0x03a724 to 0x0311c9 (0x955a bytes) 000029 1787819225.857247 trunc from 0x028a3d to 0x0311c9 (0x878d bytes) 000030 1787819225.878749 fallocate 0x017eec thru 0x026796 (0xe8ab bytes) 000031 1787819225.892996 trunc from 0x002e19 to 0x009689 (0x6871 bytes) 000032 1787819225.905568 read 0x02dadc thru 0x0311c9 (0x36ee bytes) 000033 1787819225.928669 trunc from 0x010fac to 0x013173 (0x21c8 bytes) 000034 1787819225.933840 trunc from 0x0311ca to 0x00e6fc (0x22acd bytes) 000036 1787819225.961750 punch from 0x000de0 to 0x00b1b8 (0xa3d9 bytes) 000037 1787819225.974242 trunc from 0x002f9e to 0x00e6fc (0xb75f bytes) 000038 1787819225.987762 punch from 0x001c89 to 0x007d8f (0x6107 bytes) 000039 1787819225.994910 mapread 0x0001cf thru 0x005453 (0x5285 bytes) 000040 1787819226.049906 read 0x00a625 thru 0x00c1f0 (0x1bcc bytes) 000041 1787819226.071038 write 0x0334d6 thru 0x03ffff (0xcb2a bytes) 000042 1787819226.080092 trunc from 0x040000 to 0x020ff6 (0x1f009 bytes) 000043 1787819226.128781 mapread 0x00f7b7 thru 0x016910 (0x715a bytes) 000044 1787819226.140826 fallocate 0x02bf66 thru 0x0396af (0xd74a bytes) 000045 1787819226.147087 mapread 0x00ff43 thru 0x012259 (0x2317 bytes) 000046 1787819226.150591 write 0x02dd68 thru 0x03d5a5 (0xf83e bytes) 000048 1787819226.155489 punch from 0x035fdc to 0x03d5a5 (0x75ca bytes) 000049 1787819226.178254 mapread 0x0221fa thru 0x029d94 (0x7b9b bytes) 000050 1787819226.191274 mapread 0x03a6b9 thru 0x03d5a5 (0x2eed bytes) 000051 1787819226.195626 trunc from 0x03d5a6 to 0x021817 (0x1bd8e bytes) 000052 1787819226.237387 write 0x00637b thru 0x013161 (0xcde7 bytes) 000053 1787819226.247565 punch from 0x00f7e6 to 0x01e0d0 (0xe8eb bytes) fallocating to largest ever: 0x3f329 000054 1787819226.277580 fallocate 0x0365ad thru 0x03f328 (0x8d7c bytes) 000055 1787819226.286806 trunc from 0x00843d to 0x014cfb (0xc8bf bytes) 000056 1787819226.301680 mapwrite 0x005ae5 thru 0x0074b7 (0x19d3 bytes) 000057 1787819226.340565 mapread 0x008d85 thru 0x0094f2 (0x76e bytes) 000058 1787819226.356595 mapwrite 0x00b257 thru 0x00e798 (0x3542 bytes) 000059 1787819226.393287 punch from 0x010cb4 to 0x0161e3 (0x5530 bytes) 000060 1787819226.397747 write 0x00763e thru 0x011d43 (0xa706 bytes) 000061 1787819226.404190 trunc from 0x014715 to 0x014d58 (0x644 bytes) 000062 1787819226.412358 MIRROR_EXTEND to 2 mirrors 000063 1787819227.021911 punch from 0x005d08 to 0x012fb8 (0xd2b1 bytes) 000064 1787819227.074102 trunc from 0x021818 to 0x021a7e (0x267 bytes) 000065 1787819227.136532 punch from 0x01d2a0 to 0x021a7e (0x47df bytes) 000066 1787819227.142257 mapread 0x010cec thru 0x017515 (0x682a bytes) 000067 1787819227.160825 trunc from 0x021a7f to 0x029179 (0x76fb bytes) 000068 1787819227.175541 punch from 0x02388f to 0x029179 (0x58eb bytes) 000069 1787819227.185253 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2917a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2917a): Mirror 2: 0x27b6c955 (stale) CRC-32 checksum value for chunk [0, 0x2917a): Mirror 1: 0x61a7cc32 Mirror 3: 0x61a7cc32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2917a: skipped 000070 1787819228.378055 trunc from 0x02265b to 0x029179 (0x6b1f bytes) 000071 1787819228.421130 trunc from 0x0152fb to 0x01e3e9 (0x90ef bytes) 000072 1787819228.431651 MIRROR_RESYNC 3 mirrors 000073 1787819229.117984 trunc from 0x02917a to 0x00cef9 (0x1c280 bytes) 000074 1787819229.163531 fallocate 0x022774 thru 0x02925c (0x6ae9 bytes) 000075 1787819229.177127 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2925d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2925d): Mirror 2: 0xe88ce1b5 (stale) Mirror 3: 0xe88ce1b5 (stale) CRC-32 checksum value for chunk [0, 0x2925d): Mirror 1: 0xe88ce1b5 Mirror 4: 0xe88ce1b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2925d: skipped 000077 1787819230.119512 mapread 0x006e33 thru 0x0090ff (0x22cd bytes) 000078 1787819230.162350 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2925d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2925d): Mirror 2: 0xe88ce1b5 (stale) Mirror 3: 0xe88ce1b5 (stale) CRC-32 checksum value for chunk [0, 0x2925d): Mirror 1: 0xe88ce1b5 Mirror 4: 0xe88ce1b5 Mirror 5: 0xe88ce1b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2925d: skipped 000079 1787819230.983322 trunc from 0x006577 to 0x014bef (0xe679 bytes) 000080 1787819231.028839 read 0x0024d4 thru 0x00763e (0x516b bytes) 000081 1787819231.037545 read 0x024a5c thru 0x024c72 (0x217 bytes) 000083 1787819231.046717 punch from 0x025638 to 0x02925c (0x3c25 bytes) 000084 1787819231.052605 trunc from 0x01fb52 to 0x02925c (0x970b bytes) 000085 1787819231.057436 trunc from 0x0162e5 to 0x0204db (0xa1f7 bytes) 000086 1787819231.063226 trunc from 0x01baa5 to 0x01e985 (0x2ee1 bytes) 000087 1787819231.068445 trunc from 0x02925d to 0x024060 (0x51fc bytes) 000088 1787819231.093005 trunc from 0x024061 to 0x00012e (0x23f32 bytes) 000090 1787819231.117249 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x12f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12f): Mirror 2: 0xece348c8 (stale) Mirror 3: 0xece348c8 (stale) Mirror 4: 0xece348c8 (stale) Mirror 5: 0xece348c8 (stale) CRC-32 checksum value for chunk [0, 0x12f): Mirror 1: 0xece348c8 Mirror 6: 0xece348c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12f: skipped 000091 1787819231.923232 mapwrite 0x01ae15 thru 0x01b8d9 (0xac5 bytes) 000092 1787819232.026257 mapread 0x00768e thru 0x014d8c (0xd6ff bytes) 000093 1787819232.045325 mapread 0x009749 thru 0x010d1a (0x75d2 bytes) 000094 1787819232.046305 fallocate 0x022178 thru 0x02920f (0x7098 bytes) 000095 1787819232.051175 mapwrite 0x023a9c thru 0x02cd5a (0x92bf bytes) 000096 1787819232.105798 trunc from 0x0202c2 to 0x021503 (0x1242 bytes) 000097 1787819232.110779 mapread 0x02b6e0 thru 0x02cd5a (0x167b bytes) 000099 1787819232.116695 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cd5b): Mirror 3: 0x8575412a (stale) Mirror 4: 0x8575412a (stale) Mirror 5: 0x8575412a (stale) Mirror 6: 0xf6cd231e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd5b): Mirror 3: 0x8575412a (stale) Mirror 4: 0x8575412a (stale) Mirror 5: 0x8575412a (stale) Mirror 6: 0xf6cd231e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd5b: skipped 000100 1787819232.612759 punch from 0x008b0a to 0x00c5ef (0x3ae6 bytes) 000101 1787819232.622379 trunc from 0x00e2b5 to 0x017daf (0x9afb bytes) 000102 1787819232.627939 fallocate 0x023ffa thru 0x027bc2 (0x3bc9 bytes) 000103 1787819232.638545 trunc from 0x02cd5b to 0x012a4f (0x1a30b bytes) 000104 1787819232.671621 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x12a50): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12a50): Mirror 3: 0xe024268d (stale) Mirror 4: 0xe024268d (stale) Mirror 5: 0xe024268d (stale) Mirror 6: 0x5b03ea06 (stale) CRC-32 checksum value for chunk [0, 0x12a50): Mirror 1: 0x5b03ea06 Mirror 7: 0x5b03ea06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12a50: skipped 000105 1787819233.590340 trunc from 0x012a50 to 0x026b46 (0x140f7 bytes) 000106 1787819233.652245 MIRROR_RESYNC 6 mirrors 000107 1787819234.629704 write 0x03e210 thru 0x03ec98 (0xa89 bytes) 000109 1787819234.678427 read 0x0162c3 thru 0x02200d (0xbd4b bytes) 000111 1787819234.690614 punch from 0x018825 to 0x0283eb (0xfbc7 bytes) 000112 1787819234.695056 trunc from 0x03ec99 to 0x035bce (0x90ca bytes) fallocating to largest ever: 0x40000 000113 1787819234.712724 fallocate 0x0340bc thru 0x03ffff (0xbf44 bytes) 000114 1787819234.725715 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35bcf): Mirror 4: 0x4bc0d935 (stale) Mirror 5: 0x4bc0d935 (stale) Mirror 6: 0x4bc0d935 (stale) Mirror 7: 0x4bc0d935 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35bcf): Mirror 4: 0x4bc0d935 (stale) Mirror 5: 0x4bc0d935 (stale) Mirror 6: 0x4bc0d935 (stale) Mirror 7: 0x4bc0d935 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35bcf: skipped 000115 1787819235.088660 write 0x025b99 thru 0x030563 (0xa9cb bytes) 000116 1787819235.117535 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35bcf): Mirror 4: 0x4bc0d935 (stale) Mirror 5: 0x4bc0d935 (stale) Mirror 7: 0x4bc0d935 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35bcf): Mirror 4: 0x4bc0d935 (stale) Mirror 5: 0x4bc0d935 (stale) Mirror 7: 0x4bc0d935 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35bcf: skipped 000117 1787819235.701589 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35bcf): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35bcf): Mirror 4: 0x4bc0d935 (stale) Mirror 5: 0x4bc0d935 (stale) Mirror 7: 0x4bc0d935 (stale) CRC-32 checksum value for chunk [0, 0x35bcf): Mirror 1: 0x52978bd9 Mirror 8: 0x52978bd9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35bcf: skipped 000118 1787819236.929306 trunc from 0x01c431 to 0x022d7b (0x694b bytes) 000119 1787819236.976947 mapread 0x00d795 thru 0x0156b1 (0x7f1d bytes) 000120 1787819237.001586 punch from 0x034b5f to 0x035bce (0x1070 bytes) 000121 1787819237.009750 read 0x01571a thru 0x02369e (0xdf85 bytes) 000122 1787819237.027126 fallocate 0x00ba87 thru 0x00fb8c (0x4106 bytes) 000123 1787819237.032127 trunc from 0x035bcf to 0x032b05 (0x30c9 bytes) 000124 1787819237.053273 punch from 0x00e654 to 0x0196be (0xb06b bytes) 000125 1787819237.059279 punch from 0x01422f to 0x01f1b3 (0xaf85 bytes) 000126 1787819237.063411 trunc from 0x01afe6 to 0x026da4 (0xbdbf bytes) 000127 1787819237.070042 trunc from 0x00f4eb to 0x01b07e (0xbb94 bytes) 000128 1787819237.074973 read 0x002c79 thru 0x0108cd (0xdc55 bytes) 000129 1787819237.085696 punch from 0x005639 to 0x00d5d9 (0x7fa1 bytes) 000130 1787819237.093629 punch from 0x02f9e4 to 0x032b05 (0x3122 bytes) 000131 1787819237.099756 mapwrite 0x00aaee thru 0x00eb87 (0x409a bytes) 000132 1787819237.142731 mapwrite 0x03c49c thru 0x03ffff (0x3b64 bytes) 000133 1787819237.187305 trunc from 0x040000 to 0x034e08 (0xb1f7 bytes) 000134 1787819237.206663 trunc from 0x034e09 to 0x03dc67 (0x8e5f bytes) 000135 1787819237.230469 write 0x021418 thru 0x02de15 (0xc9fe bytes) 000136 1787819237.232670 fallocate 0x006b0e thru 0x007c34 (0x1127 bytes) 000137 1787819237.254522 fallocate 0x0240b5 thru 0x02ce7a (0x8dc6 bytes) 000139 1787819237.265082 read 0x0078bc thru 0x00f560 (0x7ca5 bytes) 000140 1787819237.281514 trunc from 0x03dc68 to 0x024ace (0x19199 bytes) 000141 1787819237.303630 trunc from 0x010c46 to 0x012a0a (0x1dc5 bytes) 000142 1787819237.313152 punch from 0x0168c0 to 0x022ff7 (0xc738 bytes) 000143 1787819237.327624 mapwrite 0x001b63 thru 0x0051b1 (0x364f bytes) 000144 1787819237.347031 trunc from 0x024acf to 0x007fe2 (0x1caec bytes) 000145 1787819237.372378 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7fe3): Mirror 4: 0x692cbdef (stale) Mirror 5: 0x692cbdef (stale) Mirror 8: 0x692cbdef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7fe3): Mirror 4: 0x692cbdef (stale) Mirror 5: 0x692cbdef (stale) Mirror 8: 0x692cbdef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7fe3: skipped 000146 1787819237.758718 read 0x0042ea thru 0x007fe2 (0x3cf9 bytes) 000147 1787819237.768317 mapwrite 0x02aa7e thru 0x02f67e (0x4c01 bytes) 000149 1787819237.866729 punch from 0x0196c9 to 0x01effa (0x5932 bytes) 000150 1787819237.873687 mapwrite 0x01d356 thru 0x01f938 (0x25e3 bytes) 000151 1787819237.906238 trunc from 0x02f67f to 0x030c2b (0x15ad bytes) 000153 1787819237.932562 mapread 0x005e7c thru 0x00e714 (0x8899 bytes) 000154 1787819237.946942 mapwrite 0x03f16e thru 0x03ffff (0xe92 bytes) 000155 1787819237.984371 trunc from 0x040000 to 0x02f98d (0x10672 bytes) 000156 1787819238.001625 mapread 0x02d252 thru 0x02f98d (0x273c bytes) 000158 1787819238.004675 punch from 0x01a9de to 0x0296a4 (0xecc7 bytes) 000159 1787819238.020957 MIRROR_RESYNC 4 mirrors 000160 1787819239.072797 trunc from 0x01cefd to 0x02b943 (0xea47 bytes) 000161 1787819239.101570 mapread 0x023471 thru 0x02f3c2 (0xbf52 bytes) 000162 1787819239.125381 mapwrite 0x007778 thru 0x00e49f (0x6d28 bytes) 000163 1787819239.154634 write 0x03b7ac thru 0x03ffff (0x4854 bytes) 000165 1787819239.156289 trunc from 0x040000 to 0x030efa (0xf105 bytes) 000166 1787819239.172646 trunc from 0x030efb to 0x020a96 (0x10464 bytes) 000167 1787819239.186663 mapread 0x00cbf6 thru 0x0105f9 (0x3a04 bytes) 000168 1787819239.188423 trunc from 0x0133b1 to 0x0133e1 (0x31 bytes) 000169 1787819239.196418 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x20a97): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20a97): Mirror 4: 0x2faacd18 (stale) Mirror 5: 0x2faacd18 (stale) Mirror 8: 0x2faacd18 (stale) CRC-32 checksum value for chunk [0, 0x20a97): Mirror 1: 0x3e24c12c Mirror 9: 0x3e24c12c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20a97: skipped 000170 1787819239.953547 fallocate 0x03f297 thru 0x03ffff (0xd69 bytes) 000171 1787819239.991623 read 0x00ec26 thru 0x017979 (0x8d54 bytes) 000172 1787819240.010045 mapwrite 0x02400c thru 0x0324d9 (0xe4ce bytes) 000173 1787819240.081744 punch from 0x034dc4 to 0x03ffff (0xb23c bytes) 000174 1787819240.086540 trunc from 0x03f04b to 0x03ffff (0xfb5 bytes) 000175 1787819240.090760 fallocate 0x01a835 thru 0x0259af (0xb17b bytes) 000176 1787819240.095403 fallocate 0x0070d5 thru 0x007ce7 (0xc13 bytes) 000177 1787819240.100224 write 0x03f358 thru 0x03ffff (0xca8 bytes) 000178 1787819240.109438 punch from 0x02c614 to 0x03573b (0x9128 bytes) 000179 1787819240.117159 mapwrite 0x021eab thru 0x02f0c4 (0xd21a bytes) 000180 1787819240.176598 read 0x038c59 thru 0x03ffff (0x73a7 bytes) 000181 1787819240.190488 trunc from 0x03d7f0 to 0x03ffff (0x2810 bytes) 000182 1787819240.200327 punch from 0x00e199 to 0x012695 (0x44fd bytes) 000183 1787819240.225359 MIRROR_RESYNC 5 mirrors 000184 1787819241.567644 mapread 0x02f2d5 thru 0x036ca4 (0x79d0 bytes) 000185 1787819241.586691 mapread 0x0136bc thru 0x01fea9 (0xc7ee bytes) 000186 1787819241.605660 trunc from 0x040000 to 0x0392ad (0x6d52 bytes) 000187 1787819241.646554 read 0x0008c1 thru 0x009e60 (0x95a0 bytes) 000189 1787819241.658991 write 0x02f5e5 thru 0x0376b4 (0x80d0 bytes) 000190 1787819241.672344 write 0x03b682 thru 0x03efbf (0x393e bytes) 000191 1787819241.673686 mapwrite 0x02dd88 thru 0x037a02 (0x9c7b bytes) 000192 1787819241.701650 trunc from 0x0126ff to 0x020ddb (0xe6dd bytes) 000193 1787819241.706891 read 0x0054f2 thru 0x00b5c6 (0x60d5 bytes) 000194 1787819241.717661 read 0x032d9f thru 0x03efbf (0xc221 bytes) 000195 1787819241.718538 trunc from 0x03efc0 to 0x01c058 (0x22f67 bytes) 000196 1787819241.755553 write 0x001872 thru 0x00e1e5 (0xc974 bytes) 000197 1787819241.758281 read 0x005a33 thru 0x00ccd7 (0x72a5 bytes) 000200 1787819241.760401 fallocate 0x022e8e thru 0x0307f7 (0xd96a bytes) 000201 1787819241.765941 write 0x00f24c thru 0x01aff3 (0xbda8 bytes) 000202 1787819241.768494 trunc from 0x016002 to 0x01c058 (0x6057 bytes) 000203 1787819241.787560 read 0x0144bf thru 0x01c058 (0x7b9a bytes) 000204 1787819241.796611 fallocate 0x00fb44 thru 0x01c51a (0xc9d7 bytes) 000205 1787819241.802471 trunc from 0x00a1a1 to 0x017d02 (0xdb62 bytes) 000206 1787819241.808694 mapwrite 0x0230b9 thru 0x02c90b (0x9853 bytes) 000207 1787819241.860611 punch from 0x01679d to 0x01f916 (0x917a bytes) 000208 1787819241.868356 punch from 0x01dc72 to 0x0289c5 (0xad54 bytes) 000209 1787819241.874019 mapwrite 0x003f9b thru 0x00541b (0x1481 bytes) 000210 1787819241.887642 trunc from 0x02b41b to 0x02c90b (0x14f1 bytes) 000211 1787819241.892144 read 0x00dce4 thru 0x01b82b (0xdb48 bytes) 000212 1787819241.912779 punch from 0x017056 to 0x021186 (0xa131 bytes) 000213 1787819241.922840 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c90c): Mirror 4: 0xcec483d0 (stale) Mirror 5: 0xcec483d0 (stale) Mirror 8: 0xcec483d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c90c): Mirror 4: 0xcec483d0 (stale) Mirror 5: 0xcec483d0 (stale) Mirror 8: 0xcec483d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c90c: skipped 000214 1787819242.214205 trunc from 0x01c740 to 0x01f5ad (0x2e6e bytes) 000215 1787819242.219775 mapwrite 0x01808f thru 0x01b9b3 (0x3925 bytes) 000216 1787819242.252195 punch from 0x019696 to 0x01d3e9 (0x3d54 bytes) 000217 1787819242.257765 write 0x01744d thru 0x01afac (0x3b60 bytes) 000218 1787819242.267507 trunc from 0x02c90c to 0x0073e2 (0x25529 bytes) 000219 1787819242.288775 trunc from 0x0073e3 to 0x022d76 (0x1b994 bytes) 000221 1787819242.307979 trunc from 0x001ed4 to 0x00a95d (0x8a8a bytes) 000222 1787819242.314785 fallocate 0x021233 thru 0x02dd5e (0xcb2c bytes) 000223 1787819242.320062 trunc from 0x022d77 to 0x0028d0 (0x204a6 bytes) 000224 1787819242.341655 fallocate 0x00dcd0 thru 0x015da5 (0x80d6 bytes) 000225 1787819242.347023 read 0x001215 thru 0x0028d0 (0x16bc bytes) 000226 1787819242.357570 mapwrite 0x01a0af thru 0x025d79 (0xbccb bytes) 000227 1787819242.462024 write 0x00f482 thru 0x0104ba (0x1039 bytes) 000228 1787819242.476353 mapread 0x01e806 thru 0x025ab5 (0x72b0 bytes) 000229 1787819242.478424 punch from 0x00ff7c to 0x01c67d (0xc702 bytes) 000230 1787819242.500724 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25d7a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25d7a): Mirror 4: 0x2eaa1bed (stale) Mirror 5: 0x2eaa1bed (stale) Mirror 8: 0x2eaa1bed (stale) CRC-32 checksum value for chunk [0, 0x25d7a): Mirror 1: 0xb168842e Mirror 9: 0xb168842e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25d7a: skipped 000231 1787819243.641737 mapread 0x00c5fc thru 0x01b09c (0xeaa1 bytes) 000232 1787819243.684804 trunc from 0x00b84a to 0x013b5d (0x8314 bytes) 000233 1787819243.733194 fallocate 0x011cfe thru 0x01929c (0x759f bytes) 000234 1787819243.739355 fallocate 0x02108b thru 0x025803 (0x4779 bytes) 000235 1787819243.745084 trunc from 0x025d7a to 0x0347f6 (0xea7d bytes) 000236 1787819243.791597 fallocate 0x02293e thru 0x024481 (0x1b44 bytes) 000237 1787819243.797510 punch from 0x01d4dc to 0x023fe2 (0x6b07 bytes) 000238 1787819243.802392 trunc from 0x03115e to 0x0347f6 (0x3699 bytes) 000239 1787819243.809564 mapwrite 0x02d54c thru 0x03adff (0xd8b4 bytes) 000240 1787819243.878745 punch from 0x01f0a2 to 0x0272c4 (0x8223 bytes) 000241 1787819243.887755 fallocate 0x036998 thru 0x03e116 (0x777f bytes) 000242 1787819243.894255 mapwrite 0x037743 thru 0x03c0df (0x499d bytes) 000243 1787819243.934202 mapread 0x03560b thru 0x037c55 (0x264b bytes) 000244 1787819243.940647 MIRROR_RESYNC 5 mirrors 000245 1787819245.170791 read 0x037fc7 thru 0x03d8d7 (0x5911 bytes) 000246 1787819245.183810 fallocate 0x02c2a7 thru 0x031d5f (0x5ab9 bytes) 000247 1787819245.233292 trunc from 0x011fbd to 0x0197db (0x781f bytes) 000249 1787819245.240466 mapread 0x0338d2 thru 0x03627a (0x29a9 bytes) 000250 1787819245.276530 trunc from 0x01399e to 0x01bea7 (0x850a bytes) 000251 1787819245.283464 trunc from 0x03e117 to 0x03abb9 (0x355d bytes) 000252 1787819245.321767 MIRROR_RESYNC 5 mirrors 000253 1787819246.609802 fallocate 0x00ed42 thru 0x01d92b (0xebea bytes) 000254 1787819246.645106 punch from 0x01ed15 to 0x026c0e (0x7efa bytes) 000256 1787819246.651228 trunc from 0x03abba to 0x01c636 (0x1e583 bytes) 000257 1787819246.673009 read 0x017855 thru 0x01c636 (0x4de2 bytes) 000258 1787819246.687491 read 0x01a79d thru 0x01c636 (0x1e9a bytes) 000259 1787819246.688068 write 0x038143 thru 0x03ffff (0x7ebd bytes) 000260 1787819246.689335 read 0x01cf9d thru 0x01d4b3 (0x517 bytes) 000261 1787819246.703666 fallocate 0x01fa3e thru 0x020e04 (0x13c7 bytes) 000262 1787819246.708484 write 0x03eb41 thru 0x03ffff (0x14bf bytes) 000263 1787819246.709542 mapread 0x03eed5 thru 0x03ffff (0x112b bytes) 000264 1787819246.710804 mapread 0x0029f9 thru 0x00ee83 (0xc48b bytes) 000265 1787819246.713077 trunc from 0x040000 to 0x02033c (0x1fcc3 bytes) 000266 1787819246.728123 punch from 0x01547e to 0x01acfe (0x5881 bytes) 000267 1787819246.732477 trunc from 0x02033d to 0x014451 (0xbeeb bytes) 000268 1787819246.758790 fallocate 0x03318e thru 0x035839 (0x26ac bytes) 000269 1787819246.764760 mapread 0x008dcc thru 0x00f72a (0x695f bytes) 000270 1787819246.771799 MIRROR_RESYNC 5 mirrors 000271 1787819247.780445 MIRROR_EXTEND to 6 mirrors 000272 1787819248.676535 punch from 0x011315 to 0x01281f (0x150b bytes) 000273 1787819248.720185 trunc from 0x03583a to 0x026b7e (0xecbb bytes) 000274 1787819248.744238 write 0x00c803 thru 0x019d82 (0xd580 bytes) 000275 1787819248.756477 mapwrite 0x031a78 thru 0x031ea7 (0x430 bytes) 000276 1787819248.796599 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31ea8): Mirror 4: 0xa6dc2f20 (stale) Mirror 5: 0xa6dc2f20 (stale) Mirror 8: 0xa6dc2f20 (stale) Mirror 9: 0xa6dc2f20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31ea8): Mirror 4: 0xa6dc2f20 (stale) Mirror 5: 0xa6dc2f20 (stale) Mirror 8: 0xa6dc2f20 (stale) Mirror 9: 0xa6dc2f20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31ea8: skipped 000277 1787819249.114763 punch from 0x026c52 to 0x02c95d (0x5d0c bytes) 000278 1787819249.121679 read 0x024047 thru 0x031ea7 (0xde61 bytes) 000279 1787819249.131318 mapread 0x002360 thru 0x00f627 (0xd2c8 bytes) 000280 1787819249.147977 trunc from 0x02ff36 to 0x031ea7 (0x1f72 bytes) 000281 1787819249.152554 fallocate 0x006098 thru 0x00b33c (0x52a5 bytes) 000282 1787819249.157465 punch from 0x0031cb to 0x009c9b (0x6ad1 bytes) 000283 1787819249.161804 read 0x022ed4 thru 0x030892 (0xd9bf bytes) 000284 1787819249.175229 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31ea8): Mirror 4: 0xa6dc2f20 (stale) Mirror 8: 0xa6dc2f20 (stale) Mirror 9: 0xa6dc2f20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31ea8): Mirror 4: 0xa6dc2f20 (stale) Mirror 8: 0xa6dc2f20 (stale) Mirror 9: 0xa6dc2f20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31ea8: skipped 000285 1787819249.545488 write 0x03d3ac thru 0x03fc05 (0x285a bytes) 000287 1787819249.552368 mapwrite 0x003888 thru 0x0086fe (0x4e77 bytes) 000288 1787819249.584637 read 0x02a74d thru 0x034691 (0x9f45 bytes) 000289 1787819249.593726 trunc from 0x019829 to 0x01a102 (0x8da bytes) 000290 1787819249.600820 mapread 0x000f08 thru 0x00928c (0x8385 bytes) 000291 1787819249.606275 write 0x01c2eb thru 0x021137 (0x4e4d bytes) 000292 1787819249.626741 trunc from 0x01a653 to 0x01f105 (0x4ab3 bytes) 000293 1787819249.646818 read 0x033f73 thru 0x0375e5 (0x3673 bytes) 000294 1787819249.657407 mapread 0x011917 thru 0x017640 (0x5d2a bytes) 000295 1787819249.658773 trunc from 0x03fc06 to 0x008a4d (0x371b8 bytes) 000296 1787819249.680213 trunc from 0x008a4e to 0x005ff0 (0x2a5d bytes) 000298 1787819249.697008 trunc from 0x005ff1 to 0x008a97 (0x2aa7 bytes) 000299 1787819249.712544 write 0x03d4aa thru 0x03f09f (0x1bf6 bytes) 000300 1787819249.713731 trunc from 0x03f0a0 to 0x01ba3e (0x23661 bytes) 000301 1787819249.727305 read 0x00a2eb thru 0x00f284 (0x4f9a bytes) 000302 1787819249.736585 trunc from 0x01ba3f to 0x03139c (0x1595e bytes) 000303 1787819249.747786 trunc from 0x008c6f to 0x00bfbb (0x334d bytes) 000304 1787819249.754612 mapwrite 0x01c59d thru 0x026496 (0x9efa bytes) 000305 1787819249.799417 punch from 0x00949a to 0x009d4e (0x8b5 bytes) 000306 1787819249.807318 punch from 0x01c922 to 0x027b27 (0xb206 bytes) 000307 1787819249.816397 trunc from 0x03139d to 0x02a9b4 (0x69e8 bytes) 000308 1787819249.838189 read 0x0263e7 thru 0x02a9b4 (0x45ce bytes) 000309 1787819249.855334 read 0x01fdb3 thru 0x020463 (0x6b1 bytes) 000310 1787819249.856010 trunc from 0x02a9b5 to 0x011f07 (0x18aad bytes) 000312 1787819249.887897 mapwrite 0x03321c thru 0x0341f0 (0xfd5 bytes) 000314 1787819249.935491 trunc from 0x001a3e to 0x009404 (0x79c7 bytes) 000315 1787819249.943248 mapwrite 0x0048e4 thru 0x00e9b8 (0xa0d5 bytes) 000316 1787819249.996810 trunc from 0x0087d6 to 0x0160cb (0xd8f6 bytes) 000317 1787819250.002831 punch from 0x0056c7 to 0x00cd9d (0x76d7 bytes) 000318 1787819250.017460 fallocate 0x02a1a9 thru 0x02eb2d (0x4985 bytes) 000319 1787819250.023821 punch from 0x014baf to 0x02109c (0xc4ee bytes) 000320 1787819250.032348 write 0x02ff41 thru 0x03f430 (0xf4f0 bytes) 000321 1787819250.036067 mapwrite 0x00c5dc thru 0x0123cd (0x5df2 bytes) 000322 1787819250.114957 read 0x039c1d thru 0x03d55d (0x3941 bytes) 000323 1787819250.115666 trunc from 0x030b1b to 0x039b76 (0x905c bytes) 000324 1787819250.122520 mapwrite 0x027267 thru 0x02bae4 (0x487e bytes) 000327 1787819250.150077 trunc from 0x038293 to 0x03f430 (0x719e bytes) 000329 1787819250.165091 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f431): Mirror 4: 0xd14211c3 (stale) Mirror 9: 0xd14211c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f431): Mirror 4: 0xd14211c3 (stale) Mirror 9: 0xd14211c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f431: skipped 000330 1787819250.621455 punch from 0x02143a to 0x03040a (0xefd1 bytes) 000331 1787819250.633956 fallocate 0x02deec thru 0x0386c0 (0xa7d5 bytes) 000332 1787819250.646528 trunc from 0x03f431 to 0x012c4a (0x2c7e6 bytes) 000333 1787819250.668439 trunc from 0x012c4b to 0x0251a8 (0x1255e bytes) 000334 1787819250.690729 punch from 0x010c75 to 0x015867 (0x4bf3 bytes) 000335 1787819250.696555 mapwrite 0x03b21f thru 0x03ffff (0x4de1 bytes) 000336 1787819250.753507 mapwrite 0x03eb1a thru 0x03ffff (0x14e6 bytes) 000337 1787819250.766351 trunc from 0x0174d5 to 0x0262f7 (0xee23 bytes) 000340 1787819250.773395 punch from 0x01986d to 0x020c7c (0x7410 bytes) 000341 1787819250.778669 trunc from 0x035ad4 to 0x03ade7 (0x5314 bytes) 000342 1787819250.783706 fallocate 0x01a131 thru 0x028fe9 (0xeeb9 bytes) 000343 1787819250.788795 punch from 0x026d60 to 0x02a739 (0x39da bytes) 000344 1787819250.798836 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x5e1612f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x5e1612f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000346 1787819251.060329 fallocate 0x011f51 thru 0x01ff9c (0xe04c bytes) 000347 1787819251.069732 trunc from 0x008589 to 0x008856 (0x2ce bytes) 000348 1787819251.074001 trunc from 0x040000 to 0x00c3c0 (0x33c3f bytes) 000349 1787819251.105298 mapwrite 0x0123e2 thru 0x01c6fd (0xa31c bytes) 000350 1787819251.225656 read 0x00a222 thru 0x012fa4 (0x8d83 bytes) 000351 1787819251.233956 fallocate 0x01b81f thru 0x01e17a (0x295c bytes) 000352 1787819251.245314 trunc from 0x014d33 to 0x01c6fd (0x79cb bytes) 000353 1787819251.256755 trunc from 0x019f14 to 0x01a704 (0x7f1 bytes) 000354 1787819251.261484 trunc from 0x01c6fe to 0x01727e (0x547f bytes) 000355 1787819251.275461 mapwrite 0x029ad7 thru 0x02c41a (0x2944 bytes) 000356 1787819251.316924 punch from 0x024e74 to 0x02c41a (0x75a7 bytes) 000357 1787819251.327257 mapwrite 0x03e006 thru 0x03ffff (0x1ffa bytes) 000358 1787819251.364741 MIRROR_RESYNC 2 mirrors 000359 1787819252.302297 fallocate 0x02c176 thru 0x02c804 (0x68f bytes) 000361 1787819252.356971 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) 000362 1787819252.540759 write 0x029257 thru 0x02e1e2 (0x4f8c bytes) 000363 1787819252.559516 read 0x028c7c thru 0x030561 (0x78e6 bytes) 000364 1787819252.570477 trunc from 0x032f6b to 0x03ffff (0xd095 bytes) 000365 1787819252.588691 trunc from 0x040000 to 0x01f8b4 (0x2074b bytes) 000366 1787819252.640429 trunc from 0x003003 to 0x011026 (0xe024 bytes) 000368 1787819252.647449 mapread 0x0004ff thru 0x0059a7 (0x54a9 bytes) 000369 1787819252.696145 fallocate 0x036816 thru 0x03725e (0xa49 bytes) 000370 1787819252.706300 MIRROR_EXTEND to 2 mirrors 000371 1787819253.371726 trunc from 0x03725f to 0x01c618 (0x1ac46 bytes) 000372 1787819253.423964 punch from 0x00cc85 to 0x010cf0 (0x406c bytes) 000373 1787819253.442729 read 0x0099fb thru 0x0121c8 (0x87ce bytes) 000374 1787819253.477464 fallocate 0x01ebb5 thru 0x02d40e (0xe85a bytes) 000375 1787819253.493185 punch from 0x000eb1 to 0x00d423 (0xc573 bytes) 000376 1787819253.498462 trunc from 0x01375a to 0x01c618 (0x8ebf bytes) 000377 1787819253.509190 punch from 0x009c54 to 0x01965d (0xfa0a bytes) 000378 1787819253.513748 read 0x016d63 thru 0x018a10 (0x1cae bytes) 000379 1787819253.529616 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c619): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c619): Mirror 2: 0xd5c020e7 (stale) CRC-32 checksum value for chunk [0, 0x1c619): Mirror 1: 0xd0512c79 Mirror 3: 0xd0512c79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c619: skipped 000380 1787819254.126413 write 0x02c0b1 thru 0x02c382 (0x2d2 bytes) 000381 1787819254.172696 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c383): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c383): Mirror 2: 0x2fe0c4ab (stale) Mirror 3: 0x8ac81e4b (stale) CRC-32 checksum value for chunk [0, 0x2c383): Mirror 1: 0xc960f211 Mirror 4: 0xc960f211 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c383: skipped 000382 1787819255.144260 trunc from 0x005103 to 0x012e2f (0xdd2d bytes) 000383 1787819255.190112 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c383): Mirror 2: 0x2fe0c4ab (stale) Mirror 3: 0x8ac81e4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c383): Mirror 2: 0x2fe0c4ab (stale) Mirror 3: 0x8ac81e4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c383: skipped 000384 1787819255.530905 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c383): Mirror 3: 0x8ac81e4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c383): Mirror 3: 0x8ac81e4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c383: skipped 000386 1787819255.820272 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c383): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c383): Mirror 3: 0x8ac81e4b (stale) CRC-32 checksum value for chunk [0, 0x2c383): Mirror 1: 0xc960f211 Mirror 4: 0xc960f211 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c383: skipped 000387 1787819256.792383 trunc from 0x01225b to 0x01764b (0x53f1 bytes) 000388 1787819256.835264 write 0x0163c9 thru 0x01a73c (0x4374 bytes) 000389 1787819256.849078 mapread 0x012b10 thru 0x0135db (0xacc bytes) 000390 1787819256.859434 trunc from 0x00b747 to 0x00c42b (0xce5 bytes) 000391 1787819256.867516 read 0x01021f thru 0x016f1b (0x6cfd bytes) 000392 1787819256.876684 trunc from 0x02c383 to 0x02f5c1 (0x323f bytes) 000393 1787819256.896882 mapwrite 0x017d30 thru 0x022628 (0xa8f9 bytes) 000394 1787819256.928067 trunc from 0x02f5c2 to 0x0197f2 (0x15dcf bytes) 000395 1787819256.950569 punch from 0x00db6c to 0x010023 (0x24b8 bytes) 000396 1787819256.956304 trunc from 0x00a19e to 0x0160e9 (0xbf4c bytes) 000397 1787819256.961599 fallocate 0x014822 thru 0x014e44 (0x623 bytes) 000398 1787819256.967386 punch from 0x007498 to 0x012110 (0xac79 bytes) 000399 1787819256.973207 mapread 0x00e487 thru 0x00f67d (0x11f7 bytes) 000400 1787819256.985403 mapwrite 0x02c5bc thru 0x03647e (0x9ec3 bytes) 000402 1787819257.039384 punch from 0x007809 to 0x016068 (0xe860 bytes) 000403 1787819257.047576 mapread 0x00d6f7 thru 0x019a57 (0xc361 bytes) 000404 1787819257.073479 trunc from 0x03647f to 0x003168 (0x33316 bytes) 000405 1787819257.094264 punch from 0x002ab8 to 0x003168 (0x6b1 bytes) 000407 1787819257.099314 trunc from 0x001276 to 0x003168 (0x1ef3 bytes) 000408 1787819257.104411 punch from 0x002ff7 to 0x003168 (0x172 bytes) 000409 1787819257.109210 trunc from 0x001ea8 to 0x003168 (0x12c1 bytes) 000410 1787819257.114901 mapread 0x00245e thru 0x003168 (0xd0b bytes) 000411 1787819257.127070 MIRROR_RESYNC 3 mirrors 000413 1787819257.633468 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) 000414 1787819258.015047 punch from 0x000af5 to 0x003168 (0x2674 bytes) 000417 1787819258.085293 fallocate 0x031a30 thru 0x032b90 (0x1161 bytes) 000418 1787819258.107023 mapread 0x001a06 thru 0x003168 (0x1763 bytes) 000419 1787819258.143501 mapread 0x0020bc thru 0x003168 (0x10ad bytes) 000420 1787819258.144926 trunc from 0x002eb6 to 0x003168 (0x2b3 bytes) 000421 1787819258.150131 fallocate 0x034f11 thru 0x038746 (0x3836 bytes) 000422 1787819258.156273 read 0x036999 thru 0x038746 (0x1dae bytes) 000423 1787819258.162275 mapread 0x01e64c thru 0x02c473 (0xde28 bytes) 000424 1787819258.192050 write 0x01e553 thru 0x024132 (0x5be0 bytes) 000425 1787819258.201144 read 0x02b796 thru 0x02ef9c (0x3807 bytes) 000426 1787819258.201920 trunc from 0x038747 to 0x020d0a (0x17a3c bytes) 000427 1787819258.269647 fallocate 0x037b4a thru 0x03ffff (0x84b6 bytes) 000428 1787819258.278666 mapread 0x0027b9 thru 0x004fab (0x27f3 bytes) 000429 1787819258.312001 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20d0b): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20d0b): Mirror 4: 0x2e539e11 (stale) CRC-32 checksum value for chunk [0, 0x20d0b): Mirror 1: 0x1683232c Mirror 5: 0x1683232c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20d0b: skipped 000430 1787819258.970493 trunc from 0x00e7b6 to 0x01cf37 (0xe782 bytes) 000431 1787819259.047493 fallocate 0x02f345 thru 0x03e481 (0xf13d bytes) 000432 1787819259.064119 read 0x007568 thru 0x0158b4 (0xe34d bytes) 000433 1787819259.087834 fallocate 0x03dceb thru 0x03ffff (0x2315 bytes) 000434 1787819259.094168 trunc from 0x040000 to 0x022a1a (0x1d5e5 bytes) 000435 1787819259.143262 fallocate 0x017db4 thru 0x01ab76 (0x2dc3 bytes) 000437 1787819259.149524 mapwrite 0x03006b thru 0x031361 (0x12f7 bytes) 000438 1787819259.199128 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31362): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31362): Mirror 4: 0xbc987a39 (stale) Mirror 5: 0x64a1cc50 (stale) CRC-32 checksum value for chunk [0, 0x31362): Mirror 1: 0x76693f5f Mirror 6: 0x76693f5f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31362: skipped 000439 1787819260.053678 fallocate 0x00e971 thru 0x018627 (0x9cb7 bytes) 000440 1787819260.115209 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31362): Mirror 5: 0x64a1cc50 (stale) Mirror 6: 0x76693f5f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31362): Mirror 5: 0x64a1cc50 (stale) Mirror 6: 0x76693f5f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31362: skipped 000441 1787819260.462712 mapwrite 0x027522 thru 0x035ac6 (0xe5a5 bytes) 000442 1787819260.607287 mapwrite 0x00f1b3 thru 0x01ceac (0xdcfa bytes) 000443 1787819260.641025 mapwrite 0x03ec2d thru 0x03ffff (0x13d3 bytes) 000444 1787819260.687380 fallocate 0x014c52 thru 0x019faa (0x5359 bytes) 000446 1787819260.693207 read 0x02e4be thru 0x03d96b (0xf4ae bytes) 000447 1787819260.698256 trunc from 0x040000 to 0x029992 (0x1666d bytes) 000448 1787819260.718143 read 0x0173a3 thru 0x01d93b (0x6599 bytes) 000449 1787819260.718925 punch from 0x013104 to 0x02023e (0xd13b bytes) 000450 1787819260.726224 write 0x0207a8 thru 0x023b9e (0x33f7 bytes) 000453 1787819260.739441 mapwrite 0x00db5f thru 0x019973 (0xbe15 bytes) 000454 1787819260.771565 write 0x000932 thru 0x00855a (0x7c29 bytes) 000456 1787819260.781170 punch from 0x00718c to 0x00f03d (0x7eb2 bytes) 000457 1787819260.805473 mapread 0x020832 thru 0x029992 (0x9161 bytes) 000458 1787819260.806457 mapwrite 0x02f98a thru 0x039a4d (0xa0c4 bytes) 000459 1787819260.912088 mapread 0x029c22 thru 0x0305c9 (0x69a8 bytes) 000460 1787819260.914132 write 0x00ab16 thru 0x010912 (0x5dfd bytes) 000461 1787819260.919082 mapwrite 0x037398 thru 0x03ffff (0x8c68 bytes) 000462 1787819260.983554 write 0x008401 thru 0x00ee0b (0x6a0b bytes) 000464 1787819261.000416 punch from 0x03fff7 to 0x03ffff (0x9 bytes) 000465 1787819261.013524 read 0x01a369 thru 0x01d9df (0x3677 bytes) 000466 1787819261.014554 fallocate 0x007685 thru 0x0148ef (0xd26b bytes) 000467 1787819261.033114 MIRROR_RESYNC 3 mirrors 000468 1787819261.983290 fallocate 0x01f1e3 thru 0x02111a (0x1f38 bytes) 000469 1787819262.016356 mapwrite 0x022fc1 thru 0x032e16 (0xfe56 bytes) 000470 1787819262.064812 mapwrite 0x02ee86 thru 0x03e270 (0xf3eb bytes) 000471 1787819262.101311 mapwrite 0x020830 thru 0x02fd78 (0xf549 bytes) 000472 1787819262.127502 read 0x01312d thru 0x01ef29 (0xbdfd bytes) 000473 1787819262.136493 trunc from 0x040000 to 0x036309 (0x9cf6 bytes) 000474 1787819262.155046 fallocate 0x031061 thru 0x03a51e (0x94be bytes) 000476 1787819262.160812 trunc from 0x0189c4 to 0x024800 (0xbe3d bytes) 000477 1787819262.166460 mapwrite 0x00636d thru 0x0138bd (0xd551 bytes) 000478 1787819262.205356 read 0x016fe5 thru 0x025841 (0xe85d bytes) 000479 1787819262.220025 read 0x0111fa thru 0x0128de (0x16e5 bytes) 000480 1787819262.220588 read 0x00a9e5 thru 0x00aa4c (0x68 bytes) 000481 1787819262.221119 read 0x01f11c thru 0x024e4b (0x5d30 bytes) 000482 1787819262.221652 write 0x00a969 thru 0x0164df (0xbb77 bytes) 000483 1787819262.224223 mapwrite 0x0348ae thru 0x03630d (0x1a60 bytes) 000484 1787819262.245158 mapread 0x035868 thru 0x03a51e (0x4cb7 bytes) 000485 1787819262.246298 mapwrite 0x038215 thru 0x03d08c (0x4e78 bytes) 000486 1787819262.288288 write 0x000822 thru 0x009d70 (0x954f bytes) 000487 1787819262.290456 write 0x0076cf thru 0x00be41 (0x4773 bytes) 000488 1787819262.292633 mapread 0x006845 thru 0x00a284 (0x3a40 bytes) 000490 1787819262.295405 write 0x03345d thru 0x039d88 (0x692c bytes) 000491 1787819262.298104 fallocate 0x032fe5 thru 0x033df2 (0xe0e bytes) 000492 1787819262.303090 fallocate 0x01cdb1 thru 0x0210b9 (0x4309 bytes) 000493 1787819262.308239 trunc from 0x03d08d to 0x01e3a7 (0x1ece5 bytes) 000494 1787819262.334556 read 0x01daf6 thru 0x01e3a7 (0x8b2 bytes) 000495 1787819262.341010 MIRROR_RESYNC 3 mirrors 000496 1787819262.876117 write 0x002b1b thru 0x007bab (0x5091 bytes) 000497 1787819262.920095 trunc from 0x007333 to 0x00e068 (0x6d36 bytes) 000498 1787819262.936670 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1e3a8): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e3a8): Mirror 5: 0x2842c5b (stale) Mirror 6: 0x2842c5b (stale) CRC-32 checksum value for chunk [0, 0x1e3a8): Mirror 1: 0x906022de Mirror 7: 0x906022de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e3a8: skipped 000499 1787819263.714627 trunc from 0x01e3a8 to 0x01fb33 (0x178c bytes) 000500 1787819263.771400 trunc from 0x00705e to 0x00ec10 (0x7bb3 bytes) 000501 1787819263.778132 trunc from 0x01fb34 to 0x007a83 (0x180b0 bytes) 000502 1787819263.800212 write 0x036faa thru 0x03ffff (0x9056 bytes) 000503 1787819263.801652 mapread 0x00da6b thru 0x00f040 (0x15d6 bytes) 000504 1787819263.826373 trunc from 0x036f8b to 0x03ffff (0x9075 bytes) 000506 1787819263.850024 fallocate 0x0094dc thru 0x00dca1 (0x47c6 bytes) 000507 1787819263.862907 fallocate 0x01f516 thru 0x01f523 (0xe bytes) 000508 1787819263.886854 mapread 0x03f3e3 thru 0x03ffff (0xc1d bytes) 000509 1787819263.900675 mapread 0x01a33d thru 0x02038f (0x6053 bytes) 000510 1787819263.924519 punch from 0x024796 to 0x033dde (0xf649 bytes) 000511 1787819263.928670 mapwrite 0x0085e8 thru 0x00ce0e (0x4827 bytes) 000512 1787819263.966079 write 0x0313c9 thru 0x03ffff (0xec37 bytes) 000513 1787819263.977123 fallocate 0x02a84a thru 0x039cec (0xf4a3 bytes) 000514 1787819264.003324 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x95f703fb (stale) Mirror 7: 0xbfb33f03 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x95f703fb (stale) Mirror 7: 0xbfb33f03 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000515 1787819264.384482 trunc from 0x02fc92 to 0x03270b (0x2a7a bytes) 000516 1787819264.390989 trunc from 0x003bf9 to 0x011278 (0xd680 bytes) 000517 1787819264.395513 trunc from 0x029494 to 0x031e60 (0x89cd bytes) 000518 1787819264.400885 read 0x01c7e3 thru 0x02b425 (0xec43 bytes) 000519 1787819264.413574 mapwrite 0x039e74 thru 0x03ffff (0x618c bytes) 000520 1787819264.458825 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbfb33f03 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbfb33f03 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000521 1787819264.774444 trunc from 0x03262b to 0x038bf2 (0x65c8 bytes) 000523 1787819264.787033 read 0x025631 thru 0x02c2b7 (0x6c87 bytes) 000524 1787819264.800214 trunc from 0x040000 to 0x033a59 (0xc5a6 bytes) 000525 1787819264.824372 mapwrite 0x000017 thru 0x00b147 (0xb131 bytes) 000526 1787819264.866424 write 0x0023bd thru 0x00f722 (0xd366 bytes) 000527 1787819264.868104 read 0x00fbe3 thru 0x01b197 (0xb5b5 bytes) 000528 1787819264.880536 MIRROR_RESYNC 2 mirrors 000529 1787819265.532164 punch from 0x0318fb to 0x033a59 (0x215f bytes) 000530 1787819265.578117 mapwrite 0x00194c thru 0x008d87 (0x743c bytes) 000531 1787819265.651523 read 0x02d28e thru 0x033a59 (0x67cc bytes) 000532 1787819265.668151 trunc from 0x005919 to 0x00b44c (0x5b34 bytes) 000533 1787819265.676476 fallocate 0x014627 thru 0x020962 (0xc33c bytes) 000534 1787819265.681891 trunc from 0x0293ce to 0x02f495 (0x60c8 bytes) 000535 1787819265.687101 trunc from 0x033a5a to 0x034f3c (0x14e3 bytes) 000536 1787819265.718495 trunc from 0x02f085 to 0x034f3c (0x5eb8 bytes) 000537 1787819265.725135 fallocate 0x032f34 thru 0x036ea0 (0x3f6d bytes) 000538 1787819265.735931 read 0x01c28c thru 0x01f899 (0x360e bytes) 000539 1787819265.759323 read 0x028ef4 thru 0x034f3c (0xc049 bytes) 000540 1787819265.760152 write 0x010469 thru 0x017da6 (0x793e bytes) 000541 1787819265.763798 mapread 0x0334bf thru 0x034f3c (0x1a7e bytes) 000542 1787819265.765180 trunc from 0x034f3d to 0x0105dc (0x24960 bytes) 000543 1787819265.799547 trunc from 0x00d9be to 0x0105dc (0x2c1f bytes) 000545 1787819265.830305 punch from 0x0067c0 to 0x0073f3 (0xc34 bytes) 000546 1787819265.857343 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) 000550 1787819266.065439 mapwrite 0x000809 thru 0x0051ac (0x49a4 bytes) 000551 1787819266.143658 mapwrite 0x033de8 thru 0x03ffff (0xc218 bytes) 000552 1787819266.250295 fallocate 0x034c73 thru 0x03d70f (0x8a9d bytes) 000553 1787819266.274618 MIRROR_EXTEND to 2 mirrors 000554 1787819267.334301 trunc from 0x040000 to 0x02ddd3 (0x1222c bytes) 000555 1787819267.412445 read 0x027f83 thru 0x02b441 (0x34bf bytes) 000556 1787819267.421606 fallocate 0x001e30 thru 0x004b54 (0x2d25 bytes) 000557 1787819267.431788 read 0x01987c thru 0x02057e (0x6d03 bytes) 000558 1787819267.464373 trunc from 0x02ddd4 to 0x032d7f (0x4fac bytes) 000559 1787819267.492250 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) 000560 1787819267.668561 trunc from 0x032d80 to 0x03cb2b (0x9dac bytes) 000561 1787819267.690760 trunc from 0x03cb2c to 0x023b8b (0x18fa0 bytes) 000562 1787819267.707827 trunc from 0x023b8c to 0x023532 (0x659 bytes) 000563 1787819267.724554 trunc from 0x023533 to 0x01d2a7 (0x628b bytes) 000564 1787819267.746832 MIRROR_EXTEND to 2 mirrors 000565 1787819268.411200 trunc from 0x01d2a8 to 0x01e0f4 (0xe4d bytes) 000566 1787819268.446825 mapwrite 0x033ed7 thru 0x03a981 (0x6aab bytes) 000567 1787819268.498625 mapread 0x009bea thru 0x00cdc8 (0x31df bytes) 000568 1787819268.511631 trunc from 0x014b94 to 0x01b6d4 (0x6b41 bytes) 000569 1787819268.516319 mapread 0x01b4f5 thru 0x0211eb (0x5cf7 bytes) 000571 1787819268.529124 fallocate 0x01fb63 thru 0x02ab32 (0xafd0 bytes) 000573 1787819268.535554 mapread 0x023f01 thru 0x02d35d (0x945d bytes) 000574 1787819268.552399 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a982): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a982): Mirror 2: 0xfa5136b2 (stale) CRC-32 checksum value for chunk [0, 0x3a982): Mirror 1: 0x5edcbed8 Mirror 3: 0x5edcbed8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a982: skipped 000575 1787819269.480626 trunc from 0x005eed to 0x008890 (0x29a4 bytes) 000576 1787819269.536568 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a982): Mirror 3: 0x5edcbed8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a982): Mirror 3: 0x5edcbed8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a982: skipped 000577 1787819269.771638 trunc from 0x033ca3 to 0x03708a (0x33e8 bytes) 000578 1787819269.776448 read 0x016f66 thru 0x01cc1e (0x5cb9 bytes) 000579 1787819269.787844 write 0x00319d thru 0x00d1c3 (0xa027 bytes) 000580 1787819269.806824 mapread 0x01085b thru 0x017a2b (0x71d1 bytes) 000581 1787819269.823068 trunc from 0x006e66 to 0x0149c5 (0xdb60 bytes) 000582 1787819269.837650 mapread 0x0070be thru 0x00787d (0x7c0 bytes) 000583 1787819269.847652 read 0x031455 thru 0x033cb9 (0x2865 bytes) 000584 1787819269.859057 fallocate 0x026872 thru 0x029086 (0x2815 bytes) 000585 1787819269.863636 mapwrite 0x03c7d5 thru 0x03ffff (0x382b bytes) 000586 1787819269.937116 trunc from 0x040000 to 0x00c374 (0x33c8b bytes) 000587 1787819269.958350 punch from 0x00ad1e to 0x00c374 (0x1657 bytes) 000588 1787819269.964190 fallocate 0x019b4d thru 0x01b913 (0x1dc7 bytes) 000589 1787819269.969572 write 0x000e69 thru 0x00f7e6 (0xe97e bytes) 000590 1787819269.972129 mapread 0x00cf18 thru 0x00f7e6 (0x28cf bytes) 000591 1787819269.973442 read 0x00d96c thru 0x00f7e6 (0x1e7b bytes) 000592 1787819269.974410 read 0x003d84 thru 0x00f7e6 (0xba63 bytes) 000593 1787819269.975670 read 0x009da2 thru 0x00f7e6 (0x5a45 bytes) 000594 1787819269.976558 trunc from 0x00f7e7 to 0x03f6f3 (0x2ff0d bytes) 000595 1787819269.989679 read 0x02261a thru 0x02b79d (0x9184 bytes) 000596 1787819270.005698 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f6f4): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6f4): Mirror 3: 0x981390dc (stale) CRC-32 checksum value for chunk [0, 0x3f6f4): Mirror 1: 0x9a3ef67e Mirror 4: 0x9a3ef67e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6f4: skipped 000597 1787819270.943524 trunc from 0x00135d to 0x0057af (0x4453 bytes) 000598 1787819270.990228 mapwrite 0x03079c thru 0x035344 (0x4ba9 bytes) 000599 1787819271.029403 mapwrite 0x036849 thru 0x03ffff (0x97b7 bytes) 000600 1787819271.121095 read 0x008233 thru 0x017750 (0xf51e bytes) 000601 1787819271.131059 fallocate 0x03030d thru 0x034a5b (0x474f bytes) 000602 1787819271.136971 mapwrite 0x021fd3 thru 0x025594 (0x35c2 bytes) 000603 1787819271.177821 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8518ae16 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8518ae16 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000605 1787819271.516340 write 0x034e22 thru 0x03f8b2 (0xaa91 bytes) 000606 1787819271.535270 mapread 0x02add6 thru 0x032342 (0x756d bytes) 000607 1787819271.551272 punch from 0x03d1aa to 0x03ffff (0x2e56 bytes) 000608 1787819271.570759 mapwrite 0x03ccd2 thru 0x03ffff (0x332e bytes) 000609 1787819271.594533 fallocate 0x0035c3 thru 0x00a871 (0x72af bytes) 000610 1787819271.604641 trunc from 0x040000 to 0x00f1ad (0x30e52 bytes) 000612 1787819271.623485 trunc from 0x00f1ae to 0x036bc3 (0x27a16 bytes) 000614 1787819271.643432 trunc from 0x036bc4 to 0x00f1dc (0x279e7 bytes) 000616 1787819271.661580 trunc from 0x00f1dd to 0x0336ad (0x244d1 bytes) 000617 1787819271.678697 fallocate 0x016126 thru 0x01ec28 (0x8b03 bytes) 000618 1787819271.683375 trunc from 0x0336ae to 0x003726 (0x2ff87 bytes) 000619 1787819271.713443 punch from 0x00249e to 0x003726 (0x1289 bytes) 000621 1787819271.724413 read 0x0035e9 thru 0x003726 (0x13e bytes) 000622 1787819271.728734 read 0x002062 thru 0x003726 (0x16c5 bytes) 000623 1787819271.737679 trunc from 0x003727 to 0x03d6e3 (0x39fbd bytes) 000624 1787819271.769798 read 0x03a5a4 thru 0x03d6e3 (0x3140 bytes) 000625 1787819271.792216 read 0x03d5a3 thru 0x03d6e3 (0x141 bytes) 000626 1787819271.793066 write 0x033256 thru 0x03c833 (0x95de bytes) 000628 1787819271.796320 trunc from 0x03d6e4 to 0x0219db (0x1bd08 bytes) 000629 1787819271.843371 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x219dc): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x219dc): Mirror 4: 0xae18f100 (stale) CRC-32 checksum value for chunk [0, 0x219dc): Mirror 1: 0xe2d50b58 Mirror 5: 0xe2d50b58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x219dc: skipped 000630 1787819272.839799 read 0x0208ae thru 0x0219db (0x112e bytes) 000631 1787819272.855636 write 0x026bca thru 0x03044e (0x9885 bytes) 000632 1787819272.926659 mapwrite 0x03c222 thru 0x03ffff (0x3dde bytes) 000633 1787819273.053798 trunc from 0x040000 to 0x03990f (0x66f0 bytes) 000634 1787819273.076439 trunc from 0x02ff00 to 0x038f20 (0x9021 bytes) 000635 1787819273.087411 read 0x022383 thru 0x02ed07 (0xc985 bytes) 000637 1787819273.110470 punch from 0x00350e to 0x00fa5b (0xc54e bytes) 000638 1787819273.121552 mapwrite 0x00966d thru 0x017d66 (0xe6fa bytes) 000639 1787819273.270239 trunc from 0x0390e7 to 0x03990f (0x829 bytes) 000640 1787819273.276528 write 0x00702f thru 0x00cfb8 (0x5f8a bytes) 000641 1787819273.282837 read 0x013ffa thru 0x0157e9 (0x17f0 bytes) 000642 1787819273.283752 read 0x02b4e2 thru 0x02c4d9 (0xff8 bytes) 000643 1787819273.284546 mapread 0x02ca90 thru 0x0384f8 (0xba69 bytes) 000644 1787819273.286420 mapread 0x02f448 thru 0x03990f (0xa4c8 bytes) 000645 1787819273.306791 fallocate 0x023556 thru 0x0243d5 (0xe80 bytes) 000646 1787819273.321647 write 0x0033bb thru 0x011e02 (0xea48 bytes) 000647 1787819273.329442 trunc from 0x039910 to 0x0111bf (0x28750 bytes) 000648 1787819273.385819 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x111c0): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x111c0): Mirror 4: 0x23989f6 (stale) Mirror 5: 0x5def04ce (stale) CRC-32 checksum value for chunk [0, 0x111c0): Mirror 1: 0xff6a920f Mirror 6: 0xff6a920f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x111c0: skipped 000649 1787819274.227310 write 0x0342e8 thru 0x03c432 (0x814b bytes) 000650 1787819274.278333 MIRROR_RESYNC 4 mirrors 000651 1787819275.416954 trunc from 0x03c433 to 0x005ed0 (0x36562 bytes) 000652 1787819275.461756 punch from 0x002aec to 0x005ed0 (0x33e5 bytes) 000653 1787819275.468675 punch from 0x003326 to 0x005ed0 (0x2bab bytes) 000654 1787819275.474819 fallocate 0x006778 thru 0x008b7b (0x2404 bytes) 000655 1787819275.480459 mapread 0x0014a4 thru 0x008b7b (0x76d8 bytes) 000656 1787819275.503953 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x8b7c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8b7c): Mirror 4: 0x8f9c042a (stale) Mirror 5: 0x8f9c042a (stale) Mirror 6: 0x8f9c042a (stale) CRC-32 checksum value for chunk [0, 0x8b7c): Mirror 1: 0xb9aedb4 Mirror 7: 0xb9aedb4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8b7c: skipped 000657 1787819276.448904 mapwrite 0x026a80 thru 0x02c6b9 (0x5c3a bytes) 000658 1787819276.665994 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2c6ba): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c6ba): Mirror 4: 0xd0c73a6 (stale) Mirror 5: 0xd0c73a6 (stale) Mirror 6: 0xd0c73a6 (stale) Mirror 7: 0xeadd0a44 (stale) CRC-32 checksum value for chunk [0, 0x2c6ba): Mirror 1: 0x544df9e8 Mirror 8: 0x544df9e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c6ba: skipped 000659 1787819277.895184 fallocate 0x01749f thru 0x025250 (0xddb2 bytes) 000660 1787819277.928499 trunc from 0x023b48 to 0x02c6b9 (0x8b72 bytes) 000661 1787819277.934195 read 0x00d183 thru 0x013b2e (0x69ac bytes) 000662 1787819277.942691 punch from 0x009ceb to 0x01404f (0xa365 bytes) 000663 1787819277.947111 trunc from 0x00074c to 0x005509 (0x4dbe bytes) 000664 1787819277.951778 punch from 0x005e3c to 0x01000e (0xa1d3 bytes) 000665 1787819277.956524 trunc from 0x01cc9d to 0x02b963 (0xecc7 bytes) 000666 1787819277.960831 punch from 0x022cf1 to 0x027055 (0x4365 bytes) 000667 1787819277.965278 mapread 0x023524 thru 0x028868 (0x5345 bytes) 000668 1787819277.983286 write 0x00e990 thru 0x0133ec (0x4a5d bytes) 000669 1787819277.995260 trunc from 0x02c6ba to 0x02ab1b (0x1b9e bytes) 000670 1787819278.018551 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ab1c): Mirror 4: 0x62458eaf (stale) Mirror 6: 0x62458eaf (stale) Mirror 7: 0x20398873 (stale) Mirror 8: 0x2b91497c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ab1c): Mirror 4: 0x62458eaf (stale) Mirror 6: 0x62458eaf (stale) Mirror 7: 0x20398873 (stale) Mirror 8: 0x2b91497c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ab1c: skipped 000671 1787819278.346402 punch from 0x009e49 to 0x0144e8 (0xa6a0 bytes) 000672 1787819278.353605 trunc from 0x02ab1c to 0x00b084 (0x1fa97 bytes) 000673 1787819278.376038 mapwrite 0x00530e thru 0x007103 (0x1df6 bytes) 000674 1787819278.406233 trunc from 0x00b085 to 0x019c40 (0xebbc bytes) 000675 1787819278.422497 read 0x01860f thru 0x019c40 (0x1632 bytes) 000676 1787819278.428783 mapwrite 0x02c48e thru 0x038849 (0xc3bc bytes) 000677 1787819278.476338 read 0x02ddd3 thru 0x033403 (0x5631 bytes) 000678 1787819278.476964 punch from 0x02a7d8 to 0x0327eb (0x8014 bytes) 000679 1787819278.482264 read 0x027e12 thru 0x033641 (0xb830 bytes) 000681 1787819278.497523 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3884a): Mirror 4: 0xf0f0668b (stale) Mirror 6: 0xf0f0668b (stale) Mirror 8: 0x62436395 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3884a): Mirror 4: 0xf0f0668b (stale) Mirror 6: 0xf0f0668b (stale) Mirror 8: 0x62436395 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3884a: skipped 000682 1787819278.904152 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3884a): Mirror 6: 0xf0f0668b (stale) Mirror 8: 0x62436395 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3884a): Mirror 6: 0xf0f0668b (stale) Mirror 8: 0x62436395 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3884a: skipped 000683 1787819279.259422 MIRROR_RESYNC 3 mirrors 000684 1787819280.210233 MIRROR_RESYNC 3 mirrors 000685 1787819280.712497 fallocate 0x039110 thru 0x03e7ab (0x569c bytes) 000686 1787819280.785111 read 0x00209c thru 0x00bea7 (0x9e0c bytes) 000687 1787819280.818233 read 0x01d508 thru 0x029cbd (0xc7b6 bytes) 000688 1787819280.864015 mapread 0x0267e1 thru 0x02ac89 (0x44a9 bytes) 000689 1787819280.866016 trunc from 0x03e7ac to 0x028c54 (0x15b57 bytes) 000690 1787819280.922636 write 0x0354c7 thru 0x03ffff (0xab39 bytes) 000691 1787819280.929427 mapread 0x036320 thru 0x03b2b3 (0x4f94 bytes) 000692 1787819280.930462 punch from 0x023245 to 0x028061 (0x4e1d bytes) 000693 1787819280.940388 trunc from 0x040000 to 0x00cc77 (0x33388 bytes) 000694 1787819280.986634 write 0x025f38 thru 0x02b7a8 (0x5871 bytes) 000695 1787819280.988181 trunc from 0x02b7a9 to 0x028e40 (0x2968 bytes) 000696 1787819281.025274 fallocate 0x02ac5f thru 0x032e86 (0x8228 bytes) 000698 1787819281.040332 fallocate 0x03dd44 thru 0x03ffff (0x22bc bytes) 000699 1787819281.053965 trunc from 0x040000 to 0x0117e3 (0x2e81c bytes) 000700 1787819281.093823 write 0x004716 thru 0x00f042 (0xa92d bytes) 000701 1787819281.105801 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x117e4): Mirror 8: 0xeb1eeaca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x117e4): Mirror 8: 0xeb1eeaca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x117e4: skipped 000702 1787819281.457303 fallocate 0x00e3f0 thru 0x00f619 (0x122a bytes) 000703 1787819281.464404 read 0x00c188 thru 0x0117e3 (0x565c bytes) 000705 1787819281.479718 fallocate 0x02959b thru 0x038fdd (0xfa43 bytes) 000706 1787819281.500062 trunc from 0x00336e to 0x012739 (0xf3cc bytes) 000707 1787819281.507428 mapread 0x019413 thru 0x01e0bd (0x4cab bytes) 000708 1787819281.543410 mapread 0x01f644 thru 0x026e8e (0x784b bytes) 000709 1787819281.552896 mapwrite 0x01b869 thru 0x02a8bb (0xf053 bytes) 000710 1787819281.584382 mapread 0x00e5cd thru 0x015622 (0x7056 bytes) 000713 1787819281.600443 write 0x0380a7 thru 0x03ffff (0x7f59 bytes) 000714 1787819281.613654 punch from 0x01559f to 0x025196 (0xfbf8 bytes) 000715 1787819281.623547 punch from 0x0226e4 to 0x024cb0 (0x25cd bytes) 000716 1787819281.628577 mapread 0x02fc21 thru 0x032be5 (0x2fc5 bytes) 000718 1787819281.639518 mapread 0x03f549 thru 0x03ffff (0xab7 bytes) 000719 1787819281.640976 mapwrite 0x03ce58 thru 0x03ffff (0x31a8 bytes) 000720 1787819281.666354 fallocate 0x0322c9 thru 0x03e33c (0xc074 bytes) 000721 1787819281.676069 write 0x028a22 thru 0x032c59 (0xa238 bytes) 000722 1787819281.679344 trunc from 0x040000 to 0x0111e3 (0x2ee1c bytes) 000723 1787819281.714353 punch from 0x001367 to 0x009134 (0x7dce bytes) 000724 1787819281.720755 trunc from 0x0111e4 to 0x015643 (0x4460 bytes) 000725 1787819281.744916 trunc from 0x015644 to 0x03fcfc (0x2a6b9 bytes) 000726 1787819281.764308 mapread 0x028036 thru 0x02bd6e (0x3d39 bytes) 000727 1787819281.779942 fallocate 0x03dcda thru 0x03f303 (0x162a bytes) 000728 1787819281.786947 mapwrite 0x003a94 thru 0x010d0f (0xd27c bytes) 000729 1787819281.843218 mapread 0x028d9e thru 0x029568 (0x7cb bytes) 000731 1787819281.844590 mapwrite 0x002d8d thru 0x003bae (0xe22 bytes) 000732 1787819281.879596 write 0x00b366 thru 0x017a1c (0xc6b7 bytes) 000733 1787819281.887808 punch from 0x008f8c to 0x01867a (0xf6ef bytes) 000734 1787819281.908510 mapwrite 0x01d9c9 thru 0x02c0aa (0xe6e2 bytes) 000735 1787819281.956336 write 0x021543 thru 0x02bbc3 (0xa681 bytes) 000736 1787819281.958978 trunc from 0x008cbc to 0x010c31 (0x7f76 bytes) 000737 1787819281.964033 trunc from 0x03fcfd to 0x03622e (0x9ace bytes) 000738 1787819281.982791 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) 000739 1787819282.164735 mapwrite 0x0122eb thru 0x01cde4 (0xaafa bytes) 000740 1787819282.233265 write 0x02c1e6 thru 0x036905 (0xa720 bytes) 000741 1787819282.249539 trunc from 0x036906 to 0x024519 (0x123ec bytes) 000742 1787819282.269061 write 0x02d708 thru 0x038e41 (0xb73a bytes) 000744 1787819282.272539 trunc from 0x002f5a to 0x009a79 (0x6b20 bytes) 000745 1787819282.279395 trunc from 0x0019b1 to 0x003469 (0x1ab9 bytes) 000746 1787819282.283613 punch from 0x01a721 to 0x02982f (0xf10f bytes) 000749 1787819282.290508 mapread 0x01c5b0 thru 0x01eb8c (0x25dd bytes) 000750 1787819282.298573 trunc from 0x01f4ab to 0x021d5d (0x28b3 bytes) 000752 1787819282.303585 write 0x02b1f6 thru 0x02d4b5 (0x22c0 bytes) 000753 1787819282.310000 fallocate 0x020f19 thru 0x02bd01 (0xade9 bytes) 000754 1787819282.315068 trunc from 0x035703 to 0x0378f7 (0x21f5 bytes) 000755 1787819282.328158 trunc from 0x000f9d to 0x006ac2 (0x5b26 bytes) 000756 1787819282.332987 trunc from 0x032120 to 0x038183 (0x6064 bytes) 000758 1787819282.338948 read 0x005034 thru 0x00c3a9 (0x7376 bytes) 000759 1787819282.351435 trunc from 0x038e42 to 0x008de5 (0x3005c bytes) 000760 1787819282.374482 mapread 0x008bc2 thru 0x008de5 (0x224 bytes) 000761 1787819282.376293 write 0x01f8a8 thru 0x02b47d (0xbbd6 bytes) 000762 1787819282.382741 read 0x02098b thru 0x02b47d (0xaaf3 bytes) 000764 1787819282.390838 trunc from 0x02b47e to 0x036536 (0xb0b9 bytes) 000766 1787819282.421438 trunc from 0x01a19c to 0x029c60 (0xfac5 bytes) 000767 1787819282.450541 punch from 0x02f030 to 0x036536 (0x7507 bytes) 000768 1787819282.461177 mapread 0x01d0f5 thru 0x021cfa (0x4c06 bytes) 000769 1787819282.488529 read 0x003224 thru 0x00ac6b (0x7a48 bytes) 000771 1787819282.489781 read 0x02a0fa thru 0x02a317 (0x21e bytes) 000772 1787819282.490744 mapread 0x014ee1 thru 0x01e99e (0x9abe bytes) 000773 1787819282.492314 mapwrite 0x00ccde thru 0x00d6bb (0x9de bytes) 000774 1787819282.515655 trunc from 0x036537 to 0x000f73 (0x355c3 bytes) 000775 1787819282.547602 read 0x000d23 thru 0x000f73 (0x251 bytes) 000776 1787819282.548381 trunc from 0x000634 to 0x000f73 (0x940 bytes) 000778 1787819282.563398 punch from 0x0000f9 to 0x000f73 (0xe7b bytes) 000779 1787819282.571770 mapwrite 0x0228e8 thru 0x02f2b3 (0xc9cc bytes) 000780 1787819282.669353 mapread 0x00f414 thru 0x01e31d (0xef0a bytes) 000781 1787819282.670490 mapread 0x027b1d thru 0x02801a (0x4fe bytes) 000782 1787819282.673854 trunc from 0x02f2b4 to 0x02b420 (0x3e93 bytes) 000783 1787819282.693766 trunc from 0x02b421 to 0x002431 (0x28fef bytes) 000784 1787819282.714129 mapwrite 0x03f864 thru 0x03ffff (0x79c bytes) 000785 1787819282.788369 mapwrite 0x007f44 thru 0x009e77 (0x1f34 bytes) 000786 1787819282.816802 write 0x03fcef thru 0x03ffff (0x311 bytes) 000787 1787819282.818023 read 0x029acb thru 0x0305fe (0x6b34 bytes) 000788 1787819282.827764 mapwrite 0x01a871 thru 0x026e6a (0xc5fa bytes) 000789 1787819282.847262 write 0x003dce thru 0x00ef76 (0xb1a9 bytes) 000790 1787819282.849067 fallocate 0x01d2f2 thru 0x01e383 (0x1092 bytes) 000791 1787819282.853457 punch from 0x03b90e to 0x03ffff (0x46f2 bytes) 000792 1787819282.858109 punch from 0x00c4c1 to 0x00f6c3 (0x3203 bytes) 000793 1787819282.872012 fallocate 0x039252 thru 0x03ffff (0x6dae bytes) 000794 1787819282.878600 write 0x016ad9 thru 0x018c22 (0x214a bytes) 000795 1787819282.879766 read 0x006987 thru 0x00d348 (0x69c2 bytes) 000797 1787819282.895475 trunc from 0x034a8b to 0x039703 (0x4c79 bytes) 000798 1787819282.901424 punch from 0x02e328 to 0x0382c0 (0x9f99 bytes) 000800 1787819282.915224 read 0x031c93 thru 0x03fb71 (0xdedf bytes) 000801 1787819282.930794 punch from 0x02fa6e to 0x039b0f (0xa0a2 bytes) 000802 1787819282.936725 mapread 0x00b704 thru 0x017e2a (0xc727 bytes) 000803 1787819282.942546 MIRROR_EXTEND to 2 mirrors 000805 1787819283.881694 mapwrite 0x013299 thru 0x01ffdb (0xcd43 bytes) 000807 1787819284.026195 punch from 0x010253 to 0x01f10a (0xeeb8 bytes) 000808 1787819284.031534 trunc from 0x040000 to 0x029120 (0x16edf bytes) 000809 1787819284.057458 trunc from 0x022aa6 to 0x0264d2 (0x3a2d bytes) 000810 1787819284.063285 write 0x00b96e thru 0x011101 (0x5794 bytes) 000811 1787819284.073433 punch from 0x010a31 to 0x0119bf (0xf8f bytes) 000812 1787819284.092237 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x29121): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29121): Mirror 2: 0xb4d801e9 (stale) CRC-32 checksum value for chunk [0, 0x29121): Mirror 1: 0x9789113 Mirror 3: 0x9789113 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29121: skipped 000813 1787819284.883650 mapread 0x0127e6 thru 0x014b6f (0x238a bytes) 000814 1787819284.980305 punch from 0x027755 to 0x029120 (0x19cc bytes) 000815 1787819285.015390 mapwrite 0x037d07 thru 0x038a20 (0xd1a bytes) 000816 1787819285.066196 write 0x008c26 thru 0x00a2ac (0x1687 bytes) 000817 1787819285.103571 write 0x03f0c9 thru 0x03ffff (0xf37 bytes) 000818 1787819285.105462 punch from 0x006757 to 0x009233 (0x2add bytes) 000819 1787819285.138809 read 0x02fb85 thru 0x03866e (0x8aea bytes) 000820 1787819285.146031 trunc from 0x00f225 to 0x018efc (0x9cd8 bytes) 000821 1787819285.156493 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdb442302 (stale) Mirror 3: 0xb0f6f3be (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeeb19ea Mirror 4: 0xeeb19ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000822 1787819286.483912 punch from 0x03753d to 0x03859c (0x1060 bytes) 000824 1787819286.542452 read 0x0160b3 thru 0x01c1f0 (0x613e bytes) 000825 1787819286.556516 write 0x006a39 thru 0x00f1cf (0x8797 bytes) 000826 1787819286.576149 trunc from 0x040000 to 0x0187d8 (0x27827 bytes) 000827 1787819286.643435 fallocate 0x03b088 thru 0x03ffff (0x4f78 bytes) 000828 1787819286.649580 mapwrite 0x03766f thru 0x03ffff (0x8991 bytes) 000830 1787819286.742471 punch from 0x031b6e to 0x03ffff (0xe492 bytes) 000831 1787819286.753050 write 0x021239 thru 0x025839 (0x4601 bytes) 000832 1787819286.764157 trunc from 0x03c49c to 0x03f89f (0x3404 bytes) 000833 1787819286.773563 fallocate 0x012fe7 thru 0x01552f (0x2549 bytes) 000834 1787819286.780515 mapwrite 0x03fce2 thru 0x03ffff (0x31e bytes) 000835 1787819286.863823 MIRROR_RESYNC 4 mirrors 000837 1787819288.310467 read 0x031e8b thru 0x03b56b (0x96e1 bytes) 000839 1787819288.320750 write 0x01c22e thru 0x026741 (0xa514 bytes) 000840 1787819288.394737 MIRROR_RESYNC 4 mirrors 000841 1787819289.657353 mapwrite 0x03c438 thru 0x03ffff (0x3bc8 bytes) 000842 1787819289.790133 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdcd59bea (stale) Mirror 4: 0xdcd59bea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdcd59bea (stale) Mirror 4: 0xdcd59bea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000843 1787819290.226359 mapwrite 0x009fe6 thru 0x01807c (0xe097 bytes) 000844 1787819290.286680 read 0x00c3e0 thru 0x01b741 (0xf362 bytes) 000845 1787819290.287589 mapread 0x03d719 thru 0x03ffff (0x28e7 bytes) 000846 1787819290.302144 mapwrite 0x020865 thru 0x024073 (0x380f bytes) 000847 1787819290.331767 read 0x030c1f thru 0x03aca7 (0xa089 bytes) 000848 1787819290.344568 read 0x02850b thru 0x02a866 (0x235c bytes) 000849 1787819290.345487 trunc from 0x03e6e9 to 0x03ffff (0x1917 bytes) 000850 1787819290.352444 write 0x0236aa thru 0x02733b (0x3c92 bytes) 000851 1787819290.369033 MIRROR_RESYNC 3 mirrors 000852 1787819291.590430 fallocate 0x02ab39 thru 0x036a26 (0xbeee bytes) 000853 1787819291.624393 punch from 0x021c10 to 0x02d3bc (0xb7ad bytes) 000854 1787819291.630547 trunc from 0x040000 to 0x03c52c (0x3ad3 bytes) 000855 1787819291.652853 write 0x033dea thru 0x039bf0 (0x5e07 bytes) 000856 1787819291.665543 punch from 0x02dbb8 to 0x038a70 (0xaeb9 bytes) 000858 1787819291.692630 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c52d): Mirror 4: 0xc686835f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c52d): Mirror 4: 0xc686835f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c52d: skipped 000859 1787819292.085015 read 0x01ad9f thru 0x0236ac (0x890e bytes) 000860 1787819292.122516 write 0x02d328 thru 0x03a6aa (0xd383 bytes) 000861 1787819292.138027 mapread 0x0173cb thru 0x01d080 (0x5cb6 bytes) 000862 1787819292.161158 fallocate 0x0303ef thru 0x0392dd (0x8eef bytes) 000863 1787819292.167051 write 0x01b392 thru 0x02ad90 (0xf9ff bytes) 000864 1787819292.174279 mapwrite 0x002e4d thru 0x00fd51 (0xcf05 bytes) 000865 1787819292.221351 punch from 0x0024e3 to 0x003c9d (0x17bb bytes) 000867 1787819292.229413 punch from 0x00d865 to 0x00d8ba (0x56 bytes) 000868 1787819292.235651 trunc from 0x00c294 to 0x00ca91 (0x7fe bytes) 000870 1787819292.243242 write 0x02a9c6 thru 0x02c85c (0x1e97 bytes) 000872 1787819292.245118 trunc from 0x0223f6 to 0x02d65b (0xb266 bytes) 000873 1787819292.255692 punch from 0x020ee1 to 0x025bff (0x4d1f bytes) 000874 1787819292.261617 read 0x030735 thru 0x03698f (0x625b bytes) 000875 1787819292.263505 read 0x01ed64 thru 0x02ec29 (0xfec6 bytes) 000876 1787819292.294392 MIRROR_RESYNC 2 mirrors 000877 1787819293.162374 mapwrite 0x03e94d thru 0x03ffff (0x16b3 bytes) 000878 1787819293.282625 read 0x0197d8 thru 0x01b433 (0x1c5c bytes) 000879 1787819293.293049 mapread 0x0293cc thru 0x02dbaf (0x47e4 bytes) 000880 1787819293.305328 trunc from 0x017b28 to 0x01ac74 (0x314d bytes) 000881 1787819293.314478 mapread 0x002747 thru 0x00fe4a (0xd704 bytes) 000882 1787819293.331604 trunc from 0x032090 to 0x03b6bf (0x9630 bytes) 000883 1787819293.344678 MIRROR_RESYNC 2 mirrors 000884 1787819294.451878 read 0x00d1cd thru 0x00fcf7 (0x2b2b bytes) 000886 1787819294.459196 trunc from 0x040000 to 0x0303f3 (0xfc0c bytes) 000887 1787819294.506815 read 0x02d28d thru 0x02e3c2 (0x1136 bytes) 000890 1787819294.523662 mapread 0x01bbf2 thru 0x01d6b6 (0x1ac5 bytes) 000891 1787819294.525234 mapwrite 0x016481 thru 0x023082 (0xcc02 bytes) 000892 1787819294.568262 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) 000893 1787819294.793277 trunc from 0x027113 to 0x02d43b (0x6329 bytes) 000894 1787819294.800472 mapwrite 0x033182 thru 0x038047 (0x4ec6 bytes) 000895 1787819294.949443 punch from 0x01ae88 to 0x01ddf6 (0x2f6f bytes) 000896 1787819294.954826 write 0x0104ac thru 0x014f72 (0x4ac7 bytes) 000897 1787819294.983532 punch from 0x00a211 to 0x0166b3 (0xc4a3 bytes) 000898 1787819295.012565 mapwrite 0x035f0e thru 0x03b419 (0x550c bytes) 000899 1787819295.085392 write 0x03b142 thru 0x03becb (0xd8a bytes) 000900 1787819295.091204 fallocate 0x00fac3 thru 0x01c260 (0xc79e bytes) 000901 1787819295.095953 read 0x0245de thru 0x02a95d (0x6380 bytes) 000902 1787819295.114207 MIRROR_EXTEND to 2 mirrors 000903 1787819296.289307 trunc from 0x03becc to 0x00b68c (0x3083f bytes) 000905 1787819296.373785 trunc from 0x00b68d to 0x02b27c (0x1fbf0 bytes) 000906 1787819296.399226 write 0x01866b thru 0x01b846 (0x31dc bytes) 000907 1787819296.411979 fallocate 0x01f9c6 thru 0x028b88 (0x91c3 bytes) 000908 1787819296.417061 mapwrite 0x01a852 thru 0x024732 (0x9ee1 bytes) 000909 1787819296.495588 trunc from 0x02b27d to 0x039364 (0xe0e8 bytes) 000910 1787819296.532349 mapread 0x03028c thru 0x03196d (0x16e2 bytes) 000911 1787819296.568049 trunc from 0x039365 to 0x01fc94 (0x196d0 bytes) 000912 1787819296.594527 read 0x01e7ee thru 0x01fc94 (0x14a7 bytes) 000913 1787819296.598389 mapread 0x008a24 thru 0x00ba8b (0x3068 bytes) 000914 1787819296.634731 fallocate 0x018ca9 thru 0x0219bc (0x8d14 bytes) 000915 1787819296.643637 fallocate 0x004227 thru 0x00f7cf (0xb5a9 bytes) 000916 1787819296.663251 read 0x00689f thru 0x00ee26 (0x8588 bytes) 000917 1787819296.670577 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) 000918 1787819296.919825 trunc from 0x0219bd to 0x021748 (0x274 bytes) 000919 1787819296.942742 mapwrite 0x032f18 thru 0x03ba8b (0x8b74 bytes) 000922 1787819297.021003 write 0x0268af thru 0x02f8e9 (0x903b bytes) 000923 1787819297.034951 fallocate 0x0133ae thru 0x02112d (0xdd80 bytes) 000924 1787819297.043689 write 0x0053e9 thru 0x0092fd (0x3f15 bytes) 000925 1787819297.058720 mapwrite 0x003dc8 thru 0x012f77 (0xf1b0 bytes) 000926 1787819297.125054 write 0x01e851 thru 0x024848 (0x5ff8 bytes) 000927 1787819297.138474 write 0x02e2cc thru 0x03bea7 (0xdbdc bytes) 000929 1787819297.142020 read 0x002272 thru 0x00d008 (0xad97 bytes) 000930 1787819297.143372 trunc from 0x03bea8 to 0x03c0e3 (0x23c bytes) 000931 1787819297.177512 trunc from 0x03c0e4 to 0x024f86 (0x1715d bytes) 000934 1787819297.204596 fallocate 0x02c566 thru 0x02d6af (0x114a bytes) 000935 1787819297.215388 write 0x00c634 thru 0x014587 (0x7f54 bytes) 000937 1787819297.229814 write 0x03723e thru 0x03ffff (0x8dc2 bytes) 000938 1787819297.236172 mapwrite 0x033768 thru 0x035d35 (0x25ce bytes) 000939 1787819297.284690 mapread 0x024a0d thru 0x033d75 (0xf369 bytes) 000940 1787819297.299310 fallocate 0x00cf3b thru 0x012b32 (0x5bf8 bytes) 000941 1787819297.306103 mapwrite 0x01fe56 thru 0x025ece (0x6079 bytes) 000942 1787819297.329493 trunc from 0x025d52 to 0x02a24e (0x44fd bytes) 000943 1787819297.335394 mapread 0x035f04 thru 0x03f1c9 (0x92c6 bytes) 000944 1787819297.336495 write 0x00dde0 thru 0x00eb16 (0xd37 bytes) 000946 1787819297.338814 trunc from 0x0214ca to 0x02d10b (0xbc42 bytes) 000947 1787819297.345687 punch from 0x0358f9 to 0x03ffff (0xa707 bytes) 000948 1787819297.351123 mapwrite 0x02dc9b thru 0x032406 (0x476c bytes) 000949 1787819297.376303 mapread 0x0027fa thru 0x004a74 (0x227b bytes) 000950 1787819297.377209 read 0x01a07f thru 0x024c13 (0xab95 bytes) 000951 1787819297.403344 write 0x022ef0 thru 0x02eacc (0xbbdd bytes) 000952 1787819297.408030 mapread 0x03ec82 thru 0x03ffff (0x137e bytes) 000953 1787819297.409453 mapwrite 0x022c81 thru 0x02d165 (0xa4e5 bytes) 000954 1787819297.435258 write 0x003f0e thru 0x0116d9 (0xd7cc bytes) 000955 1787819297.436840 mapread 0x0036cc thru 0x00a09c (0x69d1 bytes) 000958 1787819297.440724 punch from 0x039be1 to 0x03ffff (0x641f bytes) 000959 1787819297.448158 write 0x008b08 thru 0x00f84f (0x6d48 bytes) 000961 1787819297.453779 mapread 0x00ed13 thru 0x01b0a5 (0xc393 bytes) 000962 1787819297.455415 mapread 0x01a314 thru 0x02772a (0xd417 bytes) 000963 1787819297.457478 write 0x030893 thru 0x0344d2 (0x3c40 bytes) 000964 1787819297.459267 mapwrite 0x00c266 thru 0x015412 (0x91ad bytes) 000965 1787819297.502014 punch from 0x000773 to 0x004be4 (0x4472 bytes) 000966 1787819297.515356 read 0x009094 thru 0x00f227 (0x6194 bytes) 000967 1787819297.522320 write 0x02c6bb thru 0x034b00 (0x8446 bytes) 000968 1787819297.528634 trunc from 0x005f8c to 0x0094ec (0x3561 bytes) 000969 1787819297.539330 mapread 0x03e7e0 thru 0x03ffff (0x1820 bytes) 000970 1787819297.561637 write 0x00e187 thru 0x01c10c (0xdf86 bytes) 000971 1787819297.566541 mapread 0x03a2e2 thru 0x03ffff (0x5d1e bytes) 000972 1787819297.573187 trunc from 0x040000 to 0x03836c (0x7c93 bytes) 000973 1787819297.626193 MIRROR_EXTEND to 2 mirrors 000974 1787819298.907958 fallocate 0x0149d9 thru 0x02397d (0xefa5 bytes) 000975 1787819298.961197 mapread 0x020488 thru 0x02122f (0xda8 bytes) 000976 1787819298.984179 mapread 0x016c32 thru 0x01e6ac (0x7a7b bytes) 000977 1787819298.999363 read 0x018e4b thru 0x028bc7 (0xfd7d bytes) 000978 1787819299.018354 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3836d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3836d): Mirror 2: 0x37dcf86f (stale) CRC-32 checksum value for chunk [0, 0x3836d): Mirror 1: 0x37dcf86f Mirror 3: 0x37dcf86f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3836d: skipped 000979 1787819299.997005 trunc from 0x00d8b0 to 0x00e6c1 (0xe12 bytes) 000981 1787819300.034287 mapread 0x017318 thru 0x025bc1 (0xe8aa bytes) 000982 1787819300.059888 trunc from 0x03836d to 0x03883f (0x4d3 bytes) 000983 1787819300.079287 read 0x010490 thru 0x0134f9 (0x306a bytes) 000985 1787819300.085933 write 0x000674 thru 0x00d74b (0xd0d8 bytes) 000986 1787819300.095828 trunc from 0x038840 to 0x01ffdb (0x18864 bytes) 000987 1787819300.110967 mapread 0x00cf85 thru 0x016752 (0x97ce bytes) 000988 1787819300.115934 mapread 0x00e619 thru 0x012858 (0x4240 bytes) 000989 1787819300.116761 fallocate 0x03adfa thru 0x03c5f7 (0x17fe bytes) 000990 1787819300.121255 write 0x0237a9 thru 0x02b32e (0x7b86 bytes) 000991 1787819300.122513 trunc from 0x005a80 to 0x00fdc6 (0xa347 bytes) 000992 1787819300.145586 write 0x00b0d1 thru 0x01440c (0x933c bytes) 000993 1787819300.150381 mapread 0x003942 thru 0x00e2b6 (0xa975 bytes) 000994 1787819300.158412 trunc from 0x015564 to 0x01e8c2 (0x935f bytes) 000996 1787819300.163624 mapwrite 0x02baf5 thru 0x03b545 (0xfa51 bytes) 000997 1787819300.229333 fallocate 0x0232f5 thru 0x02dd2d (0xaa39 bytes) 000998 1787819300.238064 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b546): Mirror 3: 0x9ccf220 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b546): Mirror 3: 0x9ccf220 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b546: skipped 000999 1787819300.573307 mapread 0x0289e3 thru 0x02ffc0 (0x75de bytes) 001000 1787819300.594712 mapread 0x023364 thru 0x0250f2 (0x1d8f bytes) All operations completed A-OK! PASS 70a (82s) == sanity-flr test 71: check mirror extend parent fid ==== 04:28:25 (1787819305) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x202:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 04:28:36 (1787819316) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.196596 s, 10.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949102 s, 11.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xea:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xea:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 04:28:44 (1787819324) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre2 Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1683040 .. Extending file size to 6482560 .. failed Extending file size to 6439712 .. Extending file size to 8077280 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7916160 .. Extending file size to 8103136 .. Extending file size to 8432032 .. Extending file size to 8324800 .. done Extending file size to 8806560 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8482016 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8720992 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8880608 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61376 61377 61378 61381 61382 failed 192.168.206.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre2 (opts:) 192.168.206.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:30:08 (1787819408) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre2 Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3748800 .. Extending file size to 1832640 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4439168 .. Extending file size to 8646816 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6612640 .. Extending file size to 8553152 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8733504 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8865696 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8934976 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62809 62810 62812 62814 62815 192.168.206.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre2 (opts:) 192.168.206.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:31:28 (1787819488) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2188 1269116 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18620 3588400 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24160 3582860 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42780 7171260 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2188 1269116 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18620 3588400 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24160 3582860 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42780 7171260 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10982 s, 19.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18418 s, 11.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xbe1b2c4 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:31:46 (1787819506) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0885751 s, 23.7 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 04:31:52 (1787819512) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x37:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x100:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x100:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x101:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x121:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:31:58 (1787819518) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113418 s, 18.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x240:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:32:06 (1787819526) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0870999 s, 24.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x241:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x242:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x240:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x241:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x242:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x240:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x241:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:32:14 (1787819534) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0800842 s, 26.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x102:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x122:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x102:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x122:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x123:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:32:48 (1787819568) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg656-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732739 s, 14.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x243:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x243:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x242:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x243:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x243:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x242:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x244:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:33:32 (1787819612) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg656-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x245:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x246:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0953141 s, 22.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x245:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x246:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x244:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x245:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x246:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x244:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:34:17 (1787819657) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg656-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg656-server: Pool lustre.test_204f created Adding targets to pool oleg656-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg656-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0913206 s, 23.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x247:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x245:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x247:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x245:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg656-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg656-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg656-server: Pool lustre.test_204f destroyed PASS 204f (61s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:35:19 (1787819719) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:35:25 (1787819725) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:35:32 (1787819732) oleg656-server: Pool lustre.test_206 created Adding targets to pool oleg656-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg656-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg656-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg656-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg656-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:35:57 (1787819757) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956559 s, 11.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442297 s, 23.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x61bd097d Mirror 3: 0x61bd097d PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:36:08 (1787819768) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:36:12 (1787819772) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:36:15 (1787819775) Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0524711 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ad0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1160000 Read this (4 bytes): BBBB 192.168.206.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg656-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:36:52 (1787819812) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0177569 s, 461 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:36:59 (1787819819) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488097 s, 21.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x52 lmm_fid: [0x200000405:0x52:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x52 lmm_fid: [0x200000405:0x52:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611218 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 04:37:01.000000000 -0400 Modify: 2026-08-27 04:37:01.000000000 -0400 Change: 2026-08-27 04:37:02.000000000 -0400 Birth: 2026-08-27 04:37:02.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:37:07 (1787819827) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528595 s, 19.8 MB/s oleg656-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg656-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg656-server: O/280000401/d14/10e: File not found by ext2_lookup PASS 210b (19s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 04:37:26 (1787819846) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.228729 s, 179 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.031739 s, 129 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 04:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 04:37 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 04:37:34 (1787819854) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 04:37:42 (1787819862) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1822 sec =========== 04:37:53 (1787819873) === sanity-flr: start cleanup 04:37:54 (1787819874) === === sanity-flr: finish cleanup 04:38:02 (1787819882) ===