-----============= acceptance-small: sanity-flr ============----- Fri May 15 06:02:26 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg141-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 06:02:48 (1778839368) === oleg141-client.virtnet: executing check_config_client /mnt/lustre oleg141-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg141-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9dd70308e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9dd70308e000.idle_timeout=debug disable quota as required oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 06:03:33 (1778839413) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8230] == sanity-flr test 0a: lfs mirror create with -N option == 06:03:36 (1778839416) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:03:57 (1778839437) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:04:01 (1778839441) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:04:05 (1778839445) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:04:32 (1778839472) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:04:37 (1778839477) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 06:04:41 (1778839481) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9dd70308e000.stats=clear osc.lustre-OST0001-osc-ffff9dd70308e000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:04:51 (1778839491) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:05:05 (1778839505) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:05:15 (1778839515) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:05:19 (1778839519) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:05:29 (1778839529) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 06:05:39 (1778839539) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:05:48 (1778839548) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:05:56 (1778839556) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:06:04 (1778839564) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:06:16 (1778839576) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:06:24 (1778839584) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.04867 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173782 s, 6.0 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:06:35 (1778839595) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.140406 s, 52.3 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:06:46 (1778839606) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:06:56 (1778839616) Stopping /mnt/lustre-ost1 (opts:) on oleg141-server oleg141-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dd70308e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dd70308e000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg141-server: oleg141-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg141-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg141-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg141-server oleg141-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9dd70308e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dd70308e000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg141-server: oleg141-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg141-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (137s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:09:13 (1778839753) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.338185 s, 18.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg141-server oleg141-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dd70308e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dd70308e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg141-server: oleg141-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg141-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:09:55 (1778839795) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg141-server oleg141-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dd70308e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dd70308e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg141-server: oleg141-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg141-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg141-server oleg141-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9dd70308e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dd70308e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg141-server: oleg141-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg141-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:11:07 (1778839867) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.78793 s, 18.8 MB/s -rw-r--r-- 1 root root 52428800 May 15 06:11 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.038s user 0m0.001s sys 0m0.194s Stopping /mnt/lustre-ost1 (opts:) on oleg141-server oleg141-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dd70308e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dd70308e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.113s user 0m0.000s sys 0m0.209s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg141-server: oleg141-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg141-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg141-server oleg141-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9dd70308e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dd70308e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.260s user 0m0.001s sys 0m0.186s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg141-server: oleg141-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg141-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9dd70308e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dd70308e000.ost_server_uuid in FULL state after 0 sec PASS 33b (85s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:12:32 (1778839952) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:12:35 (1778839955) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:12:38 (1778839958) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:12:41 (1778839961) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 06:12:49 (1778839969) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.495444 s, 25.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:13:00 (1778839980) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373253 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417604 s, 25.1 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:13:08 (1778839988) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:13:16 (1778839996) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00144137 s, 710 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000794376 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00631991 s, 162 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00378543 s, 271 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 06:13:25 (1778840005) create test files with size 15752962 .. 7b4201cc4d3cfe6fdde7f8637c057ae0 - 38c47e33332615ae633d0072369ac68e - 5397d7ae9c5a120b7b8253120c1df886 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg141-client.virtnet /mnt/lustre (opts:) Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 06:13:50 (1778840030) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:14:04 (1778840044) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:14:11 (1778840051) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628065 s, 16.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589108 s, 17.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 06:14:21 (1778840061) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108058 s, 19.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.235957 s, 17.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168082 s, 12.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 06:14:36 (1778840076) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 06:14:40 (1778840080) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:14:43 (1778840083) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235723 s, 44.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 06:14:52 (1778840092) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.305356 s, 10.3 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:15:00 (1778840100) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:15:10 (1778840110) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.312396 s, 33.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg141-client.virtnet /mnt/lustre (opts:) Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:15:22 (1778840122) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9dd707e87000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.354302 s, 29.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg141-client.virtnet /mnt/lustre (opts:) Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9dd708edf800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:15:37 (1778840137) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:16:07 (1778840167) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (29s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:16:37 (1778840197) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 06:16:40 (1778840200) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.316455 s, 9.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.132531 s, 23.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8cf259a2 (nosync,stale) Mirror 2: 0x8cf259a2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7be66bf573c28990a1d0a3cc3ed2bf34 - mirror 1 checksum: cf55680ef1a84d426be06ccb2af1ac22 - mirror 2 checksum: 7be66bf573c28990a1d0a3cc3ed2bf34 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 15 06:16:42 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 15 06:16:42 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7be66bf573c28990a1d0a3cc3ed2bf34 - mirror 1 checksum: cf55680ef1a84d426be06ccb2af1ac22 - mirror 2 checksum: cf55680ef1a84d426be06ccb2af1ac22 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 15 06:16:42 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 15 06:16:42 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 06:16:49 (1778840209) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229156 s, 45.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:16:57 (1778840217) Failing mds1 on oleg141-server Stopping /mnt/lustre-mds1 (opts:) on oleg141-server 06:17:09 (1778840229) shut down facet: mds1 facet_host: oleg141-server facet_failover_host: oleg141-server Failover mds1 to oleg141-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 06:17:30 (1778840250) targets are mounted 06:17:30 (1778840250) facet_failover done oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370855 s, 28.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:17:44 (1778840264) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0350189 s, 585 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0118459 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:17:54 (1778840274) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0570926 s, 359 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0116674 s, 1.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:18:02 (1778840282) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0101599 s, 1.6 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:18:10 (1778840290) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0430209 s, 476 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:18:17 (1778840297) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00498081 s, 822 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0359258 s, 114 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:18:23 (1778840303) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x14:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:18:43 (1778840323) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:19:10 (1778840350) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x3e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:19:31 (1778840371) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:19:53 (1778840393) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 06:20:00 (1778840400) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57184 mirror split pid 57185 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57184 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57185 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 06:21:06 (1778840466) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 468 000002 1778840468.435894 write 0x019048 thru 0x02407b (0xb034 bytes) truncating to largest ever: 0x308bb 000004 1778840468.445071 trunc from 0x02407c to 0x0308ba (0xc83f bytes) 000005 1778840468.481036 mapwrite 0x02fa26 thru 0x033966 (0x3f41 bytes) truncating to largest ever: 0x31cae 000006 1778840468.591678 trunc from 0x033967 to 0x031cad (0x1cb9 bytes) 000007 1778840468.617998 trunc from 0x031cae to 0x024393 (0xd91a bytes) 000008 1778840468.638328 punch from 0x000f06 to 0x00907e (0x8179 bytes) 000009 1778840468.643086 punch from 0x0235f8 to 0x024393 (0xd9c bytes) 000010 1778840468.649420 trunc from 0x009bfa to 0x0121b0 (0x85b7 bytes) 000012 1778840468.656022 trunc from 0x024394 to 0x019728 (0xac6b bytes) 000013 1778840468.671270 read 0x00c950 thru 0x00e3aa (0x1a5b bytes) 000014 1778840468.686998 read 0x0123b2 thru 0x019728 (0x7377 bytes) 000016 1778840468.717067 write 0x0296e0 thru 0x034aa6 (0xb3c7 bytes) 000017 1778840468.722025 read 0x02eb56 thru 0x033562 (0x4a0d bytes) 000018 1778840468.723028 write 0x032dd8 thru 0x03e392 (0xb5bb bytes) 000019 1778840468.727787 read 0x030722 thru 0x03a300 (0x9bdf bytes) 000020 1778840468.729372 write 0x02b46b thru 0x03702e (0xbbc4 bytes) 000021 1778840468.732364 trunc from 0x02142f to 0x030fe4 (0xfbb6 bytes) 000022 1778840468.760654 fallocate 0x01f85d thru 0x020f0c (0x16b0 bytes) 000023 1778840468.768621 trunc from 0x03e393 to 0x02e8e1 (0xfab1 bytes) 000024 1778840468.792472 trunc from 0x02e8e2 to 0x02241d (0xc4c4 bytes) 000025 1778840468.815207 read 0x01a1c5 thru 0x01a7fb (0x637 bytes) fallocating to largest ever: 0x39f88 000026 1778840468.829574 fallocate 0x031b97 thru 0x039f87 (0x83f1 bytes) 000027 1778840468.836241 punch from 0x000943 to 0x001679 (0xd37 bytes) 000029 1778840468.847172 trunc from 0x0039bc to 0x006900 (0x2f45 bytes) 000030 1778840468.855727 trunc from 0x01620f to 0x02060a (0xa3fc bytes) 000031 1778840468.864587 mapwrite 0x01c612 thru 0x01fad0 (0x34bf bytes) 000032 1778840468.921424 write 0x03ae06 thru 0x03ff22 (0x511d bytes) 000033 1778840468.922969 read 0x035111 thru 0x03c9a6 (0x7896 bytes) 000035 1778840468.924573 mapwrite 0x02de47 thru 0x03dd1c (0xfed6 bytes) 000036 1778840468.955237 trunc from 0x03ff23 to 0x0176af (0x28873 bytes) 000037 1778840468.972219 punch from 0x00e1d0 to 0x00e6cc (0x4fd bytes) 000038 1778840468.978254 read 0x0094fa thru 0x01468c (0xb193 bytes) 000039 1778840468.988460 fallocate 0x01c068 thru 0x0236a8 (0x7641 bytes) 000040 1778840468.995452 write 0x030ff1 thru 0x03d6c3 (0xc6d3 bytes) 000041 1778840468.998012 read 0x000f80 thru 0x0042ae (0x332f bytes) 000042 1778840468.999523 read 0x024326 thru 0x02d725 (0x9400 bytes) fallocating to largest ever: 0x3fdea 000043 1778840469.012729 fallocate 0x031c42 thru 0x03fde9 (0xe1a8 bytes) 000044 1778840469.020109 trunc from 0x03d6c4 to 0x026548 (0x1717b bytes) 000045 1778840469.041002 read 0x0010ef thru 0x00e227 (0xd139 bytes) 000046 1778840469.042666 trunc from 0x026549 to 0x032048 (0xbb00 bytes) 000047 1778840469.060292 trunc from 0x032049 to 0x02c8c9 (0x577f bytes) 000048 1778840469.084460 fallocate 0x017260 thru 0x021f64 (0xad05 bytes) 000049 1778840469.094194 read 0x0138e2 thru 0x015aa2 (0x21c1 bytes) 000050 1778840469.096155 trunc from 0x003696 to 0x011f52 (0xe8bd bytes) 000051 1778840469.105454 mapread 0x00dd81 thru 0x019a47 (0xbcc7 bytes) 000053 1778840469.132784 write 0x036969 thru 0x03f727 (0x8dbf bytes) 000054 1778840469.137627 trunc from 0x03f728 to 0x00d6d4 (0x32053 bytes) 000056 1778840469.172954 trunc from 0x00aa09 to 0x00b6f8 (0xcf0 bytes) 000058 1778840469.181774 trunc from 0x0033df to 0x00d6d4 (0xa2f6 bytes) 000059 1778840469.188196 mapwrite 0x011d0b thru 0x02010e (0xe404 bytes) 000060 1778840469.333923 read 0x0008c4 thru 0x007f3a (0x7677 bytes) 000061 1778840469.347611 read 0x01c103 thru 0x02010e (0x400c bytes) 000063 1778840469.350231 trunc from 0x02010f to 0x032096 (0x11f88 bytes) 000065 1778840469.376925 punch from 0x020460 to 0x02fc55 (0xf7f6 bytes) 000066 1778840469.384696 trunc from 0x032097 to 0x031a9c (0x5fa bytes) 000067 1778840469.410541 MIRROR_EXTEND to 2 mirrors 000068 1778840470.000548 trunc from 0x031a9d to 0x00303e (0x2ea5e bytes) 000069 1778840470.093570 mapread 0x000756 thru 0x00303e (0x28e9 bytes) 000070 1778840470.119888 trunc from 0x000853 to 0x00303e (0x27ec bytes) 000071 1778840470.126007 read 0x000db5 thru 0x00303e (0x228a bytes) 000072 1778840470.134599 trunc from 0x001b4d to 0x00303e (0x14f2 bytes) 000073 1778840470.143986 write 0x012602 thru 0x014981 (0x2380 bytes) 000074 1778840470.146669 mapwrite 0x02e6df thru 0x033d90 (0x56b2 bytes) 000075 1778840470.193552 trunc from 0x00ea62 to 0x010e2f (0x23ce bytes) 000076 1778840470.203035 write 0x03c0d7 thru 0x03ffff (0x3f29 bytes) 000077 1778840470.204773 trunc from 0x040000 to 0x036fc5 (0x903a bytes) 000078 1778840470.233089 MIRROR_RESYNC 2 mirrors 000080 1778840470.960873 mapwrite 0x01e322 thru 0x02aa02 (0xc6e1 bytes) 000081 1778840471.066635 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36fc6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36fc6): Mirror 2: 0x7ae0c071 (stale) CRC-32 checksum value for chunk [0, 0x36fc6): Mirror 1: 0x77a511b8 Mirror 3: 0x77a511b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36fc6: skipped 000082 1778840472.280827 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36fc6): Mirror 2: 0x7ae0c071 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36fc6): Mirror 2: 0x7ae0c071 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36fc6: skipped 000083 1778840472.674177 trunc from 0x036fc6 to 0x029c84 (0xd341 bytes) 000084 1778840472.728862 read 0x027d99 thru 0x029c84 (0x1eec bytes) 000085 1778840472.736694 mapread 0x020497 thru 0x02203e (0x1ba8 bytes) 000086 1778840472.767384 mapread 0x017cc3 thru 0x025ee3 (0xe221 bytes) 000087 1778840472.783867 mapread 0x012352 thru 0x016559 (0x4208 bytes) 000088 1778840472.786083 punch from 0x011587 to 0x01a3fd (0x8e77 bytes) fallocating to largest ever: 0x40000 000089 1778840472.794938 fallocate 0x036dc9 thru 0x03ffff (0x9237 bytes) 000090 1778840472.807837 mapwrite 0x03d2a1 thru 0x03ffff (0x2d5f bytes) 000091 1778840472.843974 punch from 0x01fd07 to 0x021bcd (0x1ec7 bytes) 000092 1778840472.856320 mapread 0x022c75 thru 0x02f965 (0xccf1 bytes) 000093 1778840472.874980 trunc from 0x040000 to 0x0190b6 (0x26f49 bytes) 000094 1778840472.901809 fallocate 0x03852e thru 0x03a700 (0x21d3 bytes) 000095 1778840472.908093 mapread 0x0101ba thru 0x0145ff (0x4446 bytes) 000096 1778840472.924261 punch from 0x017d0c to 0x025985 (0xdc7a bytes) 000097 1778840472.936526 punch from 0x023966 to 0x0324a6 (0xeb41 bytes) 000099 1778840472.942385 mapread 0x01fa96 thru 0x01fccd (0x238 bytes) 000100 1778840472.951894 read 0x0098c8 thru 0x00eb47 (0x5280 bytes) 000101 1778840472.965550 trunc from 0x0217cd to 0x0310f9 (0xf92d bytes) 000102 1778840472.971580 punch from 0x0351b8 to 0x03a700 (0x5549 bytes) 000103 1778840472.977621 write 0x008b97 thru 0x009255 (0x6bf bytes) 000104 1778840472.979045 trunc from 0x001919 to 0x005fde (0x46c6 bytes) 000105 1778840472.985587 trunc from 0x022410 to 0x02fe52 (0xda43 bytes) 000106 1778840472.996483 MIRROR_RESYNC 2 mirrors 000108 1778840473.744323 punch from 0x02d521 to 0x0387a7 (0xb287 bytes) 000109 1778840473.798579 punch from 0x01d10f to 0x02d009 (0xfefb bytes) 000110 1778840473.811834 write 0x02b347 thru 0x0347b6 (0x9470 bytes) 000111 1778840473.831057 trunc from 0x01f2a7 to 0x02576f (0x64c9 bytes) 000112 1778840473.839363 read 0x00cc18 thru 0x01673a (0x9b23 bytes) 000113 1778840473.849696 mapwrite 0x023c7d thru 0x024e4a (0x11ce bytes) 000115 1778840473.894533 mapread 0x0149f7 thru 0x015cf6 (0x1300 bytes) 000117 1778840473.901485 punch from 0x00cd3a to 0x01985c (0xcb23 bytes) 000118 1778840473.913499 mapread 0x0214d4 thru 0x02ece3 (0xd810 bytes) 000119 1778840473.916639 trunc from 0x03a701 to 0x01c402 (0x1e2fe bytes) 000120 1778840473.973487 trunc from 0x01c403 to 0x036b99 (0x1a797 bytes) 000121 1778840474.024024 punch from 0x02e157 to 0x036b99 (0x8a43 bytes) 000122 1778840474.031326 fallocate 0x00bf18 thru 0x0151ae (0x9297 bytes) 000123 1778840474.040069 punch from 0x01fad8 to 0x027b26 (0x804f bytes) 000125 1778840474.047438 trunc from 0x036b9a to 0x017801 (0x1f398 bytes) 000126 1778840474.077322 mapwrite 0x0227fa thru 0x028bd1 (0x63d8 bytes) 000127 1778840474.156119 fallocate 0x019cb5 thru 0x027895 (0xdbe1 bytes) 000128 1778840474.173516 MIRROR_RESYNC 2 mirrors 000129 1778840474.799974 mapread 0x01847b thru 0x01b326 (0x2eac bytes) 000130 1778840474.826934 mapwrite 0x00906a thru 0x00c422 (0x33b9 bytes) 000131 1778840474.930903 trunc from 0x028bd2 to 0x023f72 (0x4c5f bytes) 000132 1778840474.982112 trunc from 0x021726 to 0x023f72 (0x284d bytes) 000133 1778840474.990652 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23f73): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23f73): Mirror 2: 0x255417fc (stale) CRC-32 checksum value for chunk [0, 0x23f73): Mirror 1: 0x36447e11 Mirror 3: 0x36447e11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23f73: skipped 000134 1778840475.834999 MIRROR_RESYNC 3 mirrors 000135 1778840476.630611 trunc from 0x023f73 to 0x01f6ff (0x4873 bytes) 000136 1778840476.715427 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1f700): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f700): Mirror 2: 0xf35fb173 (stale) Mirror 3: 0xf35fb173 (stale) CRC-32 checksum value for chunk [0, 0x1f700): Mirror 1: 0xf35fb173 Mirror 4: 0xf35fb173 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f700: skipped 000137 1778840477.710769 trunc from 0x01f700 to 0x00511a (0x1a5e5 bytes) 000138 1778840477.785580 fallocate 0x03e50c thru 0x03ffff (0x1af4 bytes) 000139 1778840477.790964 trunc from 0x00511b to 0x00c74d (0x7633 bytes) 000140 1778840477.818626 trunc from 0x00c74e to 0x03dc37 (0x314ea bytes) 000141 1778840477.853703 MIRROR_RESYNC 4 mirrors 000142 1778840478.645502 read 0x03d894 thru 0x03dc37 (0x3a4 bytes) 000144 1778840478.649471 punch from 0x03d811 to 0x03dc37 (0x427 bytes) 000145 1778840478.693250 mapread 0x036198 thru 0x03dc37 (0x7aa0 bytes) 000146 1778840478.719079 read 0x02de18 thru 0x039fb1 (0xc19a bytes) 000148 1778840478.733651 trunc from 0x03dc38 to 0x00a2c5 (0x33972 bytes) 000149 1778840478.756237 trunc from 0x009a5a to 0x00a2c5 (0x86c bytes) 000150 1778840478.761422 mapread 0x006a93 thru 0x00a2c5 (0x3833 bytes) 000152 1778840478.776273 fallocate 0x016547 thru 0x017575 (0x102f bytes) 000153 1778840478.782761 trunc from 0x003c32 to 0x0095e0 (0x59af bytes) 000154 1778840478.790780 trunc from 0x003191 to 0x00a2c5 (0x7135 bytes) 000155 1778840478.794824 mapwrite 0x02bfdc thru 0x035a79 (0x9a9e bytes) 000156 1778840478.892009 fallocate 0x02b2b9 thru 0x038204 (0xcf4c bytes) 000158 1778840478.897831 trunc from 0x030a77 to 0x038204 (0x778e bytes) 000159 1778840478.903969 mapwrite 0x022502 thru 0x02f31d (0xce1c bytes) 000160 1778840478.961376 trunc from 0x038205 to 0x02bec0 (0xc344 bytes) 000161 1778840478.984434 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bec1): Mirror 2: 0xcc5cc7c5 (stale) Mirror 4: 0xcc5cc7c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bec1): Mirror 2: 0xcc5cc7c5 (stale) Mirror 4: 0xcc5cc7c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bec1: skipped 000162 1778840479.280946 write 0x00c3b6 thru 0x014ea6 (0x8af1 bytes) 000163 1778840479.298120 trunc from 0x02bec1 to 0x002ed2 (0x28fee bytes) 000164 1778840479.321029 write 0x027a3c thru 0x02f4ff (0x7ac4 bytes) 000165 1778840479.323747 fallocate 0x03582b thru 0x03dc8d (0x8463 bytes) 000166 1778840479.328604 mapwrite 0x009403 thru 0x014764 (0xb362 bytes) 000167 1778840479.369334 mapread 0x00dcda thru 0x01adad (0xd0d4 bytes) 000168 1778840479.370720 mapread 0x02cf0e thru 0x02f4ff (0x25f2 bytes) 000170 1778840479.373633 mapwrite 0x006578 thru 0x00db81 (0x760a bytes) 000171 1778840479.393775 trunc from 0x02f500 to 0x01b309 (0x141f6 bytes) 000172 1778840479.412890 trunc from 0x01238c to 0x01b309 (0x8f7e bytes) 000173 1778840479.420330 fallocate 0x014526 thru 0x01836b (0x3e46 bytes) 000174 1778840479.425477 trunc from 0x019a8e to 0x01b309 (0x187c bytes) 000175 1778840479.433878 mapread 0x003c0b thru 0x00f055 (0xb44b bytes) 000176 1778840479.435864 mapwrite 0x01a012 thru 0x01e297 (0x4286 bytes) 000177 1778840479.499913 read 0x00363f thru 0x008a51 (0x5413 bytes) 000178 1778840479.500927 trunc from 0x01e298 to 0x0051a9 (0x190ee bytes) 000179 1778840479.527170 fallocate 0x0396c6 thru 0x03ffff (0x693a bytes) 000180 1778840479.532874 write 0x016013 thru 0x025c06 (0xfbf4 bytes) 000181 1778840479.534936 trunc from 0x019364 to 0x0246c4 (0xb361 bytes) 000182 1778840479.567348 mapwrite 0x03aa90 thru 0x03b031 (0x5a2 bytes) 000183 1778840479.605448 trunc from 0x03b032 to 0x03084e (0xa7e3 bytes) 000184 1778840479.624456 trunc from 0x03084f to 0x024f4b (0xb903 bytes) 000185 1778840479.643065 fallocate 0x02b6f3 thru 0x03448e (0x8d9c bytes) 000186 1778840479.651464 trunc from 0x024f4c to 0x02d026 (0x80db bytes) 000187 1778840479.670082 fallocate 0x01b252 thru 0x02592e (0xa6dd bytes) 000188 1778840479.675432 punch from 0x01e815 to 0x02626b (0x7a57 bytes) 000189 1778840479.684305 trunc from 0x02d027 to 0x03c18c (0xf166 bytes) 000190 1778840479.701791 write 0x039b64 thru 0x03ffff (0x649c bytes) 000192 1778840479.722431 fallocate 0x00d30d thru 0x015f8b (0x8c7f bytes) 000193 1778840479.726810 trunc from 0x040000 to 0x009149 (0x36eb6 bytes) 000194 1778840479.746463 trunc from 0x006530 to 0x009149 (0x2c1a bytes) 000195 1778840479.754041 write 0x016481 thru 0x02286c (0xc3ec bytes) 000196 1778840479.755706 punch from 0x003ca9 to 0x00eb38 (0xae90 bytes) 000197 1778840479.767387 punch from 0x019090 to 0x02286c (0x97dd bytes) 000198 1778840479.788377 trunc from 0x02286d to 0x016578 (0xc2f4 bytes) 000199 1778840479.809066 mapread 0x009c47 thru 0x00c814 (0x2bce bytes) 000200 1778840479.826941 fallocate 0x0026b1 thru 0x011a30 (0xf380 bytes) 000201 1778840479.832607 punch from 0x00e100 to 0x016578 (0x8479 bytes) 000202 1778840479.841114 trunc from 0x0050ac to 0x00ca7e (0x79d3 bytes) 000203 1778840479.854390 MIRROR_RESYNC 3 mirrors 000204 1778840480.505339 mapread 0x006950 thru 0x00b395 (0x4a46 bytes) 000205 1778840480.530992 punch from 0x0123d1 to 0x016578 (0x41a8 bytes) 000206 1778840480.576783 punch from 0x00068e to 0x008d2b (0x869e bytes) 000207 1778840480.583893 mapread 0x014a67 thru 0x016578 (0x1b12 bytes) 000210 1778840480.595924 fallocate 0x03d728 thru 0x03ffff (0x28d8 bytes) 000211 1778840480.601031 trunc from 0x02eba4 to 0x02ecce (0x12b bytes) 000212 1778840480.606915 trunc from 0x02577f to 0x029127 (0x39a9 bytes) 000213 1778840480.611763 punch from 0x00e4a7 to 0x01b8cb (0xd425 bytes) 000214 1778840480.617375 mapwrite 0x006ce8 thru 0x00f10c (0x8425 bytes) 000215 1778840480.653751 punch from 0x03bf81 to 0x03ffff (0x407f bytes) 000216 1778840480.664895 trunc from 0x03d906 to 0x03ffff (0x26fa bytes) 000217 1778840480.672435 trunc from 0x00d0c5 to 0x01bd8e (0xecca bytes) 000218 1778840480.677238 trunc from 0x0252b6 to 0x0289b6 (0x3701 bytes) 000219 1778840480.681964 read 0x036818 thru 0x037907 (0x10f0 bytes) 000220 1778840480.687315 punch from 0x014b12 to 0x017cc6 (0x31b5 bytes) 000221 1778840480.701689 trunc from 0x022ba5 to 0x02bad4 (0x8f30 bytes) 000222 1778840480.712006 MIRROR_RESYNC 3 mirrors 000223 1778840481.463662 trunc from 0x03bff9 to 0x03ffff (0x4007 bytes) 000224 1778840481.501713 fallocate 0x036369 thru 0x03f270 (0x8f08 bytes) 000225 1778840481.508387 trunc from 0x02ace3 to 0x031248 (0x6566 bytes) 000226 1778840481.515035 trunc from 0x040000 to 0x001fe2 (0x3e01d bytes) 000227 1778840481.547232 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1fe3): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fe3): Mirror 2: 0x1b35e8a (stale) Mirror 4: 0x1b35e8a (stale) CRC-32 checksum value for chunk [0, 0x1fe3): Mirror 1: 0x1b35e8a Mirror 5: 0x1b35e8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fe3: skipped 000228 1778840482.298979 MIRROR_RESYNC 4 mirrors 000229 1778840482.843007 read 0x0017e7 thru 0x001fe2 (0x7fc bytes) 000230 1778840482.850728 fallocate 0x03f5f5 thru 0x03ffff (0xa0b bytes) 000231 1778840482.898857 punch from 0x02cdfe to 0x03106d (0x4270 bytes) 000232 1778840482.903944 read 0x0032d8 thru 0x00afb9 (0x7ce2 bytes) 000233 1778840482.918276 fallocate 0x009d1a thru 0x00b73c (0x1a23 bytes) 000234 1778840482.923557 write 0x03dfd2 thru 0x03ffff (0x202e bytes) 000237 1778840482.930065 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf749c9cb Mirror 6: 0xf749c9cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000238 1778840483.693022 trunc from 0x014b05 to 0x017137 (0x2633 bytes) 000239 1778840483.720570 trunc from 0x040000 to 0x03a362 (0x5c9d bytes) 000240 1778840483.744151 read 0x02d41e thru 0x033a9c (0x667f bytes) 000241 1778840483.754111 write 0x03209c thru 0x03ba18 (0x997d bytes) 000242 1778840483.756585 punch from 0x00dc7b to 0x011d64 (0x40ea bytes) 000243 1778840483.771879 MIRROR_RESYNC 5 mirrors 000244 1778840484.608410 trunc from 0x002e9d to 0x006a3c (0x3ba0 bytes) 000245 1778840484.662000 write 0x01c230 thru 0x01c63d (0x40e bytes) 000246 1778840484.674810 fallocate 0x000722 thru 0x00b77b (0xb05a bytes) 000247 1778840484.682889 write 0x026b29 thru 0x02f58b (0x8a63 bytes) 000248 1778840484.697350 write 0x02d58e thru 0x035e27 (0x889a bytes) 000249 1778840484.705603 punch from 0x027e3d to 0x0319f5 (0x9bb9 bytes) 000250 1778840484.720924 fallocate 0x0044c7 thru 0x0086f8 (0x4232 bytes) 000253 1778840484.726458 read 0x031f17 thru 0x03ba18 (0x9b02 bytes) 000254 1778840484.734710 trunc from 0x03ba19 to 0x031df4 (0x9c24 bytes) 000255 1778840484.751862 fallocate 0x02fda8 thru 0x03a40b (0xa664 bytes) 000256 1778840484.756819 fallocate 0x017e72 thru 0x0238c6 (0xba55 bytes) 000257 1778840484.763072 mapwrite 0x02ddfb thru 0x034cfa (0x6f00 bytes) 000258 1778840484.803108 punch from 0x0041e5 to 0x006db0 (0x2bcc bytes) 000259 1778840484.813019 read 0x0020d2 thru 0x00a1fb (0x812a bytes) 000260 1778840484.829852 read 0x00ffef thru 0x01139f (0x13b1 bytes) 000261 1778840484.830852 mapread 0x02d74a thru 0x03a40b (0xccc2 bytes) 000262 1778840484.832771 write 0x0002c1 thru 0x00be1e (0xbb5e bytes) 000264 1778840484.835633 write 0x01fd64 thru 0x0253f4 (0x5691 bytes) 000265 1778840484.837707 punch from 0x01e28b to 0x02a8e8 (0xc65e bytes) 000266 1778840484.852649 mapread 0x0289a4 thru 0x031021 (0x867e bytes) 000267 1778840484.863293 punch from 0x02ddcf to 0x03a40b (0xc63d bytes) 000269 1778840484.869373 trunc from 0x03a40c to 0x03f956 (0x554b bytes) 000270 1778840484.891681 write 0x019c6b thru 0x02779b (0xdb31 bytes) 000271 1778840484.893875 fallocate 0x02544e thru 0x02e223 (0x8dd6 bytes) 000273 1778840484.902512 trunc from 0x0155bf to 0x02096f (0xb3b1 bytes) 000274 1778840484.918892 read 0x018a12 thru 0x01fdc8 (0x73b7 bytes) 000275 1778840484.935865 write 0x02230d thru 0x0261f0 (0x3ee4 bytes) 000276 1778840484.937236 mapread 0x00f866 thru 0x01a53f (0xacda bytes) 000277 1778840484.944182 MIRROR_RESYNC 5 mirrors 000278 1778840486.259113 mapwrite 0x00b725 thru 0x00c665 (0xf41 bytes) 000279 1778840486.351085 mapread 0x03a95e thru 0x03f956 (0x4ff9 bytes) 000280 1778840486.369047 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f957): Mirror 4: 0x9bb4dd4a (stale) Mirror 5: 0x9bb4dd4a (stale) Mirror 6: 0x9bb4dd4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f957): Mirror 4: 0x9bb4dd4a (stale) Mirror 5: 0x9bb4dd4a (stale) Mirror 6: 0x9bb4dd4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f957: skipped 000282 1778840486.753835 punch from 0x0075eb to 0x014bbd (0xd5d3 bytes) 000283 1778840486.760910 write 0x0007f1 thru 0x008522 (0x7d32 bytes) 000284 1778840486.775859 trunc from 0x007b3e to 0x00b4de (0x39a1 bytes) 000285 1778840486.790916 punch from 0x014fdc to 0x02300c (0xe031 bytes) 000286 1778840486.795868 trunc from 0x02b730 to 0x03557d (0x9e4e bytes) 000287 1778840486.801323 fallocate 0x012adb thru 0x01ccae (0xa1d4 bytes) 000288 1778840486.805833 punch from 0x018074 to 0x022612 (0xa59f bytes) 000289 1778840486.810258 mapread 0x01d112 thru 0x026e72 (0x9d61 bytes) 000290 1778840486.820205 trunc from 0x03f957 to 0x027ce6 (0x17c70 bytes) 000291 1778840486.832472 mapread 0x01feba thru 0x0245b0 (0x46f7 bytes) 000292 1778840486.833759 mapread 0x020667 thru 0x023c0d (0x35a7 bytes) 000293 1778840486.839195 MIRROR_RESYNC 4 mirrors 000294 1778840487.654101 write 0x0308d5 thru 0x037954 (0x7080 bytes) 000295 1778840487.687556 mapwrite 0x003890 thru 0x0093bf (0x5b30 bytes) 000296 1778840487.743640 read 0x01ab7b thru 0x01f58f (0x4a15 bytes) 000297 1778840487.760583 mapwrite 0x027df8 thru 0x035c4e (0xde57 bytes) 000298 1778840487.797602 punch from 0x00c07a to 0x01069c (0x4623 bytes) 000299 1778840487.804644 mapread 0x02cfa1 thru 0x037954 (0xa9b4 bytes) 000300 1778840487.807484 mapread 0x01ddbe thru 0x026c9a (0x8edd bytes) 000301 1778840487.814273 MIRROR_RESYNC 4 mirrors 000302 1778840488.716060 fallocate 0x03f0a2 thru 0x03ffff (0xf5e bytes) 000303 1778840488.751820 trunc from 0x037955 to 0x0358c7 (0x208d bytes) 000304 1778840488.783466 mapread 0x003bf8 thru 0x0125a9 (0xe9b2 bytes) 000305 1778840488.837576 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x358c8): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358c8): Mirror 4: 0xcdba1e77 (stale) Mirror 5: 0xcdba1e77 (stale) Mirror 6: 0xcdba1e77 (stale) CRC-32 checksum value for chunk [0, 0x358c8): Mirror 1: 0xcdba1e77 Mirror 7: 0xcdba1e77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358c8: skipped 000306 1778840489.871316 trunc from 0x0010d7 to 0x00f6f7 (0xe621 bytes) 000307 1778840489.925774 trunc from 0x016cae to 0x0190bd (0x2410 bytes) 000308 1778840489.933596 read 0x00f497 thru 0x013353 (0x3ebd bytes) 000309 1778840489.945571 trunc from 0x01c3fa to 0x02b0e4 (0xeceb bytes) 000310 1778840489.962578 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x358c8): Mirror 4: 0xcdba1e77 (stale) Mirror 6: 0xcdba1e77 (stale) Mirror 7: 0xcdba1e77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358c8): Mirror 4: 0xcdba1e77 (stale) Mirror 6: 0xcdba1e77 (stale) Mirror 7: 0xcdba1e77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358c8: skipped 000311 1778840490.462846 trunc from 0x0026fb to 0x00f3b4 (0xccba bytes) 000312 1778840490.481025 MIRROR_RESYNC 4 mirrors 000313 1778840491.310963 fallocate 0x0095a9 thru 0x012e02 (0x985a bytes) 000314 1778840491.362361 write 0x014a0d thru 0x020c4c (0xc240 bytes) 000315 1778840491.382437 trunc from 0x0358c8 to 0x03bdbf (0x64f8 bytes) 000316 1778840491.426604 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bdc0): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bdc0): Mirror 4: 0x1bbf196a (stale) Mirror 6: 0x1bbf196a (stale) Mirror 7: 0x1bbf196a (stale) CRC-32 checksum value for chunk [0, 0x3bdc0): Mirror 1: 0xb7c908b Mirror 8: 0xb7c908b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bdc0: skipped 000317 1778840492.517953 write 0x024aa6 thru 0x03197e (0xced9 bytes) 000318 1778840492.585651 trunc from 0x03bdc0 to 0x02a199 (0x11c26 bytes) 000319 1778840492.604502 mapread 0x01bb84 thru 0x02a199 (0xe616 bytes) 000320 1778840492.614471 trunc from 0x0041bd to 0x010639 (0xc47d bytes) 000321 1778840492.618396 fallocate 0x028b5d thru 0x030f77 (0x841b bytes) 000322 1778840492.623873 punch from 0x02ac92 to 0x030f77 (0x62e6 bytes) 000323 1778840492.637504 mapread 0x0021f8 thru 0x0058c2 (0x36cb bytes) 000324 1778840492.646089 read 0x01361a thru 0x01b1d8 (0x7bbf bytes) 000325 1778840492.650682 mapread 0x01c0c0 thru 0x01d67d (0x15be bytes) 000326 1778840492.651712 mapwrite 0x01eda8 thru 0x023702 (0x495b bytes) 000327 1778840492.669457 write 0x0391fd thru 0x03ffff (0x6e03 bytes) 000328 1778840492.670865 trunc from 0x03e38d to 0x03ffff (0x1c73 bytes) 000329 1778840492.700897 mapread 0x026bc2 thru 0x03088a (0x9cc9 bytes) 000330 1778840492.713846 punch from 0x03a4e9 to 0x03ffff (0x5b17 bytes) 000331 1778840492.719065 write 0x008b20 thru 0x014bc6 (0xc0a7 bytes) 000332 1778840492.721369 punch from 0x00b590 to 0x01827d (0xccee bytes) 000333 1778840492.733018 trunc from 0x0346c7 to 0x035565 (0xe9f bytes) 000334 1778840492.738649 punch from 0x013a1d to 0x021da5 (0xe389 bytes) 000335 1778840492.744563 trunc from 0x022f21 to 0x031e58 (0xef38 bytes) 000336 1778840492.750038 trunc from 0x040000 to 0x005677 (0x3a988 bytes) 000337 1778840492.770035 read 0x0019fe thru 0x005677 (0x3c7a bytes) 000338 1778840492.771233 write 0x017aff thru 0x023567 (0xba69 bytes) 000339 1778840492.774610 trunc from 0x006c3c to 0x0115e5 (0xa9aa bytes) 000340 1778840492.788012 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23568): Mirror 4: 0x4c81865e (stale) Mirror 6: 0x4c81865e (stale) Mirror 7: 0x4c81865e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23568): Mirror 4: 0x4c81865e (stale) Mirror 6: 0x4c81865e (stale) Mirror 7: 0x4c81865e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23568: skipped 000341 1778840493.170188 mapwrite 0x02d442 thru 0x03cfcd (0xfb8c bytes) 000342 1778840493.243680 mapwrite 0x02e769 thru 0x034871 (0x6109 bytes) 000343 1778840493.262549 read 0x01b28f thru 0x021ec8 (0x6c3a bytes) 000344 1778840493.272661 trunc from 0x03cfce to 0x00ade2 (0x321eb bytes) 000345 1778840493.288751 read 0x003efe thru 0x0066be (0x27c1 bytes) 000346 1778840493.299838 trunc from 0x00ade3 to 0x03a87d (0x2fa9b bytes) 000347 1778840493.318835 write 0x01bdf2 thru 0x028622 (0xc831 bytes) 000348 1778840493.351570 read 0x02c1be thru 0x032137 (0x5f7a bytes) 000349 1778840493.352556 mapwrite 0x010ca2 thru 0x015642 (0x49a1 bytes) 000350 1778840493.370279 fallocate 0x00aaca thru 0x00f1b6 (0x46ed bytes) 000351 1778840493.375103 trunc from 0x03a87e to 0x021fcf (0x188ae bytes) 000352 1778840493.397092 trunc from 0x021fd0 to 0x0298cb (0x78fc bytes) 000353 1778840493.413980 mapwrite 0x00c844 thru 0x014d37 (0x84f4 bytes) 000354 1778840493.435682 trunc from 0x0298cc to 0x03ab41 (0x11276 bytes) 000355 1778840493.457076 mapwrite 0x0099b7 thru 0x014f2a (0xb574 bytes) 000356 1778840493.479628 read 0x00657b thru 0x016302 (0xfd88 bytes) 000358 1778840493.481403 fallocate 0x0172be thru 0x01bdd1 (0x4b14 bytes) 000359 1778840493.487209 read 0x020967 thru 0x02a299 (0x9933 bytes) 000361 1778840493.501023 trunc from 0x03606e to 0x0398bd (0x3850 bytes) 000362 1778840493.506563 mapwrite 0x0099e9 thru 0x010367 (0x697f bytes) 000363 1778840493.527072 read 0x02c637 thru 0x039446 (0xce10 bytes) 000365 1778840493.535325 trunc from 0x00a3d7 to 0x0107ff (0x6429 bytes) 000366 1778840493.543728 trunc from 0x013d55 to 0x01e842 (0xaaee bytes) 000367 1778840493.549951 punch from 0x003cb3 to 0x004edf (0x122d bytes) 000368 1778840493.555398 mapread 0x008245 thru 0x00c4c6 (0x4282 bytes) 000369 1778840493.571742 trunc from 0x03ab42 to 0x01adf2 (0x1fd4f bytes) 000370 1778840493.585748 fallocate 0x0220cd thru 0x0278ab (0x57df bytes) 000371 1778840493.591976 read 0x012ffa thru 0x01adf2 (0x7df9 bytes) 000372 1778840493.602065 mapwrite 0x038775 thru 0x03b38c (0x2c18 bytes) 000373 1778840493.648857 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b38d): Mirror 6: 0x7fcd5b20 (stale) Mirror 7: 0x7fcd5b20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b38d): Mirror 6: 0x7fcd5b20 (stale) Mirror 7: 0x7fcd5b20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b38d: skipped 000374 1778840493.990422 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b38d): Mirror 7: 0x7fcd5b20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b38d): Mirror 7: 0x7fcd5b20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b38d: skipped 000376 1778840494.262519 read 0x0326e2 thru 0x03b38c (0x8cab bytes) 000377 1778840494.272568 mapwrite 0x0119f2 thru 0x01fcad (0xe2bc bytes) 000378 1778840494.308183 trunc from 0x03b38d to 0x0198b4 (0x21ad8 bytes) 000379 1778840494.333785 read 0x003fd0 thru 0x00d47b (0x94ac bytes) 000380 1778840494.342523 punch from 0x00665c to 0x0113e8 (0xad8d bytes) 000381 1778840494.351067 trunc from 0x0198b5 to 0x019c67 (0x3b3 bytes) 000382 1778840494.361819 punch from 0x017a19 to 0x019c67 (0x224f bytes) 000383 1778840494.367750 trunc from 0x019c68 to 0x038180 (0x1e519 bytes) 000384 1778840494.385019 trunc from 0x038181 to 0x038252 (0xd2 bytes) 000385 1778840494.406963 trunc from 0x038253 to 0x018134 (0x2011e bytes) 000387 1778840494.430990 write 0x006974 thru 0x00c71c (0x5da9 bytes) 000388 1778840494.448901 fallocate 0x01ffb1 thru 0x02aaad (0xaafd bytes) 000389 1778840494.455805 trunc from 0x00c75a to 0x00f4a5 (0x2d4c bytes) 000390 1778840494.491890 read 0x011927 thru 0x018134 (0x680e bytes) 000391 1778840494.505720 punch from 0x017d06 to 0x018134 (0x42f bytes) 000392 1778840494.513663 punch from 0x006a01 to 0x011ab8 (0xb0b8 bytes) 000393 1778840494.521034 write 0x01bcd5 thru 0x027dd6 (0xc102 bytes) 000394 1778840494.532547 mapread 0x01ed17 thru 0x027dd6 (0x90c0 bytes) 000395 1778840494.535297 trunc from 0x027dd7 to 0x026143 (0x1c93 bytes) 000396 1778840494.574692 mapread 0x00c619 thru 0x0122b0 (0x5c98 bytes) 000397 1778840494.614654 punch from 0x018952 to 0x0216e6 (0x8d95 bytes) 000399 1778840494.630720 trunc from 0x026144 to 0x032afe (0xc9bb bytes) 000400 1778840494.651659 mapwrite 0x002b33 thru 0x0041ce (0x169c bytes) 000401 1778840494.666251 read 0x02c04a thru 0x032afe (0x6ab5 bytes) 000402 1778840494.678876 fallocate 0x035f98 thru 0x03cc82 (0x6ceb bytes) 000403 1778840494.684259 read 0x03ba02 thru 0x03cc82 (0x1281 bytes) 000404 1778840494.696063 punch from 0x0041a9 to 0x012376 (0xe1ce bytes) 000406 1778840494.704986 write 0x03a530 thru 0x03ffff (0x5ad0 bytes) 000407 1778840494.715726 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x925eca4b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x97c99cd1 Mirror 8: 0x97c99cd1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000408 1778840495.551454 mapwrite 0x0039f6 thru 0x00abb6 (0x71c1 bytes) 000410 1778840495.692342 trunc from 0x040000 to 0x0389bd (0x7642 bytes) 000411 1778840495.749143 fallocate 0x00e712 thru 0x012cdc (0x45cb bytes) 000412 1778840495.765610 trunc from 0x0389be to 0x013d9f (0x24c1e bytes) 000413 1778840495.787648 mapread 0x0031e8 thru 0x009799 (0x65b2 bytes) 000414 1778840495.793047 trunc from 0x009414 to 0x00d004 (0x3bf1 bytes) 000416 1778840495.813851 read 0x00768a thru 0x00b50c (0x3e83 bytes) 000417 1778840495.824602 mapread 0x008de7 thru 0x013d9f (0xafb9 bytes) 000418 1778840495.851914 trunc from 0x013da0 to 0x037ad1 (0x23d32 bytes) 000419 1778840495.878879 write 0x0130f3 thru 0x0141b4 (0x10c2 bytes) 000420 1778840495.894685 punch from 0x017ae3 to 0x01c113 (0x4631 bytes) 000421 1778840495.901546 fallocate 0x01df6a thru 0x02324e (0x52e5 bytes) 000422 1778840495.910932 punch from 0x01edf1 to 0x029191 (0xa3a1 bytes) 000423 1778840495.918852 fallocate 0x03fdc1 thru 0x03ffff (0x23f bytes) 000424 1778840495.935108 punch from 0x02f619 to 0x03603f (0x6a27 bytes) 000425 1778840495.944434 write 0x032302 thru 0x032e12 (0xb11 bytes) 000426 1778840495.965014 mapread 0x003146 thru 0x009426 (0x62e1 bytes) 000427 1778840495.967548 punch from 0x01c7b5 to 0x028e91 (0xc6dd bytes) 000428 1778840495.979734 MIRROR_RESYNC 3 mirrors 000429 1778840497.031723 write 0x01d07c thru 0x0236f4 (0x6679 bytes) 000430 1778840497.105059 write 0x02f4bf thru 0x0335d0 (0x4112 bytes) 000431 1778840497.133964 mapread 0x017313 thru 0x01b565 (0x4253 bytes) 000432 1778840497.157345 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd3834248 (stale) Mirror 8: 0xd3834248 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2a83eb4d Mirror 9: 0x2a83eb4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000433 1778840498.460728 trunc from 0x040000 to 0x0065cf (0x39a30 bytes) 000434 1778840498.526719 trunc from 0x0065d0 to 0x02abd4 (0x24605 bytes) 000435 1778840498.546946 write 0x01ba20 thru 0x01fe9c (0x447d bytes) 000436 1778840498.555380 trunc from 0x01e2ce to 0x0289db (0xa70e bytes) 000437 1778840498.569397 trunc from 0x025993 to 0x02abd4 (0x5242 bytes) 000438 1778840498.575498 trunc from 0x0043c4 to 0x00ff0d (0xbb4a bytes) 000439 1778840498.586052 MIRROR_RESYNC 4 mirrors 000440 1778840499.828409 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) 000441 1778840500.242780 punch from 0x004f18 to 0x00d8a8 (0x8991 bytes) 000442 1778840500.296697 write 0x017c32 thru 0x026841 (0xec10 bytes) 000443 1778840500.314685 punch from 0x020a43 to 0x02a44f (0x9a0d bytes) 000444 1778840500.329074 read 0x016da6 thru 0x018b38 (0x1d93 bytes) 000445 1778840500.339808 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2abd5): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2abd5): Mirror 8: 0x96fd9599 (stale) Mirror 9: 0x96fd9599 (stale) CRC-32 checksum value for chunk [0, 0x2abd5): Mirror 1: 0xe749743b Mirror 10: 0xe749743b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2abd5: skipped 000446 1778840501.263104 fallocate 0x02c65a thru 0x037a45 (0xb3ec bytes) 000447 1778840501.325955 trunc from 0x03014b to 0x031492 (0x1348 bytes) 000448 1778840501.337174 mapwrite 0x0028f7 thru 0x00e6dc (0xbde6 bytes) 000449 1778840501.438899 mapread 0x01955f thru 0x01d526 (0x3fc8 bytes) 000450 1778840501.475648 mapread 0x0045c0 thru 0x007aa7 (0x34e8 bytes) 000452 1778840501.480784 trunc from 0x037a46 to 0x039eda (0x2495 bytes) 000453 1778840501.550415 fallocate 0x00fcc8 thru 0x01052e (0x867 bytes) 000455 1778840501.564490 punch from 0x01d302 to 0x028cdd (0xb9dc bytes) 000456 1778840501.577522 read 0x034b88 thru 0x039eda (0x5353 bytes) 000457 1778840501.606109 mapread 0x021dc8 thru 0x02a3bc (0x85f5 bytes) 000459 1778840501.653796 mapwrite 0x0372ad thru 0x03bf44 (0x4c98 bytes) 000460 1778840501.708888 write 0x01dad3 thru 0x0277e9 (0x9d17 bytes) 000463 1778840501.713586 read 0x011bc0 thru 0x01b6ec (0x9b2d bytes) 000464 1778840501.728971 write 0x00b14e thru 0x013baf (0x8a62 bytes) 000466 1778840501.757629 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bf45): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf45): Mirror 8: 0xe9e91f51 (stale) Mirror 9: 0xe9e91f51 (stale) Mirror 10: 0x59322519 (stale) CRC-32 checksum value for chunk [0, 0x3bf45): Mirror 1: 0xb4afb6c0 Mirror 11: 0xb4afb6c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf45: skipped 000467 1778840503.118554 fallocate 0x031fe2 thru 0x03813b (0x615a bytes) 000468 1778840503.206017 trunc from 0x02f536 to 0x03b06c (0xbb37 bytes) 000469 1778840503.215636 fallocate 0x00b84f thru 0x00c608 (0xdba bytes) 000470 1778840503.238362 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bf45): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf45): Mirror 8: 0xe9e91f51 (stale) Mirror 9: 0xe9e91f51 (stale) Mirror 10: 0x59322519 (stale) Mirror 11: 0xb4afb6c0 (stale) CRC-32 checksum value for chunk [0, 0x3bf45): Mirror 1: 0xc572c38f Mirror 12: 0xc572c38f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf45: skipped 000471 1778840504.363600 MIRROR_RESYNC 6 mirrors 000472 1778840505.723810 write 0x034409 thru 0x03d530 (0x9128 bytes) 000473 1778840505.789889 MIRROR_RESYNC 6 mirrors 000474 1778840507.196384 trunc from 0x03d531 to 0x02165b (0x1bed5 bytes) 000475 1778840507.232265 trunc from 0x02165c to 0x0252e5 (0x3c8a bytes) 000476 1778840507.245443 punch from 0x0209b0 to 0x0252e5 (0x4936 bytes) 000477 1778840507.249725 trunc from 0x0252e6 to 0x035769 (0x10484 bytes) 000478 1778840507.261771 trunc from 0x03576a to 0x031475 (0x42f4 bytes) 000479 1778840507.277037 MIRROR_RESYNC 6 mirrors 000481 1778840508.544228 mapread 0x0291db thru 0x031475 (0x829b bytes) 000482 1778840508.564410 read 0x00bb31 thru 0x00cbf3 (0x10c3 bytes) 000483 1778840508.569825 fallocate 0x02c8da thru 0x0331e6 (0x690d bytes) 000484 1778840508.608892 mapwrite 0x029cb3 thru 0x029ecc (0x21a bytes) 000485 1778840508.629355 read 0x004cba thru 0x00f219 (0xa560 bytes) 000486 1778840508.649446 mapread 0x0139bb thru 0x014521 (0xb67 bytes) 000487 1778840508.654204 punch from 0x02c15e to 0x031475 (0x5318 bytes) 000488 1778840508.661912 punch from 0x02aa2f to 0x02b6e6 (0xcb8 bytes) 000489 1778840508.667980 mapwrite 0x0110fa thru 0x01ed2d (0xdc34 bytes) 000490 1778840508.704958 trunc from 0x030c2b to 0x031475 (0x84b bytes) 000491 1778840508.710333 fallocate 0x00e960 thru 0x013eed (0x558e bytes) 000492 1778840508.716174 trunc from 0x031476 to 0x039083 (0x7c0e bytes) 000493 1778840508.744910 fallocate 0x00bdb3 thru 0x010014 (0x4262 bytes) 000494 1778840508.752267 fallocate 0x013a8c thru 0x0225df (0xeb54 bytes) 000495 1778840508.762753 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39084): Mirror 9: 0x9ac3e8b3 (stale) Mirror 10: 0x9ac3e8b3 (stale) Mirror 11: 0x9ac3e8b3 (stale) Mirror 12: 0x9ac3e8b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39084): Mirror 9: 0x9ac3e8b3 (stale) Mirror 10: 0x9ac3e8b3 (stale) Mirror 11: 0x9ac3e8b3 (stale) Mirror 12: 0x9ac3e8b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39084: skipped 000496 1778840509.201758 trunc from 0x00a3ff to 0x016687 (0xc289 bytes) 000497 1778840509.208093 trunc from 0x039084 to 0x03d572 (0x44ef bytes) 000498 1778840509.240396 read 0x00c325 thru 0x0129d1 (0x66ad bytes) 000499 1778840509.257709 mapwrite 0x003b3b thru 0x0119dd (0xdea3 bytes) 000500 1778840509.302188 fallocate 0x006f3f thru 0x012ed1 (0xbf93 bytes) 000501 1778840509.307023 write 0x00368c thru 0x0088f5 (0x526a bytes) 000503 1778840509.309510 write 0x01e139 thru 0x024a71 (0x6939 bytes) 000504 1778840509.320016 trunc from 0x03d573 to 0x0337ca (0x9da8 bytes) 000505 1778840509.338664 mapread 0x028171 thru 0x02a741 (0x25d1 bytes) 000506 1778840509.367039 trunc from 0x01f43f to 0x027866 (0x8428 bytes) 000507 1778840509.387325 write 0x01acea thru 0x024e16 (0xa12d bytes) 000509 1778840509.395390 write 0x01de52 thru 0x029067 (0xb216 bytes) 000510 1778840509.398104 punch from 0x00c482 to 0x00febf (0x3a3e bytes) 000511 1778840509.407631 read 0x02f93d thru 0x0337ca (0x3e8e bytes) 000512 1778840509.437836 trunc from 0x03299c to 0x0337ca (0xe2f bytes) 000513 1778840509.454829 mapwrite 0x03206d thru 0x03ad70 (0x8d04 bytes) 000514 1778840509.551896 trunc from 0x00122d to 0x0045b5 (0x3389 bytes) 000515 1778840509.556220 mapwrite 0x0036ab thru 0x00aa1b (0x7371 bytes) 000516 1778840509.589052 read 0x00009e thru 0x00c518 (0xc47b bytes) 000517 1778840509.596084 mapread 0x021035 thru 0x027787 (0x6753 bytes) 000519 1778840509.597971 write 0x03cc42 thru 0x03ffff (0x33be bytes) 000520 1778840509.601230 fallocate 0x030363 thru 0x0362b8 (0x5f56 bytes) 000522 1778840509.607969 mapread 0x001a61 thru 0x00fb39 (0xe0d9 bytes) 000523 1778840509.611231 trunc from 0x00c5dc to 0x0199f9 (0xd41e bytes) 000524 1778840509.619298 fallocate 0x0381f1 thru 0x038a95 (0x8a5 bytes) 000525 1778840509.625264 write 0x033377 thru 0x03ba16 (0x86a0 bytes) 000526 1778840509.631576 mapwrite 0x03d4bb thru 0x03ffff (0x2b45 bytes) 000529 1778840509.650965 trunc from 0x028655 to 0x034fe9 (0xc995 bytes) 000530 1778840509.661427 fallocate 0x000a32 thru 0x00fd5a (0xf329 bytes) 000531 1778840509.669822 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xc792c347 (stale) Mirror 10: 0xc792c347 (stale) Mirror 11: 0xc792c347 (stale) Mirror 12: 0xc792c347 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc855dedc Mirror 13: 0xc855dedc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000532 1778840510.726877 trunc from 0x01d341 to 0x0263e3 (0x90a3 bytes) 000533 1778840510.781746 write 0x017bd9 thru 0x01a0c0 (0x24e8 bytes) 000534 1778840510.800304 mapread 0x013579 thru 0x0141fa (0xc82 bytes) 000535 1778840510.817600 fallocate 0x00c0c9 thru 0x014c3f (0x8b77 bytes) 000536 1778840510.824298 read 0x00a238 thru 0x0188a5 (0xe66e bytes) 000537 1778840510.834232 fallocate 0x0096bc thru 0x00df0a (0x484f bytes) 000538 1778840510.839841 read 0x01dce9 thru 0x01fe02 (0x211a bytes) 000539 1778840510.854681 write 0x01fdad thru 0x0290be (0x9312 bytes) 000540 1778840510.856782 read 0x039128 thru 0x03ca36 (0x390f bytes) 000541 1778840510.858042 mapread 0x0271c6 thru 0x035f21 (0xed5c bytes) 000542 1778840510.865091 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xc792c347 (stale) Mirror 10: 0xc792c347 (stale) Mirror 11: 0xc792c347 (stale) Mirror 12: 0xc792c347 (stale) Mirror 13: 0xc855dedc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5155559c Mirror 14: 0x5155559c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000546 1778840511.973428 mapwrite 0x03d9cf thru 0x03ffff (0x2631 bytes) 000547 1778840512.077936 read 0x003bc4 thru 0x006e04 (0x3241 bytes) 000548 1778840512.088684 fallocate 0x02580d thru 0x02c05d (0x6851 bytes) 000549 1778840512.094005 mapread 0x0055f6 thru 0x00a0b9 (0x4ac4 bytes) 000550 1778840512.106229 trunc from 0x040000 to 0x0233c2 (0x1cc3d bytes) 000551 1778840512.134071 mapread 0x01fed2 thru 0x0233c2 (0x34f1 bytes) 000552 1778840512.154072 read 0x007b59 thru 0x017815 (0xfcbd bytes) 000553 1778840512.167408 trunc from 0x0233c3 to 0x030cda (0xd918 bytes) 000554 1778840512.187702 punch from 0x02d4e4 to 0x030cda (0x37f7 bytes) 000555 1778840512.192672 write 0x004169 thru 0x011269 (0xd101 bytes) 000556 1778840512.194711 read 0x014d08 thru 0x01922a (0x4523 bytes) 000557 1778840512.199456 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30cdb): Mirror 9: 0x9a5f4f3c (stale) Mirror 11: 0x9a5f4f3c (stale) Mirror 12: 0x9a5f4f3c (stale) Mirror 13: 0xa0af3c57 (stale) Mirror 14: 0x2ec4b8ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cdb): Mirror 9: 0x9a5f4f3c (stale) Mirror 11: 0x9a5f4f3c (stale) Mirror 12: 0x9a5f4f3c (stale) Mirror 13: 0xa0af3c57 (stale) Mirror 14: 0x2ec4b8ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cdb: skipped 000558 1778840512.591869 read 0x021725 thru 0x02980b (0x80e7 bytes) 000559 1778840512.602470 read 0x001e90 thru 0x00a544 (0x86b5 bytes) 000560 1778840512.614080 fallocate 0x01c201 thru 0x02b3ef (0xf1ef bytes) 000561 1778840512.623056 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30cdb): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cdb): Mirror 9: 0x9a5f4f3c (stale) Mirror 11: 0x9a5f4f3c (stale) Mirror 12: 0x9a5f4f3c (stale) Mirror 13: 0xa0af3c57 (stale) Mirror 14: 0x2ec4b8ef (stale) CRC-32 checksum value for chunk [0, 0x30cdb): Mirror 1: 0x26f1fae1 Mirror 15: 0x26f1fae1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cdb: skipped 000562 1778840513.867100 trunc from 0x030cdb to 0x00ff85 (0x20d55 bytes) 000563 1778840513.937934 trunc from 0x009cd3 to 0x00ff85 (0x62b3 bytes) 000564 1778840513.957153 punch from 0x006a2b to 0x00af4e (0x4524 bytes) 000565 1778840513.965526 trunc from 0x00891a to 0x00ff85 (0x766c bytes) 000566 1778840513.978501 mapwrite 0x026abb thru 0x02aa2b (0x3f71 bytes) 000567 1778840514.084433 write 0x033653 thru 0x03f7fa (0xc1a8 bytes) 000568 1778840514.085847 read 0x00c908 thru 0x00f22b (0x2924 bytes) 000569 1778840514.100102 MIRROR_RESYNC 7 mirrors 000570 1778840515.850660 write 0x00bd55 thru 0x018f11 (0xd1bd bytes) 000571 1778840515.895063 trunc from 0x03bb03 to 0x03e283 (0x2781 bytes) 000572 1778840515.899361 mapread 0x02020a thru 0x0261b7 (0x5fae bytes) 000573 1778840515.912101 trunc from 0x02f027 to 0x03a13f (0xb119 bytes) 000574 1778840515.919263 read 0x013fd4 thru 0x01e430 (0xa45d bytes) 000575 1778840515.920177 read 0x0000da thru 0x00350f (0x3436 bytes) 000576 1778840515.929362 read 0x0200e7 thru 0x026082 (0x5f9c bytes) 000577 1778840515.930240 fallocate 0x03e33b thru 0x03ffff (0x1cc5 bytes) 000578 1778840515.935359 write 0x009c1a thru 0x00d4e1 (0x38c8 bytes) 000579 1778840515.937651 fallocate 0x03dbb4 thru 0x03ffff (0x244c bytes) 000580 1778840515.942436 trunc from 0x00badf to 0x019c3a (0xe15c bytes) 000581 1778840515.955175 trunc from 0x040000 to 0x00bb99 (0x34466 bytes) 000582 1778840515.978604 trunc from 0x0098a1 to 0x00a60f (0xd6f bytes) 000583 1778840515.984667 trunc from 0x00a7f5 to 0x00bb99 (0x13a5 bytes) 000585 1778840515.993394 fallocate 0x03e7c4 thru 0x03ffff (0x183c bytes) 000586 1778840515.997322 read 0x008ee2 thru 0x00bb99 (0x2cb8 bytes) 000587 1778840516.001496 punch from 0x0054d2 to 0x00bb99 (0x66c8 bytes) 000588 1778840516.007601 punch from 0x00002d to 0x00bb99 (0xbb6d bytes) 000589 1778840516.012855 mapread 0x0035ba thru 0x005f0b (0x2952 bytes) 000590 1778840516.030301 read 0x000534 thru 0x001286 (0xd53 bytes) 000591 1778840516.044639 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbb9a): Mirror 9: 0xcd52d6e4 (stale) Mirror 12: 0xcd52d6e4 (stale) Mirror 13: 0xcd52d6e4 (stale) Mirror 14: 0xcd52d6e4 (stale) Mirror 15: 0xcd52d6e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbb9a): Mirror 9: 0xcd52d6e4 (stale) Mirror 12: 0xcd52d6e4 (stale) Mirror 13: 0xcd52d6e4 (stale) Mirror 14: 0xcd52d6e4 (stale) Mirror 15: 0xcd52d6e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbb9a: skipped 000592 1778840516.366115 mapwrite 0x031f8f thru 0x039cf3 (0x7d65 bytes) 000593 1778840516.411893 fallocate 0x01b029 thru 0x01f185 (0x415d bytes) 000594 1778840516.416436 punch from 0x030f51 to 0x039cf3 (0x8da3 bytes) 000595 1778840516.420810 trunc from 0x039cf4 to 0x003f13 (0x35de0 bytes) 000596 1778840516.444057 mapwrite 0x0270ec thru 0x034fa8 (0xdebd bytes) 000597 1778840516.506252 mapread 0x01d349 thru 0x023173 (0x5e2b bytes) 000598 1778840516.525994 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34fa9): Mirror 9: 0xa65ca97f (stale) Mirror 12: 0xa65ca97f (stale) Mirror 13: 0xa65ca97f (stale) Mirror 15: 0xa65ca97f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34fa9): Mirror 9: 0xa65ca97f (stale) Mirror 12: 0xa65ca97f (stale) Mirror 13: 0xa65ca97f (stale) Mirror 15: 0xa65ca97f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34fa9: skipped 000599 1778840516.898239 mapread 0x010007 thru 0x017bc5 (0x7bbf bytes) 000601 1778840516.929043 trunc from 0x02c43e to 0x034fa8 (0x8b6b bytes) 000602 1778840516.934876 trunc from 0x000126 to 0x00a413 (0xa2ee bytes) 000603 1778840516.942066 mapread 0x0272d8 thru 0x034fa8 (0xdcd1 bytes) 000604 1778840516.975841 mapread 0x01c7ef thru 0x01f146 (0x2958 bytes) 000605 1778840516.984863 trunc from 0x0240cd to 0x0285e4 (0x4518 bytes) 000606 1778840516.997339 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34fa9): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34fa9): Mirror 9: 0xa65ca97f (stale) Mirror 12: 0xa65ca97f (stale) Mirror 13: 0xa65ca97f (stale) Mirror 15: 0xa65ca97f (stale) CRC-32 checksum value for chunk [0, 0x34fa9): Mirror 1: 0x25014103 Mirror 16: 0x25014103 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34fa9: skipped 000607 1778840517.884579 punch from 0x0349d4 to 0x034fa8 (0x5d5 bytes) 000608 1778840517.917844 trunc from 0x034bef to 0x034fa8 (0x3ba bytes) 000609 1778840517.925955 read 0x015319 thru 0x015859 (0x541 bytes) 000611 1778840517.931870 mapwrite 0x011b9e thru 0x013f03 (0x2366 bytes) 000612 1778840517.964662 punch from 0x0205e2 to 0x02cae0 (0xc4ff bytes) 000613 1778840517.973774 read 0x011e75 thru 0x018d27 (0x6eb3 bytes) 000614 1778840517.974474 punch from 0x013add to 0x0186e7 (0x4c0b bytes) 000615 1778840517.979909 punch from 0x01273c to 0x018ed5 (0x679a bytes) 000616 1778840517.989696 write 0x00af0e thru 0x0117b7 (0x68aa bytes) 000618 1778840518.007406 read 0x00b8cb thru 0x010383 (0x4ab9 bytes) 000619 1778840518.008086 read 0x0226b1 thru 0x0253c8 (0x2d18 bytes) 000620 1778840518.009946 mapread 0x01966d thru 0x0224f1 (0x8e85 bytes) 000621 1778840518.016761 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x34fa9): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34fa9): Mirror 9: 0xa65ca97f (stale) Mirror 12: 0xa65ca97f (stale) Mirror 13: 0xa65ca97f (stale) Mirror 15: 0xa65ca97f (stale) Mirror 16: 0x25014103 (stale) CRC-32 checksum value for chunk [0, 0x34fa9): Mirror 1: 0x8ce005a5 Mirror 17: 0x8ce005a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34fa9: skipped 000622 1778840519.157735 read 0x015df1 thru 0x017fdc (0x21ec bytes) 000623 1778840519.169396 read 0x026b62 thru 0x033774 (0xcc13 bytes) 000624 1778840519.187074 trunc from 0x013cf4 to 0x02162b (0xd938 bytes) 000625 1778840519.239597 read 0x013a4b thru 0x020b59 (0xd10f bytes) 000626 1778840519.251063 read 0x00368c thru 0x0077bc (0x4131 bytes) 000628 1778840519.252347 write 0x009196 thru 0x013953 (0xa7be bytes) 000629 1778840519.258543 trunc from 0x034fa9 to 0x029cfc (0xb2ac bytes) 000630 1778840519.309675 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29cfd): Mirror 9: 0x7d66a0ac (stale) Mirror 12: 0x7d66a0ac (stale) Mirror 13: 0x7d66a0ac (stale) Mirror 15: 0x7d66a0ac (stale) Mirror 16: 0xcfa6d6e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29cfd): Mirror 9: 0x7d66a0ac (stale) Mirror 12: 0x7d66a0ac (stale) Mirror 13: 0x7d66a0ac (stale) Mirror 15: 0x7d66a0ac (stale) Mirror 16: 0xcfa6d6e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29cfd: skipped 000631 1778840519.734096 punch from 0x00e41d to 0x01606e (0x7c52 bytes) 000632 1778840519.744574 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x29cfd): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29cfd): Mirror 9: 0x7d66a0ac (stale) Mirror 12: 0x7d66a0ac (stale) Mirror 13: 0x7d66a0ac (stale) Mirror 15: 0x7d66a0ac (stale) Mirror 16: 0xcfa6d6e1 (stale) CRC-32 checksum value for chunk [0, 0x29cfd): Mirror 1: 0x5d5015ec Mirror 17: 0x5d5015ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29cfd: skipped 000633 1778840520.647913 trunc from 0x020044 to 0x027959 (0x7916 bytes) 000634 1778840520.672480 fallocate 0x001ba8 thru 0x00fb37 (0xdf90 bytes) 000635 1778840520.676643 trunc from 0x00d398 to 0x00fff2 (0x2c5b bytes) 000636 1778840520.680311 mapwrite 0x01ddbf thru 0x023a2c (0x5c6e bytes) 000637 1778840520.703768 read 0x00797c thru 0x01123d (0x98c2 bytes) 000640 1778840520.710093 trunc from 0x029cfd to 0x022f9f (0x6d5d bytes) 000641 1778840520.724464 write 0x039d53 thru 0x03e870 (0x4b1e bytes) 000642 1778840520.728971 mapread 0x023f0e thru 0x02e634 (0xa727 bytes) 000643 1778840520.738081 write 0x003d08 thru 0x0097b1 (0x5aaa bytes) 000644 1778840520.743966 trunc from 0x03e871 to 0x010c45 (0x2dc2b bytes) 000645 1778840520.754233 write 0x0060d8 thru 0x013eb0 (0xddd9 bytes) 000646 1778840520.755462 mapread 0x00e393 thru 0x00e64d (0x2bb bytes) 000647 1778840520.756107 trunc from 0x013eb1 to 0x00f02f (0x4e81 bytes) 000648 1778840520.765713 trunc from 0x00f030 to 0x031637 (0x22608 bytes) 000649 1778840520.776436 punch from 0x001842 to 0x00b3a4 (0x9b63 bytes) 000650 1778840520.786049 punch from 0x0228f5 to 0x022efa (0x606 bytes) 000651 1778840520.789039 punch from 0x02c95b to 0x031637 (0x4cdd bytes) 000652 1778840520.792648 punch from 0x027f98 to 0x02865b (0x6c4 bytes) 000653 1778840520.795717 write 0x02915d thru 0x03094e (0x77f2 bytes) 000654 1778840520.801501 mapread 0x00fe45 thru 0x018d01 (0x8ebd bytes) 000655 1778840520.808320 read 0x015eed thru 0x01d35c (0x7470 bytes) 000656 1778840520.815647 punch from 0x01135b to 0x0118d1 (0x577 bytes) 000658 1778840520.819177 mapwrite 0x02abbe thru 0x02f7a1 (0x4be4 bytes) 000659 1778840520.852226 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x31638): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31638): Mirror 9: 0x8f08c6be (stale) Mirror 12: 0x8f08c6be (stale) Mirror 13: 0x8f08c6be (stale) Mirror 15: 0x8f08c6be (stale) Mirror 16: 0x6028d78e (stale) Mirror 17: 0x4ef947ac (stale) CRC-32 checksum value for chunk [0, 0x31638): Mirror 1: 0x8da5b401 Mirror 18: 0x8da5b401 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31638: skipped 000660 1778840522.007916 read 0x00844b thru 0x016553 (0xe109 bytes) 000661 1778840522.028607 trunc from 0x01bd5c to 0x026068 (0xa30d bytes) 000662 1778840522.075687 punch from 0x026813 to 0x0289be (0x21ac bytes) 000663 1778840522.083437 fallocate 0x009297 thru 0x0166b7 (0xd421 bytes) 000665 1778840522.092033 fallocate 0x02a489 thru 0x039149 (0xecc1 bytes) 000666 1778840522.097578 mapread 0x00f914 thru 0x01e1d9 (0xe8c6 bytes) 000667 1778840522.124858 trunc from 0x03914a to 0x023d98 (0x153b1 bytes) 000668 1778840522.144660 mapwrite 0x0354c3 thru 0x039d53 (0x4891 bytes) 000669 1778840522.198730 fallocate 0x015ecf thru 0x023377 (0xd4a9 bytes) 000670 1778840522.205063 punch from 0x019239 to 0x01b6ce (0x2496 bytes) 000671 1778840522.210093 trunc from 0x0158ab to 0x02291c (0xd072 bytes) 000672 1778840522.214324 mapwrite 0x02cdf7 thru 0x02e321 (0x152b bytes) 000674 1778840522.244760 trunc from 0x039d54 to 0x034b19 (0x523a bytes) 000675 1778840522.260733 read 0x028260 thru 0x02da55 (0x57f6 bytes) 000676 1778840522.265479 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x34b1a): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34b1a): Mirror 9: 0xd73b7173 (stale) Mirror 12: 0xd73b7173 (stale) Mirror 13: 0xd73b7173 (stale) Mirror 15: 0xd73b7173 (stale) Mirror 16: 0x3988da01 (stale) Mirror 17: 0x4666f2db (stale) Mirror 18: 0xefd754be (stale) CRC-32 checksum value for chunk [0, 0x34b1a): Mirror 1: 0x541f816b Mirror 19: 0x541f816b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34b1a: skipped 000677 1778840523.324252 mapwrite 0x038788 thru 0x03ad08 (0x2581 bytes) 000678 1778840523.484970 MIRROR_RESYNC 9 mirrors 000680 1778840525.851109 mapread 0x02fb4d thru 0x03a588 (0xaa3c bytes) 000681 1778840525.874821 fallocate 0x03c05c thru 0x03ffff (0x3fa4 bytes) 000682 1778840525.912446 trunc from 0x0321fb to 0x03ad08 (0x8b0e bytes) 000683 1778840525.918447 mapread 0x03ac1d thru 0x03ad08 (0xec bytes) 000684 1778840525.930434 trunc from 0x03ad09 to 0x00dfa6 (0x2cd62 bytes) 000685 1778840525.949986 trunc from 0x009d32 to 0x00dfa6 (0x4275 bytes) 000687 1778840525.954626 mapwrite 0x00d9a8 thru 0x016d07 (0x9360 bytes) 000688 1778840526.006984 punch from 0x0134a3 to 0x016d07 (0x3865 bytes) 000689 1778840526.012283 trunc from 0x016d08 to 0x00dc0a (0x90fd bytes) 000690 1778840526.031416 read 0x002581 thru 0x00dc0a (0xb68a bytes) 000693 1778840526.042349 trunc from 0x00dc0b to 0x004b74 (0x9096 bytes) 000694 1778840526.062329 trunc from 0x004b75 to 0x019a22 (0x14eae bytes) 000695 1778840526.091993 write 0x00c9c0 thru 0x00dd2b (0x136c bytes) 000696 1778840526.109053 punch from 0x0112bd to 0x019a22 (0x8766 bytes) 000697 1778840526.117401 fallocate 0x00f8ac thru 0x016914 (0x7069 bytes) 000698 1778840526.123973 mapread 0x00c3a6 thru 0x00ca44 (0x69f bytes) 000699 1778840526.140797 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x19a23): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19a23): Mirror 9: 0x8cd49fe4 (stale) Mirror 12: 0x8cd49fe4 (stale) Mirror 13: 0x8cd49fe4 (stale) Mirror 15: 0x8cd49fe4 (stale) Mirror 16: 0x8cd49fe4 (stale) Mirror 17: 0x8cd49fe4 (stale) Mirror 18: 0x8cd49fe4 (stale) Mirror 19: 0x8cd49fe4 (stale) CRC-32 checksum value for chunk [0, 0x19a23): Mirror 1: 0xccfa107 Mirror 20: 0xccfa107 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19a23: skipped 000700 1778840527.169992 mapwrite 0x005f14 thru 0x007733 (0x1820 bytes) 000701 1778840527.245400 write 0x03e728 thru 0x03ffff (0x18d8 bytes) 000702 1778840527.255839 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x99ad0fdd (stale) Mirror 13: 0x99ad0fdd (stale) Mirror 15: 0x99ad0fdd (stale) Mirror 16: 0x99ad0fdd (stale) Mirror 17: 0x99ad0fdd (stale) Mirror 18: 0x99ad0fdd (stale) Mirror 19: 0x99ad0fdd (stale) Mirror 20: 0x9205112f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x99ad0fdd (stale) Mirror 13: 0x99ad0fdd (stale) Mirror 15: 0x99ad0fdd (stale) Mirror 16: 0x99ad0fdd (stale) Mirror 17: 0x99ad0fdd (stale) Mirror 18: 0x99ad0fdd (stale) Mirror 19: 0x99ad0fdd (stale) Mirror 20: 0x9205112f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000703 1778840527.743463 mapwrite 0x00e258 thru 0x01054b (0x22f4 bytes) 000704 1778840527.775109 write 0x03dccc thru 0x03ffff (0x2334 bytes) 000705 1778840527.788528 trunc from 0x03a4a5 to 0x03ffff (0x5b5b bytes) 000706 1778840527.798966 mapwrite 0x02592f thru 0x033443 (0xdb15 bytes) 000707 1778840527.839632 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x99ad0fdd (stale) Mirror 13: 0x99ad0fdd (stale) Mirror 15: 0x99ad0fdd (stale) Mirror 16: 0x99ad0fdd (stale) Mirror 17: 0x99ad0fdd (stale) Mirror 18: 0x99ad0fdd (stale) Mirror 19: 0x99ad0fdd (stale) Mirror 20: 0x9205112f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf3750591 Mirror 21: 0xf3750591 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000708 1778840528.830918 punch from 0x014b85 to 0x017dfd (0x3279 bytes) 000709 1778840528.865703 write 0x005a28 thru 0x00a774 (0x4d4d bytes) 000710 1778840528.878739 read 0x014e2e thru 0x018786 (0x3959 bytes) 000711 1778840528.890979 fallocate 0x03c6cc thru 0x03ffff (0x3934 bytes) 000712 1778840528.897489 fallocate 0x03a88f thru 0x03ffff (0x5771 bytes) 000713 1778840528.903526 read 0x025ecf thru 0x0340e7 (0xe219 bytes) 000714 1778840528.915470 mapwrite 0x02f39c thru 0x035b7a (0x67df bytes) 000716 1778840528.944080 trunc from 0x0020b5 to 0x010c40 (0xeb8c bytes) 000717 1778840528.951769 fallocate 0x00ce1c thru 0x00db96 (0xd7b bytes) 000719 1778840528.960010 punch from 0x017257 to 0x018d40 (0x1aea bytes) 000720 1778840528.970620 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x99ad0fdd (stale) Mirror 13: 0x99ad0fdd (stale) Mirror 15: 0x99ad0fdd (stale) Mirror 16: 0x99ad0fdd (stale) Mirror 17: 0x99ad0fdd (stale) Mirror 18: 0x99ad0fdd (stale) Mirror 19: 0x99ad0fdd (stale) Mirror 20: 0x9205112f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x99ad0fdd (stale) Mirror 13: 0x99ad0fdd (stale) Mirror 15: 0x99ad0fdd (stale) Mirror 16: 0x99ad0fdd (stale) Mirror 17: 0x99ad0fdd (stale) Mirror 18: 0x99ad0fdd (stale) Mirror 19: 0x99ad0fdd (stale) Mirror 20: 0x9205112f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000721 1778840529.538899 MIRROR_RESYNC 9 mirrors 000722 1778840531.680852 mapread 0x02ef35 thru 0x035c5e (0x6d2a bytes) 000723 1778840531.698321 fallocate 0x0209cc thru 0x02d002 (0xc637 bytes) 000724 1778840531.744895 read 0x002334 thru 0x00d3c0 (0xb08d bytes) 000725 1778840531.754571 trunc from 0x03feb6 to 0x03ffff (0x14a bytes) 000726 1778840531.763989 mapwrite 0x02ac04 thru 0x039d62 (0xf15f bytes) 000727 1778840531.824837 punch from 0x035bd4 to 0x03ffff (0xa42c bytes) 000728 1778840531.829033 write 0x02846e thru 0x033b4a (0xb6dd bytes) 000729 1778840531.831106 mapread 0x02a110 thru 0x031610 (0x7501 bytes) 000730 1778840531.832751 trunc from 0x01ff38 to 0x028415 (0x84de bytes) 000731 1778840531.847065 mapwrite 0x03066b thru 0x03fd08 (0xf69e bytes) 000732 1778840531.887800 fallocate 0x0007a1 thru 0x00844b (0x7cab bytes) 000733 1778840531.900416 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x1e46bc2a (stale) Mirror 13: 0x1e46bc2a (stale) Mirror 15: 0x1e46bc2a (stale) Mirror 16: 0x1e46bc2a (stale) Mirror 17: 0x1e46bc2a (stale) Mirror 18: 0x1e46bc2a (stale) Mirror 19: 0x1e46bc2a (stale) Mirror 20: 0x1e46bc2a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xecddac05 Mirror 21: 0xecddac05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000734 1778840532.670276 write 0x026e3f thru 0x031fa7 (0xb169 bytes) 000735 1778840532.716796 trunc from 0x040000 to 0x018849 (0x277b6 bytes) 000736 1778840532.737864 read 0x0038c3 thru 0x00ada2 (0x74e0 bytes) 000737 1778840532.744410 fallocate 0x0395f4 thru 0x03ffff (0x6a0c bytes) 000738 1778840532.748591 write 0x0114dd thru 0x01bf5c (0xaa80 bytes) 000739 1778840532.764787 trunc from 0x040000 to 0x022be9 (0x1d416 bytes) 000740 1778840532.775913 write 0x01ed4f thru 0x02a03e (0xb2f0 bytes) 000741 1778840532.784514 trunc from 0x02a03f to 0x000df5 (0x29249 bytes) 000742 1778840532.800959 trunc from 0x000df6 to 0x012f2e (0x12139 bytes) 000743 1778840532.828526 read 0x008e4e thru 0x00ce5b (0x400e bytes) 000744 1778840532.841680 trunc from 0x012f2f to 0x010487 (0x2aa7 bytes) 000745 1778840532.873369 mapread 0x005678 thru 0x006621 (0xfaa bytes) 000746 1778840532.874508 mapread 0x0073f0 thru 0x010487 (0x9098 bytes) 000747 1778840532.876566 fallocate 0x02705d thru 0x032f64 (0xbf08 bytes) 000748 1778840532.882268 write 0x00aaac thru 0x011c02 (0x7157 bytes) 000751 1778840532.891399 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x11c03): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11c03): Mirror 12: 0xf61b95cc (stale) Mirror 13: 0xf61b95cc (stale) Mirror 15: 0xf61b95cc (stale) Mirror 16: 0xf61b95cc (stale) Mirror 17: 0xf61b95cc (stale) Mirror 18: 0xf61b95cc (stale) Mirror 19: 0xf61b95cc (stale) Mirror 20: 0xf61b95cc (stale) Mirror 21: 0xf61b95cc (stale) CRC-32 checksum value for chunk [0, 0x11c03): Mirror 1: 0x3b05bf06 Mirror 22: 0x3b05bf06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11c03: skipped 000752 1778840533.788632 trunc from 0x011027 to 0x011c02 (0xbdc bytes) 000753 1778840533.831203 mapread 0x001297 thru 0x00c150 (0xaeba bytes) 000754 1778840533.849878 mapwrite 0x00ac57 thru 0x0171ed (0xc597 bytes) 000756 1778840533.896504 write 0x010514 thru 0x01fb49 (0xf636 bytes) 000758 1778840533.903850 punch from 0x00d825 to 0x0169e0 (0x91bc bytes) 000759 1778840533.937036 mapread 0x014eff thru 0x01e0ef (0x91f1 bytes) 000760 1778840533.946985 read 0x0044c1 thru 0x00d295 (0x8dd5 bytes) 000761 1778840533.947852 mapwrite 0x031be5 thru 0x03eed0 (0xd2ec bytes) 000763 1778840534.008054 punch from 0x02a8cc to 0x031989 (0x70be bytes) 000764 1778840534.013221 read 0x021252 thru 0x02268e (0x143d bytes) 000765 1778840534.022712 read 0x00d8ba thru 0x0192d1 (0xba18 bytes) 000766 1778840534.023482 trunc from 0x03013c to 0x034a5a (0x491f bytes) 000767 1778840534.030037 mapwrite 0x03e409 thru 0x03ffff (0x1bf7 bytes) 000768 1778840534.072546 mapwrite 0x009547 thru 0x00977f (0x239 bytes) 000769 1778840534.096667 mapwrite 0x0225cb thru 0x024f80 (0x29b6 bytes) 000770 1778840534.119297 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x1e46bc2a (stale) Mirror 15: 0x1e46bc2a (stale) Mirror 16: 0x1e46bc2a (stale) Mirror 17: 0x1e46bc2a (stale) Mirror 18: 0x1e46bc2a (stale) Mirror 19: 0x1e46bc2a (stale) Mirror 20: 0x1e46bc2a (stale) Mirror 21: 0xecddac05 (stale) Mirror 22: 0xb8e8e9e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x1e46bc2a (stale) Mirror 15: 0x1e46bc2a (stale) Mirror 16: 0x1e46bc2a (stale) Mirror 17: 0x1e46bc2a (stale) Mirror 18: 0x1e46bc2a (stale) Mirror 19: 0x1e46bc2a (stale) Mirror 20: 0x1e46bc2a (stale) Mirror 21: 0xecddac05 (stale) Mirror 22: 0xb8e8e9e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000771 1778840534.634489 mapwrite 0x02653e thru 0x030bb6 (0xa679 bytes) 000772 1778840534.681675 mapread 0x032d07 thru 0x037c49 (0x4f43 bytes) 000773 1778840534.689631 punch from 0x01d2e5 to 0x022d4d (0x5a69 bytes) 000774 1778840534.702495 fallocate 0x03a114 thru 0x03ac2e (0xb1b bytes) 000775 1778840534.712686 mapread 0x0386ce thru 0x03d4ad (0x4de0 bytes) 000776 1778840534.729088 write 0x02c39e thru 0x03999e (0xd601 bytes) 000777 1778840534.739762 mapwrite 0x0062ac thru 0x00c15f (0x5eb4 bytes) 000778 1778840534.771098 mapread 0x0170e3 thru 0x01af4b (0x3e69 bytes) 000779 1778840534.786834 trunc from 0x040000 to 0x01da2c (0x225d3 bytes) 000780 1778840534.817440 trunc from 0x01da2d to 0x02d566 (0xfb3a bytes) 000781 1778840534.843036 trunc from 0x009693 to 0x01542c (0xbd9a bytes) 000782 1778840534.848080 mapwrite 0x01a83e thru 0x029d4b (0xf50e bytes) 000783 1778840534.878862 read 0x0080f1 thru 0x009454 (0x1364 bytes) 000784 1778840534.885488 mapread 0x00ecc3 thru 0x013dce (0x510c bytes) 000785 1778840534.898199 write 0x001a7c thru 0x007113 (0x5698 bytes) 000786 1778840534.905558 MIRROR_RESYNC 10 mirrors 000787 1778840536.452511 mapread 0x004b82 thru 0x011732 (0xcbb1 bytes) 000788 1778840536.496620 MIRROR_RESYNC 10 mirrors 000789 1778840537.187777 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) 000790 1778840537.724679 fallocate 0x0308f7 thru 0x03e000 (0xd70a bytes) 000791 1778840537.767661 mapwrite 0x03d9e8 thru 0x03ffff (0x2618 bytes) 000792 1778840537.840689 fallocate 0x020f65 thru 0x0243f7 (0x3493 bytes) 000794 1778840537.850937 write 0x015b86 thru 0x019632 (0x3aad bytes) 000795 1778840537.858587 punch from 0x00f3b2 to 0x01551a (0x6169 bytes) 000796 1778840537.870830 trunc from 0x0167aa to 0x01cd7b (0x65d2 bytes) 000798 1778840537.875649 punch from 0x03abcb to 0x03acd3 (0x109 bytes) 000799 1778840537.881054 trunc from 0x0349a2 to 0x03c061 (0x76c0 bytes) 000800 1778840537.897619 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x328ab3c0 (stale) Mirror 16: 0x328ab3c0 (stale) Mirror 17: 0x328ab3c0 (stale) Mirror 18: 0x328ab3c0 (stale) Mirror 19: 0x328ab3c0 (stale) Mirror 20: 0x328ab3c0 (stale) Mirror 21: 0x328ab3c0 (stale) Mirror 22: 0x328ab3c0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x961c0512 Mirror 23: 0x961c0512 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000801 1778840538.945809 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x328ab3c0 (stale) Mirror 16: 0x328ab3c0 (stale) Mirror 17: 0x328ab3c0 (stale) Mirror 18: 0x328ab3c0 (stale) Mirror 19: 0x328ab3c0 (stale) Mirror 20: 0x328ab3c0 (stale) Mirror 21: 0x328ab3c0 (stale) Mirror 22: 0x328ab3c0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x961c0512 Mirror 23: 0x961c0512 Mirror 24: 0x961c0512 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000803 1778840540.238236 punch from 0x0104a6 to 0x0134f8 (0x3053 bytes) 000804 1778840540.300204 trunc from 0x008319 to 0x010624 (0x830c bytes) 000806 1778840540.314317 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x328ab3c0 (stale) Mirror 17: 0x328ab3c0 (stale) Mirror 18: 0x328ab3c0 (stale) Mirror 19: 0x328ab3c0 (stale) Mirror 20: 0x328ab3c0 (stale) Mirror 21: 0x328ab3c0 (stale) Mirror 22: 0x328ab3c0 (stale) Mirror 23: 0x961c0512 (stale) Mirror 24: 0x961c0512 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x328ab3c0 (stale) Mirror 17: 0x328ab3c0 (stale) Mirror 18: 0x328ab3c0 (stale) Mirror 19: 0x328ab3c0 (stale) Mirror 20: 0x328ab3c0 (stale) Mirror 21: 0x328ab3c0 (stale) Mirror 22: 0x328ab3c0 (stale) Mirror 23: 0x961c0512 (stale) Mirror 24: 0x961c0512 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000807 1778840540.886631 write 0x03ca57 thru 0x03d643 (0xbed bytes) 000808 1778840540.903324 punch from 0x0269f6 to 0x028b76 (0x2181 bytes) 000809 1778840540.910100 read 0x0076d9 thru 0x008fd6 (0x18fe bytes) 000811 1778840540.928547 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x328ab3c0 (stale) Mirror 18: 0x328ab3c0 (stale) Mirror 19: 0x328ab3c0 (stale) Mirror 20: 0x328ab3c0 (stale) Mirror 21: 0x328ab3c0 (stale) Mirror 22: 0x328ab3c0 (stale) Mirror 23: 0x961c0512 (stale) Mirror 24: 0x961c0512 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x328ab3c0 (stale) Mirror 18: 0x328ab3c0 (stale) Mirror 19: 0x328ab3c0 (stale) Mirror 20: 0x328ab3c0 (stale) Mirror 21: 0x328ab3c0 (stale) Mirror 22: 0x328ab3c0 (stale) Mirror 23: 0x961c0512 (stale) Mirror 24: 0x961c0512 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000812 1778840541.576581 mapread 0x03ae4c thru 0x03b6b5 (0x86a bytes) 000813 1778840541.596046 mapwrite 0x0094dd thru 0x01921b (0xfd3f bytes) 000814 1778840541.664273 read 0x027cbd thru 0x0282c0 (0x604 bytes) 000815 1778840541.677631 read 0x009b31 thru 0x016afe (0xcfce bytes) 000817 1778840541.678759 trunc from 0x040000 to 0x01c21f (0x23de0 bytes) 000818 1778840541.713213 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1c220): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c220): Mirror 17: 0x97af21c5 (stale) Mirror 18: 0x97af21c5 (stale) Mirror 19: 0x97af21c5 (stale) Mirror 20: 0x97af21c5 (stale) Mirror 21: 0x97af21c5 (stale) Mirror 22: 0x97af21c5 (stale) Mirror 23: 0xa7dd8bef (stale) Mirror 24: 0xa7dd8bef (stale) CRC-32 checksum value for chunk [0, 0x1c220): Mirror 1: 0xf5daff7e Mirror 25: 0xf5daff7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c220: skipped 000819 1778840542.587583 fallocate 0x03a36a thru 0x03ffff (0x5c96 bytes) 000820 1778840542.638724 punch from 0x010894 to 0x01764b (0x6db8 bytes) 000821 1778840542.645196 read 0x015073 thru 0x01c21f (0x71ad bytes) 000822 1778840542.658657 fallocate 0x0150da thru 0x02228c (0xd1b3 bytes) 000823 1778840542.665640 mapwrite 0x0007ef thru 0x00b206 (0xaa18 bytes) 000824 1778840542.723190 mapread 0x019ce4 thru 0x02228c (0x85a9 bytes) 000825 1778840542.737448 fallocate 0x03f43c thru 0x03ffff (0xbc4 bytes) 000826 1778840542.749532 mapread 0x016e8c thru 0x02228c (0xb401 bytes) 000827 1778840542.752771 read 0x01b83f thru 0x02228c (0x6a4e bytes) 000829 1778840542.754046 fallocate 0x01c51e thru 0x021ef6 (0x59d9 bytes) 000830 1778840542.761490 mapread 0x001f8d thru 0x0075f8 (0x566c bytes) 000831 1778840542.772661 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2228d): Mirror 18: 0x8e1884b6 (stale) Mirror 19: 0x8e1884b6 (stale) Mirror 20: 0x8e1884b6 (stale) Mirror 21: 0x8e1884b6 (stale) Mirror 22: 0x8e1884b6 (stale) Mirror 23: 0xc387eee (stale) Mirror 24: 0xc387eee (stale) Mirror 25: 0x6547a480 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2228d): Mirror 18: 0x8e1884b6 (stale) Mirror 19: 0x8e1884b6 (stale) Mirror 20: 0x8e1884b6 (stale) Mirror 21: 0x8e1884b6 (stale) Mirror 22: 0x8e1884b6 (stale) Mirror 23: 0xc387eee (stale) Mirror 24: 0xc387eee (stale) Mirror 25: 0x6547a480 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2228d: skipped 000832 1778840543.468858 mapread 0x018fb1 thru 0x01f081 (0x60d1 bytes) 000833 1778840543.497166 mapread 0x014710 thru 0x0172f8 (0x2be9 bytes) 000834 1778840543.498264 fallocate 0x00fc00 thru 0x0158cc (0x5ccd bytes) 000835 1778840543.505378 mapwrite 0x01c31a thru 0x022d96 (0x6a7d bytes) 000837 1778840543.605980 mapread 0x0068cb thru 0x00fcbc (0x93f2 bytes) 000839 1778840543.624376 mapwrite 0x019a74 thru 0x01e6ad (0x4c3a bytes) 000840 1778840543.653486 punch from 0x009050 to 0x00faff (0x6ab0 bytes) 000841 1778840543.661655 punch from 0x010215 to 0x011dc2 (0x1bae bytes) 000842 1778840543.672686 fallocate 0x033442 thru 0x036ffc (0x3bbb bytes) 000843 1778840543.683823 read 0x001333 thru 0x008f86 (0x7c54 bytes) 000844 1778840543.688607 mapwrite 0x0389af thru 0x03f4d2 (0x6b24 bytes) 000845 1778840543.755531 fallocate 0x033525 thru 0x03c40d (0x8ee9 bytes) 000846 1778840543.766738 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f4d3): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f4d3): Mirror 18: 0x7e3da0a3 (stale) Mirror 19: 0x7e3da0a3 (stale) Mirror 20: 0x7e3da0a3 (stale) Mirror 21: 0x7e3da0a3 (stale) Mirror 22: 0x7e3da0a3 (stale) Mirror 23: 0xad33a101 (stale) Mirror 24: 0xad33a101 (stale) Mirror 25: 0x5dcd265a (stale) CRC-32 checksum value for chunk [0, 0x3f4d3): Mirror 1: 0x97a1ffd2 Mirror 26: 0x97a1ffd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f4d3: skipped 000847 1778840545.395034 read 0x015853 thru 0x024dc7 (0xf575 bytes) 000848 1778840545.408773 trunc from 0x03f4d3 to 0x00f2bc (0x30216 bytes) 000849 1778840545.462625 punch from 0x0072f3 to 0x00f2bc (0x7fca bytes) 000850 1778840545.470595 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf2bd): Mirror 18: 0x5fe7c7e8 (stale) Mirror 19: 0x5fe7c7e8 (stale) Mirror 20: 0x5fe7c7e8 (stale) Mirror 22: 0x5fe7c7e8 (stale) Mirror 23: 0x5fe7c7e8 (stale) Mirror 24: 0x5fe7c7e8 (stale) Mirror 25: 0x330fc700 (stale) Mirror 26: 0x1c6a3f13 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf2bd): Mirror 18: 0x5fe7c7e8 (stale) Mirror 19: 0x5fe7c7e8 (stale) Mirror 20: 0x5fe7c7e8 (stale) Mirror 22: 0x5fe7c7e8 (stale) Mirror 23: 0x5fe7c7e8 (stale) Mirror 24: 0x5fe7c7e8 (stale) Mirror 25: 0x330fc700 (stale) Mirror 26: 0x1c6a3f13 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf2bd: skipped 000851 1778840545.968449 MIRROR_RESYNC 9 mirrors 000853 1778840547.704063 mapwrite 0x019304 thru 0x01e18f (0x4e8c bytes) 000854 1778840547.787240 trunc from 0x01e190 to 0x039f08 (0x1bd79 bytes) 000855 1778840547.808714 read 0x01872b thru 0x0268ed (0xe1c3 bytes) 000857 1778840547.823749 punch from 0x0139bc to 0x02229b (0xe8e0 bytes) 000858 1778840547.829020 punch from 0x029cf6 to 0x02cd9b (0x30a6 bytes) 000859 1778840547.833735 write 0x0371c7 thru 0x03fdb2 (0x8bec bytes) 000860 1778840547.845433 punch from 0x005f49 to 0x012522 (0xc5da bytes) 000861 1778840547.854850 trunc from 0x0102ed to 0x012d97 (0x2aab bytes) 000862 1778840547.860280 punch from 0x0231df to 0x028009 (0x4e2b bytes) 000863 1778840547.865395 trunc from 0x03fdb3 to 0x0005a9 (0x3f809 bytes) 000864 1778840547.880618 mapwrite 0x00337c thru 0x011ee2 (0xeb67 bytes) 000865 1778840547.947193 fallocate 0x03425a thru 0x03a277 (0x601e bytes) 000866 1778840547.952391 mapwrite 0x03d1be thru 0x03ffff (0x2e42 bytes) 000867 1778840547.989722 fallocate 0x001a84 thru 0x00e5d0 (0xcb4d bytes) 000868 1778840547.995606 trunc from 0x02f0a9 to 0x03d98f (0xe8e7 bytes) 000869 1778840548.005185 trunc from 0x0372f1 to 0x037f5a (0xc6a bytes) 000870 1778840548.018848 fallocate 0x01719c thru 0x01817a (0xfdf bytes) 000871 1778840548.028383 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x29aaf54a (stale) Mirror 20: 0x29aaf54a (stale) Mirror 22: 0x29aaf54a (stale) Mirror 23: 0x29aaf54a (stale) Mirror 24: 0x29aaf54a (stale) Mirror 25: 0x29aaf54a (stale) Mirror 26: 0x29aaf54a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x29aaf54a (stale) Mirror 20: 0x29aaf54a (stale) Mirror 22: 0x29aaf54a (stale) Mirror 23: 0x29aaf54a (stale) Mirror 24: 0x29aaf54a (stale) Mirror 25: 0x29aaf54a (stale) Mirror 26: 0x29aaf54a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000872 1778840548.462908 write 0x03567e thru 0x03ffff (0xa982 bytes) 000873 1778840548.487301 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x29aaf54a (stale) Mirror 20: 0x29aaf54a (stale) Mirror 22: 0x29aaf54a (stale) Mirror 23: 0x29aaf54a (stale) Mirror 24: 0x29aaf54a (stale) Mirror 25: 0x29aaf54a (stale) Mirror 26: 0x29aaf54a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc391541e Mirror 27: 0xc391541e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000874 1778840549.648746 trunc from 0x040000 to 0x0095c6 (0x36a39 bytes) 000875 1778840549.773573 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x95c7): Mirror 18: 0x5c5ba5ce (stale) Mirror 20: 0x5c5ba5ce (stale) Mirror 22: 0x5c5ba5ce (stale) Mirror 24: 0x5c5ba5ce (stale) Mirror 25: 0x5c5ba5ce (stale) Mirror 26: 0x5c5ba5ce (stale) Mirror 27: 0x40a8ae4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x95c7): Mirror 18: 0x5c5ba5ce (stale) Mirror 20: 0x5c5ba5ce (stale) Mirror 22: 0x5c5ba5ce (stale) Mirror 24: 0x5c5ba5ce (stale) Mirror 25: 0x5c5ba5ce (stale) Mirror 26: 0x5c5ba5ce (stale) Mirror 27: 0x40a8ae4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x95c7: skipped 000876 1778840550.396751 punch from 0x0081a9 to 0x0095c6 (0x141e bytes) 000878 1778840550.403636 punch from 0x00741d to 0x0095c6 (0x21aa bytes) 000879 1778840550.410471 write 0x0248a4 thru 0x0324ac (0xdc09 bytes) 000880 1778840550.418486 read 0x018d37 thru 0x025477 (0xc741 bytes) 000882 1778840550.428049 mapwrite 0x0205c4 thru 0x030186 (0xfbc3 bytes) 000883 1778840550.453015 mapwrite 0x01936b thru 0x01c259 (0x2eef bytes) 000884 1778840550.471524 trunc from 0x0324ad to 0x03ee7f (0xc9d3 bytes) 000886 1778840550.496668 read 0x02e85b thru 0x030250 (0x19f6 bytes) 000887 1778840550.497566 trunc from 0x006e28 to 0x00de42 (0x701b bytes) 000888 1778840550.505048 fallocate 0x035c90 thru 0x03ffff (0xa370 bytes) 000889 1778840550.512574 fallocate 0x03035f thru 0x035b5e (0x5800 bytes) 000890 1778840550.523243 mapwrite 0x03cd32 thru 0x03ffff (0x32ce bytes) 000891 1778840550.547728 read 0x02de50 thru 0x03a910 (0xcac1 bytes) 000892 1778840550.548583 write 0x03ba2c thru 0x03ffff (0x45d4 bytes) 000893 1778840550.555455 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x29aaf54a (stale) Mirror 22: 0x29aaf54a (stale) Mirror 24: 0x29aaf54a (stale) Mirror 25: 0x29aaf54a (stale) Mirror 26: 0x29aaf54a (stale) Mirror 27: 0xc391541e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x29aaf54a (stale) Mirror 22: 0x29aaf54a (stale) Mirror 24: 0x29aaf54a (stale) Mirror 25: 0x29aaf54a (stale) Mirror 26: 0x29aaf54a (stale) Mirror 27: 0xc391541e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000894 1778840551.070058 mapwrite 0x01b1b9 thru 0x02286b (0x76b3 bytes) 000895 1778840551.136255 read 0x01c046 thru 0x020822 (0x47dd bytes) 000896 1778840551.137399 trunc from 0x040000 to 0x00504b (0x3afb4 bytes) 000897 1778840551.162719 mapwrite 0x03a4b2 thru 0x03ffff (0x5b4e bytes) 000898 1778840551.207761 mapwrite 0x02084f thru 0x0237f2 (0x2fa4 bytes) 000899 1778840551.236929 mapread 0x020905 thru 0x029d52 (0x944e bytes) 000900 1778840551.247460 trunc from 0x0099f0 to 0x019977 (0xff88 bytes) 000901 1778840551.253708 write 0x02bd19 thru 0x03b849 (0xfb31 bytes) 000902 1778840551.256990 mapread 0x02ddef thru 0x031a6b (0x3c7d bytes) 000903 1778840551.258356 punch from 0x034ca0 to 0x0381f0 (0x3551 bytes) 000904 1778840551.274086 punch from 0x0130be to 0x02100f (0xdf52 bytes) 000905 1778840551.280073 trunc from 0x040000 to 0x0210bc (0x1ef43 bytes) 000906 1778840551.303727 mapread 0x020086 thru 0x0210bc (0x1037 bytes) 000907 1778840551.320397 mapread 0x013941 thru 0x0210bc (0xd77c bytes) 000908 1778840551.338991 trunc from 0x0210bd to 0x038747 (0x1768b bytes) 000909 1778840551.358391 trunc from 0x014f7c to 0x01854c (0x35d1 bytes) 000910 1778840551.368790 MIRROR_RESYNC 7 mirrors 000911 1778840553.173739 mapwrite 0x002e09 thru 0x00bdfb (0x8ff3 bytes) 000912 1778840553.275563 trunc from 0x038748 to 0x004320 (0x34427 bytes) 000913 1778840553.304535 write 0x0371cf thru 0x03ffff (0x8e31 bytes) 000914 1778840553.307445 punch from 0x02a769 to 0x0379d3 (0xd26b bytes) 000915 1778840553.324300 fallocate 0x029829 thru 0x032e7b (0x9653 bytes) 000917 1778840553.328664 punch from 0x01b28f to 0x0241d6 (0x8f48 bytes) 000918 1778840553.333116 trunc from 0x040000 to 0x009b72 (0x3648d bytes) 000919 1778840553.351995 trunc from 0x00147d to 0x002d3d (0x18c1 bytes) 000920 1778840553.358545 punch from 0x006d32 to 0x009b72 (0x2e41 bytes) 000921 1778840553.364353 punch from 0x0068af to 0x009b72 (0x32c4 bytes) 000922 1778840553.369790 trunc from 0x009b73 to 0x03b584 (0x31a12 bytes) 000923 1778840553.394157 trunc from 0x019726 to 0x028d57 (0xf632 bytes) 000925 1778840553.400757 fallocate 0x03f382 thru 0x03ffff (0xc7e bytes) 000926 1778840553.405905 mapread 0x02843b thru 0x0327d6 (0xa39c bytes) 000927 1778840553.436634 trunc from 0x03b585 to 0x009dd4 (0x317b0 bytes) 000928 1778840553.454439 read 0x003d6b thru 0x009dd4 (0x606a bytes) 000929 1778840553.465209 fallocate 0x010d31 thru 0x014555 (0x3825 bytes) 000930 1778840553.471437 read 0x00922f thru 0x010d24 (0x7af6 bytes) 000931 1778840553.488720 mapread 0x00fe1d thru 0x013634 (0x3818 bytes) 000932 1778840553.514668 MIRROR_RESYNC 7 mirrors 000933 1778840555.004482 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) 000934 1778840555.629907 trunc from 0x014556 to 0x01aee5 (0x6990 bytes) 000935 1778840555.728033 fallocate 0x03606e thru 0x03ffff (0x9f92 bytes) 000937 1778840555.738333 write 0x002cba thru 0x0040de (0x1425 bytes) 000938 1778840555.749927 trunc from 0x012999 to 0x013780 (0xde8 bytes) 000939 1778840555.760544 trunc from 0x00e756 to 0x01aee5 (0xc790 bytes) 000940 1778840555.770634 trunc from 0x00d043 to 0x017715 (0xa6d3 bytes) 000941 1778840555.778508 fallocate 0x01d965 thru 0x028895 (0xaf31 bytes) 000942 1778840555.787919 mapread 0x0024d4 thru 0x007fe2 (0x5b0f bytes) 000943 1778840555.804772 read 0x0130b4 thru 0x01aee5 (0x7e32 bytes) 000944 1778840555.820684 mapread 0x0050b2 thru 0x00e0ad (0x8ffc bytes) 000946 1778840555.823803 read 0x017410 thru 0x01aee5 (0x3ad6 bytes) 000947 1778840555.825271 trunc from 0x00a2a1 to 0x019e9d (0xfbfd bytes) 000948 1778840555.838345 mapread 0x017825 thru 0x01aee5 (0x36c1 bytes) 000949 1778840555.868761 mapwrite 0x003460 thru 0x00a28d (0x6e2e bytes) 000950 1778840555.919719 read 0x00905d thru 0x009238 (0x1dc bytes) 000951 1778840555.920720 mapread 0x0091d1 thru 0x00bdc7 (0x2bf7 bytes) 000952 1778840555.921725 punch from 0x005711 to 0x008c34 (0x3524 bytes) 000953 1778840555.933685 punch from 0x00a5c1 to 0x011861 (0x72a1 bytes) 000954 1778840555.952426 trunc from 0x01aee6 to 0x01c494 (0x15af bytes) 000955 1778840556.001722 trunc from 0x01c495 to 0x03f340 (0x22eac bytes) 000956 1778840556.021348 fallocate 0x037574 thru 0x03bea3 (0x4930 bytes) 000957 1778840556.031421 trunc from 0x03f341 to 0x00e512 (0x30e2e bytes) 000958 1778840556.055718 read 0x00daea thru 0x00e512 (0xa29 bytes) 000959 1778840556.074502 mapwrite 0x01d9e1 thru 0x023448 (0x5a68 bytes) 000960 1778840556.157753 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23449): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23449): Mirror 22: 0xa39ac6cf (stale) Mirror 24: 0xa39ac6cf (stale) Mirror 25: 0xa39ac6cf (stale) Mirror 26: 0xa39ac6cf (stale) Mirror 27: 0xa39ac6cf (stale) CRC-32 checksum value for chunk [0, 0x23449): Mirror 1: 0x3f94e9ca Mirror 28: 0x3f94e9ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23449: skipped 000961 1778840557.032234 mapwrite 0x03fa1a thru 0x03ffff (0x5e6 bytes) 000962 1778840557.128727 read 0x018bf2 thru 0x01f0ae (0x64bd bytes) 000963 1778840557.143464 mapwrite 0x01f660 thru 0x02b09d (0xba3e bytes) 000964 1778840557.203355 mapwrite 0x03d203 thru 0x03ffff (0x2dfd bytes) 000965 1778840557.237105 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xed2d0b2b (stale) Mirror 24: 0xed2d0b2b (stale) Mirror 25: 0xed2d0b2b (stale) Mirror 26: 0xed2d0b2b (stale) Mirror 27: 0xed2d0b2b (stale) Mirror 28: 0xec06b54c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x417d716d Mirror 29: 0x417d716d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000966 1778840558.209810 mapwrite 0x03b66c thru 0x03ffff (0x4994 bytes) 000967 1778840558.334004 mapread 0x01f1a4 thru 0x024d03 (0x5b60 bytes) 000968 1778840558.349669 trunc from 0x038a04 to 0x03c209 (0x3806 bytes) 000969 1778840558.364214 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xed2d0b2b (stale) Mirror 24: 0xed2d0b2b (stale) Mirror 25: 0xed2d0b2b (stale) Mirror 26: 0xed2d0b2b (stale) Mirror 27: 0xed2d0b2b (stale) Mirror 29: 0x417d716d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xed2d0b2b (stale) Mirror 24: 0xed2d0b2b (stale) Mirror 25: 0xed2d0b2b (stale) Mirror 26: 0xed2d0b2b (stale) Mirror 27: 0xed2d0b2b (stale) Mirror 29: 0x417d716d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000970 1778840559.127500 punch from 0x00fc51 to 0x01cd81 (0xd131 bytes) 000971 1778840559.141496 read 0x01e22f thru 0x020cdc (0x2aae bytes) 000972 1778840559.150873 trunc from 0x01b11f to 0x01c726 (0x1608 bytes) 000973 1778840559.160498 mapwrite 0x00ffb7 thru 0x016c18 (0x6c62 bytes) 000974 1778840559.213111 write 0x03b1b6 thru 0x03ffff (0x4e4a bytes) 000976 1778840559.260051 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xed2d0b2b (stale) Mirror 24: 0xed2d0b2b (stale) Mirror 25: 0xed2d0b2b (stale) Mirror 26: 0xed2d0b2b (stale) Mirror 27: 0xed2d0b2b (stale) Mirror 29: 0x417d716d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x42fffcc7 Mirror 30: 0x42fffcc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000977 1778840560.557060 fallocate 0x02279f thru 0x024760 (0x1fc2 bytes) 000978 1778840560.604230 read 0x034842 thru 0x03ee41 (0xa600 bytes) 000980 1778840560.620978 MIRROR_RESYNC 8 mirrors 000981 1778840563.642948 mapread 0x01d9f2 thru 0x022fb4 (0x55c3 bytes) 000982 1778840563.664883 fallocate 0x031c58 thru 0x03906c (0x7415 bytes) 000983 1778840563.714432 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x42fffcc7 (stale) Mirror 24: 0x42fffcc7 (stale) Mirror 25: 0x42fffcc7 (stale) Mirror 26: 0x42fffcc7 (stale) Mirror 27: 0x42fffcc7 (stale) Mirror 29: 0x42fffcc7 (stale) Mirror 30: 0x42fffcc7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x42fffcc7 Mirror 31: 0x42fffcc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000985 1778840565.005110 fallocate 0x0250c7 thru 0x02eb87 (0x9ac1 bytes) 000986 1778840565.052584 write 0x033592 thru 0x03d06c (0x9adb bytes) 000987 1778840565.080304 mapwrite 0x022ad5 thru 0x02bb95 (0x90c1 bytes) 000988 1778840565.190334 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1fd:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x42fffcc7 (stale) Mirror 24: 0x42fffcc7 (stale) Mirror 25: 0x42fffcc7 (stale) Mirror 26: 0x42fffcc7 (stale) Mirror 27: 0x42fffcc7 (stale) Mirror 29: 0x42fffcc7 (stale) Mirror 31: 0x42fffcc7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x42fffcc7 (stale) Mirror 24: 0x42fffcc7 (stale) Mirror 25: 0x42fffcc7 (stale) Mirror 26: 0x42fffcc7 (stale) Mirror 27: 0x42fffcc7 (stale) Mirror 29: 0x42fffcc7 (stale) Mirror 31: 0x42fffcc7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000989 1778840565.879713 mapread 0x013c98 thru 0x01a83c (0x6ba5 bytes) 000990 1778840565.906843 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x42fffcc7 (stale) Mirror 24: 0x42fffcc7 (stale) Mirror 25: 0x42fffcc7 (stale) Mirror 26: 0x42fffcc7 (stale) Mirror 27: 0x42fffcc7 (stale) Mirror 29: 0x42fffcc7 (stale) Mirror 31: 0x42fffcc7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9f221e4c Mirror 32: 0x9f221e4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000992 1778840566.972408 punch from 0x015ef4 to 0x01947a (0x3587 bytes) 000993 1778840567.016973 trunc from 0x039c1e to 0x03ffff (0x63e2 bytes) 000994 1778840567.022823 trunc from 0x040000 to 0x0353a2 (0xac5d bytes) 000996 1778840567.054815 write 0x03ca55 thru 0x03e705 (0x1cb1 bytes) 000997 1778840567.057490 write 0x029024 thru 0x0314dd (0x84ba bytes) 000998 1778840567.072516 trunc from 0x03e706 to 0x009de7 (0x3491e bytes) 000999 1778840567.097947 fallocate 0x02f88f thru 0x0371f1 (0x7963 bytes) All operations completed A-OK! PASS 70a (105s) == sanity-flr test 71: check mirror extend parent fid ==== 06:22:51 (1778840571) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x240:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 06:22:59 (1778840579) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162192 s, 12.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103951 s, 10.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xee:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 06:23:06 (1778840586) Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre2 Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6352768 .. Extending file size to 8387168 .. Extending file size to 8676960 .. Extending file size to 8783136 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9371200 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8630880 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8875936 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61202 61203 61204 61207 61210 done 192.168.201.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg141-client.virtnet /mnt/lustre2 (opts:) 192.168.201.141@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg141-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:24:24 (1778840664) Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre2 Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6163968 .. Extending file size to 7683360 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8922336 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9002464 .. Extending file size to 9181184 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62755 62756 62757 62759 62760 done 192.168.201.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg141-client.virtnet /mnt/lustre2 (opts:) 192.168.201.141@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg141-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (75s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:25:39 (1778840739) Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18824 3588196 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24772 3582248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43596 7170444 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18824 3588196 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24772 3582248 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43596 7170444 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0721013 s, 29.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165471 s, 12.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd5b52a3a (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg141-client.virtnet /mnt/lustre2 (opts:) PASS 200c (13s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:25:53 (1778840753) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165801 s, 12.6 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 06:25:58 (1778840758) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x36:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x103:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:26:04 (1778840764) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0795547 s, 26.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:26:10 (1778840770) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0681634 s, 30.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x104:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x104:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x105:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:26:16 (1778840776) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127191 s, 16.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x107:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:26:46 (1778840806) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg141-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539609 s, 19.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a9:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:27:29 (1778840849) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg141-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0800757 s, 26.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x108:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x108:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:28:15 (1778840895) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg141-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg141-server: Pool lustre.test_204f created Adding targets to pool oleg141-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg141-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0916296 s, 22.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x109:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x109:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg141-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg141-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg141-server: Pool lustre.test_204f destroyed PASS 204f (66s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:29:21 (1778840961) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:29:26 (1778840966) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:29:31 (1778840971) oleg141-server: Pool lustre.test_206 created Adding targets to pool oleg141-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg141-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg141-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg141-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg141-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:29:56 (1778840996) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549025 s, 19.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312821 s, 33.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x22c58fac Mirror 3: 0x22c58fac PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:30:05 (1778841005) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:30:08 (1778841008) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:30:11 (1778841011) Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0507799 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x25a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2230000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2230000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2230000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x910000 Read this (4 bytes): BBBB 192.168.201.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg141-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:30:44 (1778841044) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0226196 s, 362 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:30:49 (1778841049) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543106 s, 19.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x57 lmm_fid: [0x200000405:0x57:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x57 lmm_fid: [0x200000405:0x57:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-15 06:30:51.000000000 -0400 Modify: 2026-05-15 06:30:51.000000000 -0400 Change: 2026-05-15 06:30:52.000000000 -0400 Birth: 2026-05-15 06:30:50.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:30:57 (1778841057) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198712 s, 52.8 MB/s oleg141-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg141-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg141-server: O/280000401/d20/134: File not found by ext2_lookup PASS 210b (18s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:31:15 (1778841075) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.14891 s, 275 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0278066 s, 147 kB/s size after second write -rw-r--r-- 1 root root 4096 May 15 06:31 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 15 06:31 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:31:21 (1778841081) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:31:27 (1778841087) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1746 sec =========== 06:31:34 (1778841094) === sanity-flr: start cleanup 06:31:35 (1778841095) === === sanity-flr: finish cleanup 06:31:41 (1778841101) ===