-----============= acceptance-small: sanity-flr ============----- Sat Aug 15 16:49:13 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg419-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 16:49:38 (1786826978) === oleg419-client.virtnet: executing check_config_client /mnt/lustre oleg419-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg419-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa06510323000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa06510323000.idle_timeout=debug disable quota as required oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 16:50:25 (1786827025) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8231] == sanity-flr test 0a: lfs mirror create with -N option == 16:50:28 (1786827028) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:50:47 (1786827047) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:50:50 (1786827050) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:50:55 (1786827055) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:51:24 (1786827084) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:51:28 (1786827088) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 16:51:31 (1786827091) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa06510323000.stats=clear osc.lustre-OST0001-osc-ffffa06510323000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:51:39 (1786827099) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:51:50 (1786827110) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:51:58 (1786827118) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:52:01 (1786827121) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:52:10 (1786827130) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 16:52:19 (1786827139) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:52:27 (1786827147) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:52:34 (1786827154) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:52:41 (1786827161) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:52:52 (1786827172) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:52:59 (1786827179) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.976883 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131244 s, 8.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:53:09 (1786827189) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.581618 s, 19.8 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:53:20 (1786827200) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:53:32 (1786827212) Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa06510323000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa06510323000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg419-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa06510323000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa06510323000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (139s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:55:51 (1786827351) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.181023 s, 17.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa06510323000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa06510323000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (54s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:56:45 (1786827405) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa06510323000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa06510323000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa06510323000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa06510323000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:58:05 (1786827485) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.9686 s, 13.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 15 16:58 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.160s user 0m0.000s sys 0m0.186s Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa06510323000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa06510323000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.121s user 0m0.001s sys 0m0.186s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa06510323000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa06510323000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.556s user 0m0.002s sys 0m0.190s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg419-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa06510323000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa06510323000.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:59:42 (1786827582) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:59:46 (1786827586) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:59:50 (1786827590) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 16:59:55 (1786827595) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 17:00:05 (1786827605) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.513101 s, 24.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:00:19 (1786827619) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900802 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498652 s, 21.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:00:26 (1786827626) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:00:35 (1786827635) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00406678 s, 252 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00158909 s, 644 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00553144 s, 185 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00204252 s, 501 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 17:00:44 (1786827644) create test files with size 15747798 .. 8110310affd622e2ddeac514618ac7c8 - abd67d64ea9971293282b109879e6e4a - d0585b3c9ec97be67c688a89f9efcf29 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 17:01:07 (1786827667) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:01:23 (1786827683) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:01:31 (1786827691) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105394 s, 9.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380012 s, 27.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 17:01:41 (1786827701) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124518 s, 16.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.405074 s, 10.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.315492 s, 6.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 17:02:01 (1786827721) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 17:02:05 (1786827725) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:02:08 (1786827728) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229042 s, 45.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 17:02:18 (1786827738) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.539425 s, 5.8 MB/s PASS 44a (17s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:02:35 (1786827755) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (12s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:02:49 (1786827769) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.426003 s, 24.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:03:03 (1786827783) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa065118cd800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.422096 s, 24.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa06507629000.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:03:23 (1786827803) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:04:07 (1786827847) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:04:46 (1786827886) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 17:04:50 (1786827890) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.234908 s, 13.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.123631 s, 25.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4d6f2fdd (nosync,stale) Mirror 2: 0x4d6f2fdd (nosync,stale) ** verify mirror 2 doesn't change original checksum: 42aef149f2ade203544494d24ddbad40 - mirror 1 checksum: 2367d169f4f3de6f68e4e023ab306d84 - mirror 2 checksum: 42aef149f2ade203544494d24ddbad40 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T17:04:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T17:04:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 42aef149f2ade203544494d24ddbad40 - mirror 1 checksum: 2367d169f4f3de6f68e4e023ab306d84 - mirror 2 checksum: 2367d169f4f3de6f68e4e023ab306d84 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T17:04:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T17:04:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 17:05:00 (1786827900) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253479 s, 41.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:05:07 (1786827907) Failing mds1 on oleg419-server Stopping /mnt/lustre-mds1 (opts:) on oleg419-server 17:05:18 (1786827918) shut down facet: mds1 facet_host: oleg419-server facet_failover_host: oleg419-server Failover mds1 to oleg419-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 17:05:41 (1786827941) targets are mounted 17:05:41 (1786827941) facet_failover done oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616938 s, 17.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } PASS 50A (52s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:05:59 (1786827959) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0309029 s, 663 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0131253 s, 1.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:06:10 (1786827970) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0638057 s, 321 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0114833 s, 1.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:06:20 (1786827980) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0102207 s, 1.6 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:06:27 (1786827987) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0549395 s, 373 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:06:35 (1786827995) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00824249 s, 497 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0424583 s, 96.5 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:06:43 (1786828003) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:07:05 (1786828025) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:07:35 (1786828055) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:07:58 (1786828078) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:08:24 (1786828104) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 17:08:32 (1786828112) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57612 mirror split pid 57614 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57612 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57614 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 17:09:40 (1786828180) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 8182 skipping zero length zero range 000002 1786828183.019948 write 0x03280f thru 0x033ef6 (0x16e8 bytes) fallocating to largest ever: 0x14c29 000003 1786828183.024533 fallocate 0x009be1 thru 0x014c28 (0xb048 bytes) 000005 1786828183.033257 mapread 0x0239df thru 0x02b6c7 (0x7ce9 bytes) 000006 1786828183.063576 trunc from 0x033ef7 to 0x00b710 (0x287e6 bytes) 000007 1786828183.086722 punch from 0x00ae93 to 0x00b710 (0x87e bytes) 000008 1786828183.092141 punch from 0x006fe2 to 0x00b710 (0x472f bytes) 000009 1786828183.096889 mapwrite 0x019525 thru 0x01fdb2 (0x688e bytes) 000010 1786828183.167452 punch from 0x00b53d to 0x00db0a (0x25ce bytes) fallocating to largest ever: 0x1fcf3 000011 1786828183.175748 fallocate 0x01f05e thru 0x01fcf2 (0xc95 bytes) 000012 1786828183.198095 write 0x015fde thru 0x01a6c0 (0x46e3 bytes) 000013 1786828183.203074 trunc from 0x01fdb3 to 0x00c113 (0x13c9f bytes) 000014 1786828183.234872 mapread 0x0038f3 thru 0x00c113 (0x8821 bytes) 000015 1786828183.264647 punch from 0x0004d5 to 0x008bcb (0x86f7 bytes) 000018 1786828183.275957 trunc from 0x003597 to 0x007f27 (0x4991 bytes) 000019 1786828183.287128 write 0x011a74 thru 0x0139db (0x1f68 bytes) 000020 1786828183.291228 write 0x038f9c thru 0x03ffff (0x7064 bytes) 000021 1786828183.297222 mapwrite 0x0360c8 thru 0x03c67c (0x65b5 bytes) 000022 1786828183.348317 trunc from 0x0200fa to 0x022c26 (0x2b2d bytes) 000023 1786828183.366673 punch from 0x0170fc to 0x01c369 (0x526e bytes) truncating to largest ever: 0x3bef4 000025 1786828183.380159 trunc from 0x040000 to 0x03bef3 (0x410c bytes) 000026 1786828183.404322 read 0x036919 thru 0x03bef3 (0x55db bytes) 000028 1786828183.407921 punch from 0x01c186 to 0x0239f6 (0x7871 bytes) 000029 1786828183.415208 fallocate 0x022a8a thru 0x027c1b (0x5192 bytes) 000030 1786828183.421749 fallocate 0x01f1f2 thru 0x0230cb (0x3eda bytes) 000031 1786828183.429594 write 0x022e54 thru 0x02f503 (0xc6b0 bytes) 000032 1786828183.439933 mapwrite 0x032aa8 thru 0x03dad4 (0xb02d bytes) 000033 1786828183.521645 mapread 0x02e391 thru 0x0339a8 (0x5618 bytes) 000034 1786828183.525192 trunc from 0x03dad5 to 0x02005a (0x1da7a bytes) 000035 1786828183.544158 punch from 0x019613 to 0x01b701 (0x20ef bytes) 000036 1786828183.549882 fallocate 0x02c075 thru 0x03aafd (0xea89 bytes) 000037 1786828183.555948 mapwrite 0x00b01a thru 0x0178df (0xc8c6 bytes) 000040 1786828183.630189 trunc from 0x03aafe to 0x034a0e (0x60ef bytes) 000041 1786828183.648130 fallocate 0x008486 thru 0x00de86 (0x5a01 bytes) 000042 1786828183.658158 MIRROR_EXTEND to 2 mirrors 000043 1786828184.569904 write 0x00b81b thru 0x00ee35 (0x361b bytes) 000044 1786828184.619252 mapread 0x031925 thru 0x0331c8 (0x18a4 bytes) 000045 1786828184.642098 mapread 0x0272a2 thru 0x028f1a (0x1c79 bytes) 000046 1786828184.653157 mapwrite 0x011c9f thru 0x015d63 (0x40c5 bytes) 000047 1786828184.699456 fallocate 0x01b316 thru 0x023538 (0x8223 bytes) 000048 1786828184.713284 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000049 1786828184.891735 mapread 0x009f7f thru 0x0157b7 (0xb839 bytes) 000052 1786828184.968400 MIRROR_EXTEND to 2 mirrors 000053 1786828185.864660 MIRROR_RESYNC 2 mirrors 000055 1786828186.416974 punch from 0x02f098 to 0x02f5fe (0x567 bytes) 000056 1786828186.448906 fallocate 0x00be0d thru 0x00de49 (0x203d bytes) 000057 1786828186.456894 write 0x012226 thru 0x015dfc (0x3bd7 bytes) 000058 1786828186.474745 trunc from 0x034a0f to 0x022cad (0x11d61 bytes) 000059 1786828186.503685 write 0x00981f thru 0x00d3cd (0x3baf bytes) 000060 1786828186.517072 trunc from 0x00d2ce to 0x010864 (0x3597 bytes) 000062 1786828186.539893 MIRROR_RESYNC 2 mirrors 000063 1786828187.337926 read 0x001684 thru 0x009a14 (0x8391 bytes) 000064 1786828187.362931 write 0x01e6de thru 0x0200c6 (0x19e9 bytes) 000065 1786828187.450367 write 0x030951 thru 0x03bf1e (0xb5ce bytes) 000067 1786828187.474354 trunc from 0x02115c to 0x02e8ad (0xd752 bytes) 000069 1786828187.492327 mapwrite 0x001393 thru 0x007431 (0x609f bytes) 000070 1786828187.601204 mapwrite 0x02c5e2 thru 0x03612c (0x9b4b bytes) 000071 1786828187.661562 read 0x0396df thru 0x039a7f (0x3a1 bytes) 000072 1786828187.663101 read 0x006303 thru 0x0106cf (0xa3cd bytes) 000073 1786828187.665190 read 0x00f98d thru 0x01d268 (0xd8dc bytes) 000075 1786828187.683633 fallocate 0x02239f thru 0x02ad58 (0x89ba bytes) 000076 1786828187.689980 mapwrite 0x03b0ca thru 0x03ffff (0x4f36 bytes) 000077 1786828187.769031 trunc from 0x040000 to 0x022127 (0x1ded8 bytes) 000079 1786828187.806673 fallocate 0x0116c3 thru 0x014b8f (0x34cd bytes) 000080 1786828187.814916 fallocate 0x023bb8 thru 0x031b6d (0xdfb6 bytes) 000081 1786828187.831571 mapwrite 0x000a43 thru 0x00d73b (0xccf9 bytes) 000082 1786828187.923738 write 0x03ad9c thru 0x03ffff (0x5264 bytes) 000083 1786828187.932752 trunc from 0x01b054 to 0x01d40e (0x23bb bytes) 000084 1786828187.953569 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000085 1786828188.168948 trunc from 0x040000 to 0x02c602 (0x139fd bytes) 000086 1786828188.193624 trunc from 0x02c603 to 0x0225d3 (0xa02f bytes) 000087 1786828188.217546 read 0x01b80f thru 0x02004f (0x4841 bytes) 000088 1786828188.225611 read 0x007d32 thru 0x008f7b (0x124a bytes) 000089 1786828188.236870 trunc from 0x0225d4 to 0x01194c (0x10c87 bytes) 000090 1786828188.258917 trunc from 0x0011a6 to 0x00fcd1 (0xeb2c bytes) 000091 1786828188.272699 punch from 0x010eba to 0x01194c (0xa93 bytes) 000092 1786828188.282039 write 0x029733 thru 0x033d74 (0xa642 bytes) 000093 1786828188.285529 punch from 0x01b234 to 0x025b0c (0xa8d9 bytes) 000094 1786828188.290189 write 0x0139dc thru 0x01ae00 (0x7425 bytes) 000095 1786828188.302048 trunc from 0x00e6c2 to 0x00fefa (0x1839 bytes) 000096 1786828188.307255 mapwrite 0x0240ce thru 0x0289f5 (0x4928 bytes) 000097 1786828188.327279 punch from 0x02f911 to 0x033d74 (0x4464 bytes) 000098 1786828188.332692 trunc from 0x003f2c to 0x01338f (0xf464 bytes) 000099 1786828188.343533 MIRROR_EXTEND to 2 mirrors 000100 1786828189.102240 punch from 0x029c8b to 0x03105a (0x73d0 bytes) 000102 1786828189.162564 read 0x029140 thru 0x033d74 (0xac35 bytes) 000103 1786828189.170837 read 0x01b8b6 thru 0x01e59d (0x2ce8 bytes) 000105 1786828189.192101 mapwrite 0x0225c2 thru 0x02d2e7 (0xad26 bytes) 000106 1786828189.249134 MIRROR_RESYNC 2 mirrors 000107 1786828190.150913 read 0x01061b thru 0x018cee (0x86d4 bytes) 000108 1786828190.160904 punch from 0x023b27 to 0x02913e (0x5618 bytes) 000109 1786828190.207246 read 0x033636 thru 0x033d74 (0x73f bytes) 000110 1786828190.229552 trunc from 0x033d75 to 0x01c6e3 (0x17691 bytes) 000111 1786828190.294624 trunc from 0x01c6e4 to 0x01d635 (0xf52 bytes) 000112 1786828190.317271 mapwrite 0x0366c4 thru 0x03ffff (0x993c bytes) 000113 1786828190.359508 write 0x01bdad thru 0x01cf9c (0x11f0 bytes) zero_range to largest ever: 0x40000 000115 1786828190.361867 trunc from 0x03c514 to 0x03ffff (0x3aec bytes) 000118 1786828190.371349 read 0x036c25 thru 0x03e8b4 (0x7c90 bytes) 000119 1786828190.388986 trunc from 0x040000 to 0x02d1ae (0x12e51 bytes) 000120 1786828190.426572 MIRROR_RESYNC 2 mirrors 000121 1786828191.217055 mapread 0x023463 thru 0x02d1ae (0x9d4c bytes) 000122 1786828191.249217 trunc from 0x02d1af to 0x03c83b (0xf68d bytes) 000124 1786828191.325977 trunc from 0x00ca58 to 0x011812 (0x4dbb bytes) 000125 1786828191.332157 punch from 0x03be36 to 0x03c83b (0xa06 bytes) 000126 1786828191.343970 trunc from 0x03c83c to 0x014c0a (0x27c31 bytes) 000127 1786828191.370999 mapread 0x0001b4 thru 0x002fdb (0x2e28 bytes) 000128 1786828191.382568 punch from 0x00f4a7 to 0x014c0a (0x5764 bytes) 000130 1786828191.389353 mapread 0x00d384 thru 0x014c0a (0x7887 bytes) 000132 1786828191.402848 fallocate 0x005f5a thru 0x00fa33 (0x9ada bytes) 000133 1786828191.409143 write 0x00d198 thru 0x01a962 (0xd7cb bytes) 000135 1786828191.412951 fallocate 0x02e995 thru 0x036943 (0x7faf bytes) 000136 1786828191.435602 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36944): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36944): Mirror 2: 0x434941d6 (stale) CRC-32 checksum value for chunk [0, 0x36944): Mirror 1: 0xe4ab0fa Mirror 3: 0xe4ab0fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36944: skipped 000137 1786828192.472534 mapread 0x02a452 thru 0x02a83c (0x3eb bytes) 000138 1786828192.481828 write 0x0128cb thru 0x0185b0 (0x5ce6 bytes) 000139 1786828192.571663 trunc from 0x036944 to 0x03777c (0xe39 bytes) 000140 1786828192.669590 trunc from 0x03777d to 0x02dd16 (0x9a66 bytes) 000141 1786828192.704570 mapwrite 0x00b4e7 thru 0x014452 (0x8f6c bytes) 000142 1786828192.793594 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dd17): Mirror 2: 0x1c9211e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd17): Mirror 2: 0x1c9211e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd17: skipped 000143 1786828193.216206 mapwrite 0x033e40 thru 0x038361 (0x4522 bytes) 000144 1786828193.309655 mapwrite 0x02320d thru 0x02f713 (0xc507 bytes) 000145 1786828193.399861 mapwrite 0x00a8de thru 0x014010 (0x9733 bytes) 000146 1786828193.466504 read 0x012862 thru 0x015df2 (0x3591 bytes) 000147 1786828193.468099 mapwrite 0x0300f8 thru 0x03470f (0x4618 bytes) 000148 1786828193.515584 mapwrite 0x0339a9 thru 0x0382d6 (0x492e bytes) 000149 1786828193.537022 read 0x02ca0d thru 0x033fc1 (0x75b5 bytes) 000150 1786828193.538638 trunc from 0x007ecf to 0x017d9c (0xfece bytes) 000151 1786828193.544848 punch from 0x0055de to 0x00e1bc (0x8bdf bytes) 000152 1786828193.551610 punch from 0x018c69 to 0x025304 (0xc69c bytes) 000153 1786828193.557335 read 0x02981a thru 0x038361 (0xeb48 bytes) 000154 1786828193.563260 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000155 1786828193.749712 punch from 0x02b8bb to 0x02f0a3 (0x37e9 bytes) 000156 1786828193.757934 mapwrite 0x02395a thru 0x028982 (0x5029 bytes) 000157 1786828193.810178 punch from 0x009271 to 0x00b7b0 (0x2540 bytes) 000158 1786828193.816739 trunc from 0x03747e to 0x038361 (0xee4 bytes) 000159 1786828193.824252 read 0x009c81 thru 0x00c26e (0x25ee bytes) 000161 1786828193.839472 read 0x02c0f5 thru 0x0308be (0x47ca bytes) 000162 1786828193.886712 fallocate 0x01e6fc thru 0x029133 (0xaa38 bytes) 000163 1786828193.897634 trunc from 0x038362 to 0x02b440 (0xcf21 bytes) 000164 1786828193.925477 write 0x031eb1 thru 0x03a151 (0x82a1 bytes) 000165 1786828193.927230 mapread 0x0049eb thru 0x00915a (0x4770 bytes) 000166 1786828193.940408 mapread 0x0388b5 thru 0x03a151 (0x189d bytes) 000167 1786828193.945723 trunc from 0x031185 to 0x03a151 (0x8fcd bytes) 000168 1786828193.965842 write 0x01ee38 thru 0x02495d (0x5b26 bytes) 000171 1786828193.982758 mapwrite 0x02a8a3 thru 0x02b76a (0xec8 bytes) 000172 1786828194.004407 trunc from 0x003964 to 0x00399f (0x3c bytes) 000173 1786828194.011915 write 0x032fbd thru 0x039936 (0x697a bytes) 000174 1786828194.024274 write 0x00e811 thru 0x013889 (0x5079 bytes) 000175 1786828194.027021 trunc from 0x03a152 to 0x02b8de (0xe873 bytes) 000176 1786828194.058608 trunc from 0x01f607 to 0x02b8de (0xc2d8 bytes) 000177 1786828194.067483 read 0x01a465 thru 0x01dfb5 (0x3b51 bytes) 000178 1786828194.068712 mapwrite 0x02ed72 thru 0x03ba7c (0xcd0b bytes) 000179 1786828194.143258 MIRROR_EXTEND to 2 mirrors 000180 1786828194.904384 fallocate 0x03247a thru 0x03fc04 (0xd78b bytes) 000181 1786828194.955753 punch from 0x021b45 to 0x0299a9 (0x7e65 bytes) 000182 1786828194.961579 write 0x02c9d4 thru 0x02de9c (0x14c9 bytes) 000184 1786828194.977099 trunc from 0x03ba7d to 0x011d8f (0x29ced bytes) 000185 1786828195.024714 mapread 0x00981e thru 0x011123 (0x7906 bytes) 000186 1786828195.049890 mapread 0x00f03e thru 0x011d8f (0x2d52 bytes) 000187 1786828195.051179 mapwrite 0x0392e3 thru 0x03a77d (0x149b bytes) 000188 1786828195.094334 mapwrite 0x03a1e6 thru 0x03ffff (0x5e1a bytes) 000189 1786828195.147458 fallocate 0x01c006 thru 0x02b8a1 (0xf89c bytes) 000190 1786828195.154286 write 0x001859 thru 0x007a79 (0x6221 bytes) 000191 1786828195.160413 write 0x007480 thru 0x00f66b (0x81ec bytes) 000192 1786828195.162489 trunc from 0x040000 to 0x03c45a (0x3ba5 bytes) 000193 1786828195.183279 punch from 0x0338f4 to 0x038338 (0x4a45 bytes) 000194 1786828195.189403 mapwrite 0x013f2f thru 0x020cbe (0xcd90 bytes) 000195 1786828195.237068 punch from 0x00b72c to 0x015606 (0x9edb bytes) 000196 1786828195.244530 mapwrite 0x0038db thru 0x0053ec (0x1b12 bytes) 000197 1786828195.261488 mapwrite 0x03496e thru 0x03ffff (0xb692 bytes) 000198 1786828195.359812 mapwrite 0x01d6be thru 0x025fcb (0x890e bytes) 000199 1786828195.457285 punch from 0x00da40 to 0x01ccbf (0xf280 bytes) 000201 1786828195.469086 mapwrite 0x0191ca thru 0x0262a7 (0xd0de bytes) 000202 1786828195.516968 punch from 0x012f56 to 0x022034 (0xf0df bytes) 000203 1786828195.522870 trunc from 0x02c07e to 0x036d04 (0xac87 bytes) 000204 1786828195.529072 trunc from 0x0315e7 to 0x03ffff (0xea19 bytes) 000205 1786828195.534614 punch from 0x002c2d to 0x00ffbf (0xd393 bytes) 000206 1786828195.542278 mapwrite 0x03adc8 thru 0x03fdf6 (0x502f bytes) 000207 1786828195.569912 trunc from 0x01a973 to 0x01bf28 (0x15b6 bytes) 000208 1786828195.574861 fallocate 0x032a7c thru 0x03f7f9 (0xcd7e bytes) 000209 1786828195.580264 read 0x00c017 thru 0x01722a (0xb214 bytes) 000210 1786828195.590365 read 0x01b519 thru 0x029059 (0xdb41 bytes) 000211 1786828195.600583 write 0x033e3b thru 0x03fa8d (0xbc53 bytes) 000213 1786828195.604370 write 0x029fa0 thru 0x02c19f (0x2200 bytes) 000214 1786828195.611360 MIRROR_RESYNC 2 mirrors 000216 1786828196.447291 fallocate 0x00bf16 thru 0x016cc2 (0xadad bytes) 000217 1786828196.512394 punch from 0x01292d to 0x012a29 (0xfd bytes) 000218 1786828196.518303 mapread 0x038384 thru 0x03ffff (0x7c7c bytes) 000219 1786828196.542484 fallocate 0x00c5c4 thru 0x018cd6 (0xc713 bytes) 000221 1786828196.552168 trunc from 0x003de6 to 0x00c1dc (0x83f7 bytes) 000222 1786828196.560330 fallocate 0x002ab4 thru 0x0051cc (0x2719 bytes) 000223 1786828196.575448 read 0x02614c thru 0x032535 (0xc3ea bytes) 000224 1786828196.591332 mapwrite 0x03e5c4 thru 0x03ffff (0x1a3c bytes) 000225 1786828196.610508 trunc from 0x00dd54 to 0x0184ee (0xa79b bytes) 000226 1786828196.616481 read 0x037444 thru 0x03ab9f (0x375c bytes) 000228 1786828196.618996 fallocate 0x03d3b2 thru 0x03ffff (0x2c4e bytes) 000229 1786828196.625435 mapread 0x030d7c thru 0x03b174 (0xa3f9 bytes) 000230 1786828196.627661 trunc from 0x040000 to 0x0262c4 (0x19d3b bytes) 000231 1786828196.650425 punch from 0x005549 to 0x00f99c (0xa454 bytes) 000233 1786828196.661646 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x262c5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x262c5): Mirror 2: 0x355f8757 (stale) CRC-32 checksum value for chunk [0, 0x262c5): Mirror 1: 0x355f8757 Mirror 3: 0x355f8757 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x262c5: skipped 000235 1786828197.851290 mapread 0x00a3af thru 0x01364c (0x929e bytes) 000236 1786828197.889065 write 0x01332b thru 0x01b8fa (0x85d0 bytes) 000237 1786828197.956476 mapwrite 0x00dd68 thru 0x00f4d6 (0x176f bytes) 000238 1786828198.068514 trunc from 0x0262c5 to 0x01d34a (0x8f7a bytes) 000239 1786828198.120145 trunc from 0x01d34b to 0x0003d9 (0x1cf71 bytes) 000240 1786828198.151640 mapwrite 0x0077ff thru 0x011019 (0x981b bytes) 000241 1786828198.261699 mapread 0x00536a thru 0x009bb2 (0x4849 bytes) 000243 1786828198.267390 mapwrite 0x0325e1 thru 0x037fbf (0x59df bytes) 000244 1786828198.407955 write 0x00add8 thru 0x0118a9 (0x6ad2 bytes) 000245 1786828198.409481 read 0x00558e thru 0x015122 (0xfb95 bytes) 000246 1786828198.435270 read 0x015e82 thru 0x016e01 (0xf80 bytes) 000248 1786828198.481449 write 0x03f41b thru 0x03ffff (0xbe5 bytes) 000249 1786828198.504916 MIRROR_RESYNC 3 mirrors 000250 1786828199.935684 trunc from 0x009041 to 0x011ae5 (0x8aa5 bytes) 000251 1786828200.029954 punch from 0x009e12 to 0x015e25 (0xc014 bytes) 000252 1786828200.037150 mapread 0x0329dd thru 0x036ec8 (0x44ec bytes) 000253 1786828200.068660 trunc from 0x007160 to 0x00d306 (0x61a7 bytes) 000255 1786828200.084340 punch from 0x016233 to 0x017efa (0x1cc8 bytes) 000256 1786828200.090842 write 0x01d1c0 thru 0x021741 (0x4582 bytes) 000257 1786828200.117451 trunc from 0x040000 to 0x02975f (0x168a0 bytes) 000258 1786828200.222482 trunc from 0x020a7c to 0x022bd0 (0x2155 bytes) 000259 1786828200.269607 read 0x017fd0 thru 0x01e3e7 (0x6418 bytes) 000260 1786828200.291227 mapwrite 0x03b784 thru 0x03ffff (0x487c bytes) 000261 1786828200.363364 punch from 0x00d013 to 0x01c938 (0xf926 bytes) 000262 1786828200.370981 trunc from 0x040000 to 0x00a05c (0x35fa3 bytes) 000263 1786828200.395231 trunc from 0x0039af to 0x00a05c (0x66ae bytes) 000264 1786828200.401565 mapwrite 0x02cd44 thru 0x032760 (0x5a1d bytes) 000265 1786828200.475119 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x32761): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32761): Mirror 2: 0x8c28b470 (stale) Mirror 3: 0x8c28b470 (stale) CRC-32 checksum value for chunk [0, 0x32761): Mirror 1: 0xd3f72b27 Mirror 4: 0xd3f72b27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32761: skipped 000267 1786828201.770544 mapread 0x002e83 thru 0x005e44 (0x2fc2 bytes) 000269 1786828201.818830 read 0x007a3d thru 0x00e9cd (0x6f91 bytes) 000270 1786828201.827613 trunc from 0x00916d to 0x00bd4f (0x2be3 bytes) 000271 1786828201.870715 trunc from 0x024e21 to 0x032760 (0xd940 bytes) 000272 1786828201.878876 write 0x03fc81 thru 0x03ffff (0x37f bytes) 000273 1786828201.886145 punch from 0x02fb2c to 0x03d24a (0xd71f bytes) 000274 1786828201.897394 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc41716ea (stale) Mirror 3: 0xc41716ea (stale) Mirror 4: 0xfd1a5da2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4c310fd8 Mirror 5: 0x4c310fd8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000275 1786828203.415278 punch from 0x016693 to 0x01bfbb (0x5929 bytes) 000276 1786828203.509005 punch from 0x022c5e to 0x0266d9 (0x3a7c bytes) 000277 1786828203.523369 trunc from 0x040000 to 0x0104c3 (0x2fb3c bytes) 000278 1786828203.568886 trunc from 0x0104c4 to 0x00ef15 (0x15ae bytes) 000279 1786828203.605395 trunc from 0x00ef16 to 0x01d2ee (0xe3d9 bytes) 000280 1786828203.649519 write 0x023f2d thru 0x030496 (0xc56a bytes) 000281 1786828203.657321 trunc from 0x02331e to 0x024c87 (0x196a bytes) 000282 1786828203.678467 trunc from 0x030497 to 0x01d863 (0x12c33 bytes) 000283 1786828203.695987 trunc from 0x006d6d to 0x00c332 (0x55c6 bytes) 000284 1786828203.701549 trunc from 0x01d864 to 0x03e37c (0x20b19 bytes) 000285 1786828203.721309 MIRROR_RESYNC 5 mirrors 000286 1786828205.073076 trunc from 0x03e37d to 0x01c7b1 (0x21bcb bytes) 000287 1786828205.147382 mapwrite 0x0019be thru 0x0082ae (0x68f1 bytes) 000288 1786828205.239417 trunc from 0x01c7b2 to 0x010988 (0xbe29 bytes) 000289 1786828205.302697 mapread 0x00f381 thru 0x0103ab (0x102b bytes) 000290 1786828205.305156 punch from 0x0099ed to 0x0107fd (0x6e11 bytes) 000291 1786828205.324732 punch from 0x007248 to 0x010988 (0x9741 bytes) 000294 1786828205.340836 write 0x00075b thru 0x00e58a (0xde30 bytes) 000295 1786828205.359873 read 0x00f6db thru 0x010988 (0x12ae bytes) 000297 1786828205.367055 mapread 0x006637 thru 0x009651 (0x301b bytes) 000298 1786828205.368599 mapwrite 0x0191cf thru 0x021e1b (0x8c4d bytes) 000299 1786828205.435218 mapwrite 0x037537 thru 0x03ab9c (0x3666 bytes) 000300 1786828205.477495 mapwrite 0x01087f thru 0x01f5f5 (0xed77 bytes) 000301 1786828205.511806 write 0x009409 thru 0x00e466 (0x505e bytes) 000302 1786828205.514526 punch from 0x006b02 to 0x00f3d2 (0x88d1 bytes) 000303 1786828205.529341 trunc from 0x002ba0 to 0x0128e2 (0xfd43 bytes) 000304 1786828205.536133 read 0x010d81 thru 0x01bbf3 (0xae73 bytes) 000305 1786828205.547256 fallocate 0x01a6b5 thru 0x02a086 (0xf9d2 bytes) 000306 1786828205.552724 trunc from 0x03ab9d to 0x003b78 (0x37024 bytes) 000307 1786828205.577564 read 0x001f1a thru 0x003b78 (0x1c5f bytes) 000308 1786828205.579799 read 0x001615 thru 0x002cba (0x16a6 bytes) 000309 1786828205.581054 punch from 0x00161d to 0x002230 (0xc14 bytes) 000310 1786828205.587401 mapwrite 0x00ee1b thru 0x01472e (0x5914 bytes) 000311 1786828205.634824 read 0x011fc0 thru 0x01472e (0x276f bytes) 000312 1786828205.635514 fallocate 0x01f539 thru 0x025d8e (0x6856 bytes) 000313 1786828205.642172 read 0x01a770 thru 0x01dda1 (0x3632 bytes) 000314 1786828205.652359 fallocate 0x02ea1a thru 0x035c2c (0x7213 bytes) 000315 1786828205.658299 punch from 0x032ccc to 0x035c2c (0x2f61 bytes) 000316 1786828205.667832 MIRROR_RESYNC 5 mirrors 000317 1786828207.063402 punch from 0x0020cb to 0x00d2db (0xb211 bytes) 000318 1786828207.108513 write 0x03b43b thru 0x03ffff (0x4bc5 bytes) 000319 1786828207.110672 write 0x03aad6 thru 0x03ffff (0x552a bytes) 000320 1786828207.117171 mapwrite 0x02e314 thru 0x031c2a (0x3917 bytes) 000321 1786828207.155108 mapwrite 0x01e4a8 thru 0x024713 (0x626c bytes) 000322 1786828207.193532 trunc from 0x040000 to 0x003a9d (0x3c562 bytes) 000323 1786828207.228611 punch from 0x00284f to 0x003a9d (0x124f bytes) 000324 1786828207.245938 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a9e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a9e): Mirror 2: 0xdfc01c42 (stale) Mirror 3: 0xdfc01c42 (stale) Mirror 4: 0xdfc01c42 (stale) Mirror 5: 0xdfc01c42 (stale) CRC-32 checksum value for chunk [0, 0x3a9e): Mirror 1: 0x8f827ee3 Mirror 6: 0x8f827ee3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a9e: skipped 000325 1786828208.338069 mapread 0x003684 thru 0x003a9d (0x41a bytes) 000326 1786828208.365824 write 0x033051 thru 0x03ffff (0xcfaf bytes) 000327 1786828208.417643 mapwrite 0x0004f9 thru 0x001d1b (0x1823 bytes) 000328 1786828208.452018 mapread 0x02ab10 thru 0x02fb0f (0x5000 bytes) 000329 1786828208.468001 punch from 0x00d957 to 0x016dcc (0x9476 bytes) 000330 1786828208.477962 write 0x00c783 thru 0x019f19 (0xd797 bytes) 000331 1786828208.484720 read 0x03a845 thru 0x03e6d9 (0x3e95 bytes) 000332 1786828208.485518 write 0x025d6b thru 0x02b4b6 (0x574c bytes) 000333 1786828208.488419 write 0x0253b1 thru 0x03080c (0xb45c bytes) 000334 1786828208.490097 mapread 0x03a8d8 thru 0x03ffff (0x5728 bytes) 000335 1786828208.491294 trunc from 0x014dde to 0x016e2a (0x204d bytes) 000336 1786828208.506456 write 0x022a73 thru 0x02f276 (0xc804 bytes) 000337 1786828208.513493 trunc from 0x040000 to 0x01aaac (0x25553 bytes) 000338 1786828208.535985 trunc from 0x01aaad to 0x02b121 (0x10675 bytes) 000339 1786828208.553602 trunc from 0x02b122 to 0x028582 (0x2b9f bytes) 000340 1786828208.571490 trunc from 0x028583 to 0x030c59 (0x86d7 bytes) 000341 1786828208.592749 punch from 0x003e16 to 0x01218d (0xe378 bytes) 000343 1786828208.598203 trunc from 0x006895 to 0x00afc8 (0x4734 bytes) 000344 1786828208.606131 trunc from 0x01244f to 0x01f274 (0xce26 bytes) 000346 1786828208.613496 punch from 0x00236d to 0x010b2c (0xe7c0 bytes) 000347 1786828208.620140 mapread 0x01a044 thru 0x02082c (0x67e9 bytes) 000348 1786828208.632460 mapread 0x00b9ef thru 0x014d48 (0x935a bytes) 000349 1786828208.643409 fallocate 0x008023 thru 0x00ccec (0x4cca bytes) 000351 1786828208.651853 fallocate 0x031c68 thru 0x0352bb (0x3654 bytes) 000352 1786828208.657201 mapwrite 0x02205f thru 0x0311da (0xf17c bytes) 000353 1786828208.710140 mapread 0x017301 thru 0x01b57e (0x427e bytes) 000355 1786828208.712988 punch from 0x0078fe to 0x00962b (0x1d2e bytes) 000356 1786828208.718374 fallocate 0x0090ae thru 0x0166bb (0xd60e bytes) 000357 1786828208.727220 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x311db): Mirror 3: 0x782a51e5 (stale) Mirror 4: 0x782a51e5 (stale) Mirror 5: 0x782a51e5 (stale) Mirror 6: 0x964460cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x311db): Mirror 3: 0x782a51e5 (stale) Mirror 4: 0x782a51e5 (stale) Mirror 5: 0x782a51e5 (stale) Mirror 6: 0x964460cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x311db: skipped 000358 1786828209.052887 mapwrite 0x01f0d8 thru 0x025938 (0x6861 bytes) 000359 1786828209.111117 trunc from 0x02c9a4 to 0x0311da (0x4837 bytes) 000360 1786828209.119002 mapread 0x028d8a thru 0x0304bc (0x7733 bytes) 000361 1786828209.133379 write 0x024f1c thru 0x029f1e (0x5003 bytes) 000362 1786828209.146244 MIRROR_RESYNC 5 mirrors 000363 1786828210.858327 write 0x008b9b thru 0x0112d3 (0x8739 bytes) 000364 1786828210.961551 mapwrite 0x0333ca thru 0x03dae4 (0xa71b bytes) 000365 1786828211.076554 mapread 0x00452e thru 0x00c8d2 (0x83a5 bytes) 000366 1786828211.093931 mapwrite 0x0028d2 thru 0x00b18f (0x88be bytes) 000367 1786828211.128365 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dae5): Mirror 3: 0xaf2c7a72 (stale) Mirror 5: 0xaf2c7a72 (stale) Mirror 6: 0xaf2c7a72 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dae5): Mirror 3: 0xaf2c7a72 (stale) Mirror 5: 0xaf2c7a72 (stale) Mirror 6: 0xaf2c7a72 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dae5: skipped 000368 1786828211.534563 read 0x02e997 thru 0x03255e (0x3bc8 bytes) 000369 1786828211.544172 trunc from 0x03b18b to 0x03dae4 (0x295a bytes) 000370 1786828211.550020 punch from 0x03c0fb to 0x03dae4 (0x19ea bytes) 000371 1786828211.560596 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3dae5): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dae5): Mirror 3: 0xaf2c7a72 (stale) Mirror 5: 0xaf2c7a72 (stale) Mirror 6: 0xaf2c7a72 (stale) CRC-32 checksum value for chunk [0, 0x3dae5): Mirror 1: 0x44ab1531 Mirror 7: 0x44ab1531 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dae5: skipped 000372 1786828212.910240 mapwrite 0x0188f6 thru 0x01d677 (0x4d82 bytes) 000373 1786828213.042353 write 0x002746 thru 0x007ec9 (0x5784 bytes) 000374 1786828213.058623 trunc from 0x03dae5 to 0x015b6e (0x27f76 bytes) 000375 1786828213.081878 fallocate 0x02c041 thru 0x030c05 (0x4bc5 bytes) 000376 1786828213.088037 read 0x00ff31 thru 0x0152bd (0x538d bytes) 000377 1786828213.096970 read 0x01294f thru 0x022775 (0xfe27 bytes) 000378 1786828213.110308 write 0x03e37d thru 0x03ffff (0x1c83 bytes) 000379 1786828213.112070 fallocate 0x00e052 thru 0x0149fc (0x69ab bytes) 000380 1786828213.118636 trunc from 0x023404 to 0x0247c7 (0x13c4 bytes) 000381 1786828213.129476 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfdaaff2c (stale) Mirror 5: 0xfdaaff2c (stale) Mirror 6: 0xfdaaff2c (stale) Mirror 7: 0xeec7246f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9e00a93f Mirror 8: 0x9e00a93f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000382 1786828214.083297 MIRROR_RESYNC 6 mirrors 000384 1786828215.130682 trunc from 0x02cfd3 to 0x039491 (0xc4bf bytes) 000385 1786828215.240820 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9e00a93f (stale) Mirror 5: 0x9e00a93f (stale) Mirror 6: 0x9e00a93f (stale) Mirror 7: 0x9e00a93f (stale) Mirror 8: 0x9e00a93f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9e00a93f Mirror 9: 0x9e00a93f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000386 1786828216.819252 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9e00a93f (stale) Mirror 5: 0x9e00a93f (stale) Mirror 7: 0x9e00a93f (stale) Mirror 8: 0x9e00a93f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9e00a93f (stale) Mirror 5: 0x9e00a93f (stale) Mirror 7: 0x9e00a93f (stale) Mirror 8: 0x9e00a93f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9e00a93f Mirror 9: 0x9e00a93f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000387 1786828217.933196 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9e00a93f (stale) Mirror 5: 0x9e00a93f (stale) Mirror 7: 0x9e00a93f (stale) Mirror 8: 0x9e00a93f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9e00a93f Mirror 9: 0x9e00a93f Mirror 10: 0x9e00a93f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000388 1786828219.023719 trunc from 0x040000 to 0x020ea5 (0x1f15a bytes) 000389 1786828219.093480 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x20ea6): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20ea6): Mirror 3: 0xcee8bcf3 (stale) Mirror 5: 0xcee8bcf3 (stale) Mirror 7: 0xcee8bcf3 (stale) Mirror 8: 0xcee8bcf3 (stale) Mirror 9: 0xcee8bcf3 (stale) Mirror 10: 0xcee8bcf3 (stale) CRC-32 checksum value for chunk [0, 0x20ea6): Mirror 1: 0xcee8bcf3 Mirror 11: 0xcee8bcf3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20ea6: skipped 000391 1786828220.206450 trunc from 0x01d695 to 0x020ea5 (0x3811 bytes) 000392 1786828220.260661 mapwrite 0x00b208 thru 0x00b962 (0x75b bytes) 000393 1786828220.282188 mapread 0x00dc63 thru 0x01ad43 (0xd0e1 bytes) 000394 1786828220.319714 fallocate 0x0145a7 thru 0x022fbb (0xea15 bytes) 000396 1786828220.329333 read 0x01722c thru 0x01cf55 (0x5d2a bytes) 000397 1786828220.330716 read 0x0015bd thru 0x0114c1 (0xff05 bytes) 000398 1786828220.342495 trunc from 0x00fc4a to 0x012255 (0x260c bytes) 000399 1786828220.350685 trunc from 0x020ea6 to 0x01f9a2 (0x1503 bytes) 000400 1786828220.389466 mapread 0x016e49 thru 0x01cb59 (0x5d11 bytes) 000402 1786828220.395572 trunc from 0x01f9a3 to 0x00ee8b (0x10b17 bytes) 000403 1786828220.443355 fallocate 0x02dad9 thru 0x03ab49 (0xd071 bytes) 000404 1786828220.454705 write 0x03e52a thru 0x03ffff (0x1ad6 bytes) 000405 1786828220.459895 trunc from 0x040000 to 0x010853 (0x2f7ac bytes) 000407 1786828220.487970 punch from 0x00bac7 to 0x010853 (0x4d8d bytes) 000408 1786828220.497515 fallocate 0x022a09 thru 0x02e948 (0xbf40 bytes) 000410 1786828220.505531 mapread 0x004d4e thru 0x00dff4 (0x92a7 bytes) 000411 1786828220.541466 mapwrite 0x00d1ed thru 0x0164e1 (0x92f5 bytes) 000412 1786828220.608040 write 0x00a324 thru 0x011118 (0x6df5 bytes) 000414 1786828220.611257 trunc from 0x0029c5 to 0x010178 (0xd7b4 bytes) 000415 1786828220.629160 fallocate 0x01e6ba thru 0x026e00 (0x8747 bytes) 000416 1786828220.637679 write 0x032dc0 thru 0x03ffff (0xd240 bytes) 000417 1786828220.646589 punch from 0x012db1 to 0x01ccc3 (0x9f13 bytes) 000418 1786828220.655207 fallocate 0x002f1a thru 0x012d31 (0xfe18 bytes) 000419 1786828220.662652 write 0x02a761 thru 0x039fa9 (0xf849 bytes) 000420 1786828220.672811 MIRROR_RESYNC 8 mirrors 000421 1786828222.833734 mapread 0x00fa34 thru 0x0107f1 (0xdbe bytes) 000422 1786828222.844485 write 0x01246e thru 0x015f45 (0x3ad8 bytes) 000423 1786828222.926824 mapwrite 0x0058a3 thru 0x0126fb (0xce59 bytes) 000424 1786828223.023033 mapread 0x01f664 thru 0x020bb4 (0x1551 bytes) 000425 1786828223.058389 MIRROR_RESYNC 8 mirrors 000426 1786828224.972738 trunc from 0x040000 to 0x002acd (0x3d532 bytes) 000427 1786828225.016827 fallocate 0x018db5 thru 0x02042e (0x767a bytes) 000430 1786828225.024424 mapwrite 0x01e102 thru 0x024f69 (0x6e68 bytes) 000431 1786828225.084877 write 0x02a47b thru 0x02bbe1 (0x1767 bytes) 000432 1786828225.088329 read 0x018f0f thru 0x01f156 (0x6248 bytes) 000433 1786828225.104383 write 0x025f08 thru 0x02bf73 (0x606c bytes) 000434 1786828225.115099 punch from 0x029176 to 0x02bf73 (0x2dfe bytes) 000435 1786828225.146837 fallocate 0x01585f thru 0x017e59 (0x25fb bytes) 000436 1786828225.157058 mapread 0x01c089 thru 0x022f64 (0x6edc bytes) 000437 1786828225.160943 mapread 0x009186 thru 0x018c96 (0xfb11 bytes) 000438 1786828225.219134 MIRROR_RESYNC 8 mirrors 000439 1786828227.185932 punch from 0x02be43 to 0x02bf73 (0x131 bytes) 000440 1786828227.231615 mapwrite 0x036ddd thru 0x03ffff (0x9223 bytes) 000441 1786828227.337104 read 0x001c2b thru 0x00e3bb (0xc791 bytes) 000442 1786828227.351460 write 0x001360 thru 0x006953 (0x55f4 bytes) 000444 1786828227.356450 write 0x01de47 thru 0x021976 (0x3b30 bytes) 000445 1786828227.372368 trunc from 0x029059 to 0x035c64 (0xcc0c bytes) 000446 1786828227.382583 mapwrite 0x03a059 thru 0x03ffff (0x5fa7 bytes) 000447 1786828227.415698 read 0x02f9eb thru 0x038484 (0x8a9a bytes) 000448 1786828227.447573 fallocate 0x007099 thru 0x010e41 (0x9da9 bytes) 000449 1786828227.456842 mapwrite 0x024d8f thru 0x0255ef (0x861 bytes) 000451 1786828227.477597 mapread 0x01c3ce thru 0x029890 (0xd4c3 bytes) 000452 1786828227.479045 write 0x0283ea thru 0x02d0b4 (0x4ccb bytes) 000453 1786828227.483662 read 0x03e9bf thru 0x03ffff (0x1641 bytes) 000454 1786828227.485569 trunc from 0x00b1e5 to 0x017490 (0xc2ac bytes) 000455 1786828227.493090 fallocate 0x016645 thru 0x022d5a (0xc716 bytes) 000458 1786828227.506709 fallocate 0x01ec7e thru 0x029a1d (0xada0 bytes) 000459 1786828227.514056 trunc from 0x037a0a to 0x03ffff (0x85f6 bytes) 000460 1786828227.521675 write 0x0240d7 thru 0x026efb (0x2e25 bytes) 000461 1786828227.524137 punch from 0x0334b8 to 0x0381f0 (0x4d39 bytes) 000462 1786828227.535611 mapread 0x00151a thru 0x00dcd0 (0xc7b7 bytes) 000464 1786828227.559650 write 0x036174 thru 0x03d257 (0x70e4 bytes) 000465 1786828227.563086 fallocate 0x01779c thru 0x0249d8 (0xd23d bytes) 000466 1786828227.570872 trunc from 0x040000 to 0x02b8bd (0x14742 bytes) 000467 1786828227.602834 trunc from 0x02458a to 0x02b8bd (0x7334 bytes) 000468 1786828227.622720 trunc from 0x02b8be to 0x033cc4 (0x8407 bytes) 000469 1786828227.649419 read 0x008dfe thru 0x016b7e (0xdd81 bytes) 000470 1786828227.657905 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33cc5): Mirror 3: 0x65ac1a84 (stale) Mirror 5: 0x65ac1a84 (stale) Mirror 8: 0x65ac1a84 (stale) Mirror 9: 0x65ac1a84 (stale) Mirror 10: 0x65ac1a84 (stale) Mirror 11: 0x65ac1a84 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33cc5): Mirror 3: 0x65ac1a84 (stale) Mirror 5: 0x65ac1a84 (stale) Mirror 8: 0x65ac1a84 (stale) Mirror 9: 0x65ac1a84 (stale) Mirror 10: 0x65ac1a84 (stale) Mirror 11: 0x65ac1a84 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33cc5: skipped 000471 1786828228.330658 punch from 0x007a26 to 0x00fa11 (0x7fec bytes) 000472 1786828228.350302 write 0x0066cc thru 0x012044 (0xb979 bytes) 000473 1786828228.391234 mapread 0x023329 thru 0x03230b (0xefe3 bytes) 000474 1786828228.489554 write 0x0334bc thru 0x035e30 (0x2975 bytes) 000475 1786828228.490743 fallocate 0x024d7f thru 0x0287a2 (0x3a24 bytes) 000476 1786828228.503350 mapread 0x00f565 thru 0x01c68c (0xd128 bytes) 000477 1786828228.539914 read 0x02422d thru 0x028b1c (0x48f0 bytes) 000478 1786828228.540598 punch from 0x00c9cb to 0x01836e (0xb9a4 bytes) 000479 1786828228.556791 punch from 0x007a49 to 0x00f933 (0x7eeb bytes) 000480 1786828228.563625 mapwrite 0x01767a thru 0x01ab66 (0x34ed bytes) 000481 1786828228.594510 MIRROR_RESYNC 7 mirrors 000482 1786828231.185873 punch from 0x01ff35 to 0x02fc03 (0xfccf bytes) 000483 1786828231.262171 read 0x012ef7 thru 0x01fc5b (0xcd65 bytes) 000484 1786828231.300170 mapwrite 0x022011 thru 0x02962d (0x761d bytes) 000485 1786828231.367370 mapwrite 0x03565e thru 0x03ffff (0xa9a2 bytes) 000486 1786828231.465589 punch from 0x012438 to 0x0218cb (0xf494 bytes) 000487 1786828231.471052 trunc from 0x040000 to 0x01f9b7 (0x20648 bytes) 000488 1786828231.495964 fallocate 0x012d0b thru 0x0186f4 (0x59ea bytes) 000489 1786828231.503552 mapread 0x0129a7 thru 0x015ece (0x3528 bytes) 000490 1786828231.531603 write 0x01b08e thru 0x029bce (0xeb41 bytes) 000491 1786828231.548103 write 0x03a97b thru 0x03ffff (0x5685 bytes) 000492 1786828231.556019 trunc from 0x02363e to 0x02db87 (0xa54a bytes) 000493 1786828231.595308 fallocate 0x01010c thru 0x01d23c (0xd131 bytes) 000494 1786828231.612174 mapread 0x03fce1 thru 0x03ffff (0x31f bytes) 000495 1786828231.620617 mapread 0x00ba43 thru 0x01a001 (0xe5bf bytes) 000496 1786828231.626552 read 0x022280 thru 0x02da7f (0xb800 bytes) 000499 1786828231.668008 read 0x02fdb4 thru 0x02fec1 (0x10e bytes) 000500 1786828231.672840 punch from 0x03d591 to 0x03ffff (0x2a6f bytes) 000501 1786828231.687064 mapread 0x0395d7 thru 0x039a9d (0x4c7 bytes) 000502 1786828231.694116 punch from 0x032721 to 0x036e5c (0x473c bytes) 000503 1786828231.711929 mapread 0x0026b8 thru 0x010d75 (0xe6be bytes) 000504 1786828231.721399 mapwrite 0x0209d3 thru 0x02e657 (0xdc85 bytes) 000505 1786828231.770400 punch from 0x01a962 to 0x025215 (0xa8b4 bytes) 000506 1786828231.782953 write 0x0310f5 thru 0x03e2a0 (0xd1ac bytes) 000507 1786828231.808965 read 0x033fdb thru 0x03ffff (0xc025 bytes) 000508 1786828231.820700 read 0x039d1b thru 0x03ffff (0x62e5 bytes) 000509 1786828231.821527 fallocate 0x00a9bc thru 0x018bcd (0xe212 bytes) 000510 1786828231.834521 mapread 0x00fc32 thru 0x0123c6 (0x2795 bytes) 000511 1786828231.837354 trunc from 0x01f89e to 0x02c892 (0xcff5 bytes) 000512 1786828231.858361 fallocate 0x028a20 thru 0x03026b (0x784c bytes) 000513 1786828231.875634 punch from 0x01d2f1 to 0x0219c1 (0x46d1 bytes) 000515 1786828231.888976 read 0x0040e7 thru 0x013621 (0xf53b bytes) 000516 1786828231.889794 trunc from 0x01c527 to 0x0259a3 (0x947d bytes) 000518 1786828231.925208 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x28e70b00 (stale) Mirror 5: 0x28e70b00 (stale) Mirror 8: 0x28e70b00 (stale) Mirror 9: 0x28e70b00 (stale) Mirror 10: 0x28e70b00 (stale) Mirror 11: 0x28e70b00 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x32c061a Mirror 12: 0x32c061a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000519 1786828233.563649 fallocate 0x038292 thru 0x03d205 (0x4f74 bytes) 000520 1786828233.708664 trunc from 0x03e778 to 0x03ffff (0x1888 bytes) 000521 1786828233.730191 trunc from 0x040000 to 0x0193e5 (0x26c1a bytes) 000522 1786828233.822877 read 0x0191b3 thru 0x0193e5 (0x233 bytes) 000523 1786828233.845206 punch from 0x00525c to 0x00a4dd (0x5282 bytes) 000524 1786828233.860727 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x193e6): Mirror 5: 0x64d1f272 (stale) Mirror 8: 0x64d1f272 (stale) Mirror 9: 0x64d1f272 (stale) Mirror 10: 0x64d1f272 (stale) Mirror 11: 0x64d1f272 (stale) Mirror 12: 0x4e5151f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x193e6): Mirror 5: 0x64d1f272 (stale) Mirror 8: 0x64d1f272 (stale) Mirror 9: 0x64d1f272 (stale) Mirror 10: 0x64d1f272 (stale) Mirror 11: 0x64d1f272 (stale) Mirror 12: 0x4e5151f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x193e6: skipped 000526 1786828234.416544 mapwrite 0x008f15 thru 0x016ae5 (0xdbd1 bytes) 000527 1786828234.473184 mapread 0x012d39 thru 0x0146e5 (0x19ad bytes) 000529 1786828234.474923 mapwrite 0x0330f0 thru 0x033ebc (0xdcd bytes) 000530 1786828234.509010 punch from 0x02df0e to 0x033ebc (0x5faf bytes) 000532 1786828234.515648 write 0x02a787 thru 0x02f390 (0x4c0a bytes) 000534 1786828234.528058 trunc from 0x033ebd to 0x0045e0 (0x2f8dc bytes) 000535 1786828234.552088 mapwrite 0x014cf5 thru 0x020fc7 (0xc2d3 bytes) 000536 1786828234.623098 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20fc8): Mirror 5: 0x5abf4a20 (stale) Mirror 8: 0x5abf4a20 (stale) Mirror 10: 0x5abf4a20 (stale) Mirror 11: 0x5abf4a20 (stale) Mirror 12: 0xc2618b44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20fc8): Mirror 5: 0x5abf4a20 (stale) Mirror 8: 0x5abf4a20 (stale) Mirror 10: 0x5abf4a20 (stale) Mirror 11: 0x5abf4a20 (stale) Mirror 12: 0xc2618b44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20fc8: skipped 000537 1786828235.261911 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x20fc8): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20fc8): Mirror 5: 0x5abf4a20 (stale) Mirror 8: 0x5abf4a20 (stale) Mirror 10: 0x5abf4a20 (stale) Mirror 11: 0x5abf4a20 (stale) Mirror 12: 0xc2618b44 (stale) CRC-32 checksum value for chunk [0, 0x20fc8): Mirror 1: 0x99f7f9f7 Mirror 13: 0x99f7f9f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20fc8: skipped 000538 1786828236.857507 fallocate 0x011e9a thru 0x01bd5b (0x9ec2 bytes) 000539 1786828236.917293 read 0x00e4a4 thru 0x0158a2 (0x73ff bytes) 000540 1786828236.929378 trunc from 0x020fc8 to 0x029f6b (0x8fa4 bytes) 000541 1786828236.955690 read 0x01db45 thru 0x01df42 (0x3fe bytes) 000542 1786828236.979942 read 0x004422 thru 0x011cea (0xd8c9 bytes) 000543 1786828236.981438 read 0x029620 thru 0x029f6b (0x94c bytes) 000544 1786828236.982539 mapread 0x005da5 thru 0x0060f7 (0x353 bytes) 000545 1786828236.984264 trunc from 0x029f6c to 0x007401 (0x22b6a bytes) 000546 1786828237.001735 trunc from 0x004fdc to 0x007401 (0x2426 bytes) 000547 1786828237.008376 write 0x0111ed thru 0x020a4e (0xf862 bytes) 000548 1786828237.011704 mapread 0x001bf5 thru 0x005e11 (0x421d bytes) 000549 1786828237.020951 read 0x01cffd thru 0x01d8dd (0x8e1 bytes) 000550 1786828237.021751 write 0x025100 thru 0x02d58f (0x8490 bytes) 000551 1786828237.024149 write 0x012b2e thru 0x01b6a5 (0x8b78 bytes) 000552 1786828237.025903 fallocate 0x03cf92 thru 0x03ffff (0x306e bytes) 000553 1786828237.032507 write 0x038899 thru 0x03ffff (0x7767 bytes) 000554 1786828237.043288 punch from 0x037daf to 0x03ffff (0x8251 bytes) 000555 1786828237.061252 fallocate 0x03ea83 thru 0x03ffff (0x157d bytes) 000556 1786828237.066864 read 0x020a09 thru 0x02dcd8 (0xd2d0 bytes) 000557 1786828237.068592 mapread 0x01e8b2 thru 0x027cdf (0x942e bytes) 000558 1786828237.070789 trunc from 0x001b27 to 0x00226c (0x746 bytes) 000559 1786828237.075933 write 0x03ee39 thru 0x03f3fa (0x5c2 bytes) 000560 1786828237.085840 mapread 0x01d650 thru 0x02c318 (0xecc9 bytes) 000561 1786828237.088937 mapwrite 0x017ddf thru 0x0196d8 (0x18fa bytes) 000562 1786828237.106568 read 0x0367f0 thru 0x037380 (0xb91 bytes) 000563 1786828237.107833 mapread 0x0037d4 thru 0x00b1bb (0x79e8 bytes) 000564 1786828237.109619 trunc from 0x040000 to 0x033dba (0xc245 bytes) 000565 1786828237.151629 MIRROR_RESYNC 7 mirrors 000566 1786828239.964134 mapread 0x01639b thru 0x025cea (0xf950 bytes) 000567 1786828240.047059 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000568 1786828240.684842 mapwrite 0x002e7f thru 0x012c88 (0xfe0a bytes) 000569 1786828240.922564 trunc from 0x015b42 to 0x01ebd9 (0x9098 bytes) 000570 1786828240.930815 mapwrite 0x015fb4 thru 0x023c59 (0xdca6 bytes) 000571 1786828240.979423 punch from 0x00b458 to 0x00e78a (0x3333 bytes) 000573 1786828240.994042 fallocate 0x0296f4 thru 0x03077f (0x708c bytes) 000574 1786828241.007598 punch from 0x013de6 to 0x0210b3 (0xd2ce bytes) 000575 1786828241.020939 trunc from 0x033dbb to 0x02cc9a (0x7120 bytes) 000576 1786828241.087587 fallocate 0x020765 thru 0x02ba63 (0xb2ff bytes) 000578 1786828241.134470 MIRROR_RESYNC 6 mirrors 000579 1786828243.800722 trunc from 0x02cc9b to 0x02e500 (0x1866 bytes) 000580 1786828243.871610 write 0x01a60e thru 0x027597 (0xcf8a bytes) 000581 1786828243.887564 mapread 0x025a7b thru 0x02ab92 (0x5118 bytes) 000582 1786828243.908260 mapread 0x00eb38 thru 0x01d6f9 (0xebc2 bytes) 000583 1786828243.922397 punch from 0x01d992 to 0x0210b2 (0x3721 bytes) 000584 1786828243.936709 punch from 0x011e80 to 0x01d323 (0xb4a4 bytes) 000585 1786828243.941743 read 0x01d664 thru 0x01ede4 (0x1781 bytes) 000586 1786828243.947496 trunc from 0x02e501 to 0x026464 (0x809c bytes) 000587 1786828243.967344 write 0x01b30d thru 0x025ac0 (0xa7b4 bytes) 000588 1786828243.975157 mapwrite 0x008f3a thru 0x00d01f (0x40e6 bytes) 000589 1786828243.992754 trunc from 0x00e0bb to 0x01d29d (0xf1e3 bytes) 000590 1786828243.997747 trunc from 0x026465 to 0x00b76a (0x1acfa bytes) 000591 1786828244.013115 fallocate 0x000c9a thru 0x005a25 (0x4d8c bytes) 000592 1786828244.018746 trunc from 0x006e82 to 0x00b76a (0x48e9 bytes) 000593 1786828244.024229 mapwrite 0x013f33 thru 0x023308 (0xf3d6 bytes) 000594 1786828244.103714 read 0x01c921 thru 0x023308 (0x69e8 bytes) 000595 1786828244.106196 mapwrite 0x027adf thru 0x02c82c (0x4d4e bytes) 000596 1786828244.176788 trunc from 0x02ba27 to 0x02c82c (0xe06 bytes) 000597 1786828244.188716 trunc from 0x02c82d to 0x031271 (0x4a45 bytes) 000598 1786828244.219662 fallocate 0x036e60 thru 0x03b98c (0x4b2d bytes) 000600 1786828244.238283 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 8: 0x49179f7e (stale) Mirror 10: 0x49179f7e (stale) Mirror 11: 0x49179f7e (stale) Mirror 12: 0x49179f7e (stale) Mirror 13: 0x49179f7e (stale) CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 1: 0x20c32996 Mirror 14: 0x20c32996 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b98d: skipped 000601 1786828246.150650 trunc from 0x021b76 to 0x02b2d2 (0x975d bytes) 000603 1786828246.247756 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 8: 0x49179f7e (stale) Mirror 10: 0x49179f7e (stale) Mirror 11: 0x49179f7e (stale) Mirror 12: 0x49179f7e (stale) Mirror 13: 0x49179f7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 8: 0x49179f7e (stale) Mirror 10: 0x49179f7e (stale) Mirror 11: 0x49179f7e (stale) Mirror 12: 0x49179f7e (stale) Mirror 13: 0x49179f7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b98d: skipped 000604 1786828247.024667 fallocate 0x016e88 thru 0x0240fa (0xd273 bytes) 000605 1786828247.038648 trunc from 0x029a41 to 0x02b547 (0x1b07 bytes) 000606 1786828247.045398 trunc from 0x001903 to 0x00a078 (0x8776 bytes) 000607 1786828247.062858 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 8: 0x49179f7e (stale) Mirror 10: 0x49179f7e (stale) Mirror 11: 0x49179f7e (stale) Mirror 13: 0x49179f7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 8: 0x49179f7e (stale) Mirror 10: 0x49179f7e (stale) Mirror 11: 0x49179f7e (stale) Mirror 13: 0x49179f7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b98d: skipped 000608 1786828247.547376 read 0x00d92b thru 0x00eb6d (0x1243 bytes) 000609 1786828247.554740 trunc from 0x00e26f to 0x0176c4 (0x9456 bytes) 000610 1786828247.560675 trunc from 0x03b98d to 0x02c529 (0xf463 bytes) 000611 1786828247.601703 mapwrite 0x024cad thru 0x02fb7c (0xaed0 bytes) 000612 1786828247.681531 mapread 0x007209 thru 0x0113f1 (0xa1e9 bytes) 000613 1786828247.705088 punch from 0x015c92 to 0x01daff (0x7e6e bytes) 000614 1786828247.710521 write 0x0332ce thru 0x034688 (0x13bb bytes) 000615 1786828247.712447 mapread 0x0154f9 thru 0x01e164 (0x8c6c bytes) 000616 1786828247.726719 mapwrite 0x0194eb thru 0x02597b (0xc491 bytes) 000617 1786828247.753504 read 0x014c78 thru 0x021975 (0xccfe bytes) 000618 1786828247.754531 read 0x001cf8 thru 0x008bd0 (0x6ed9 bytes) 000619 1786828247.756336 write 0x025e70 thru 0x030d12 (0xaea3 bytes) 000620 1786828247.764597 fallocate 0x013081 thru 0x01b084 (0x8004 bytes) 000621 1786828247.771591 write 0x034b9a thru 0x03ffff (0xb466 bytes) 000622 1786828247.773675 write 0x037a90 thru 0x03ffff (0x8570 bytes) 000623 1786828247.776397 trunc from 0x025079 to 0x025e37 (0xdbf bytes) 000624 1786828247.790996 mapread 0x029f3a thru 0x03175e (0x7825 bytes) 000625 1786828247.798466 write 0x025578 thru 0x0295cc (0x4055 bytes) 000626 1786828247.805522 trunc from 0x040000 to 0x004b57 (0x3b4a8 bytes) 000627 1786828247.830077 mapread 0x0026cb thru 0x004b57 (0x248d bytes) 000628 1786828247.832575 mapwrite 0x02c210 thru 0x02d379 (0x116a bytes) 000629 1786828247.878933 read 0x007bf2 thru 0x011b76 (0x9f85 bytes) 000630 1786828247.898331 mapread 0x020449 thru 0x02715c (0x6d14 bytes) 000631 1786828247.904655 trunc from 0x0067b2 to 0x01542a (0xec79 bytes) 000632 1786828247.912934 trunc from 0x0248a5 to 0x028d7e (0x44da bytes) 000633 1786828247.923970 read 0x022b66 thru 0x02ad35 (0x81d0 bytes) 000634 1786828247.944581 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d37a): Mirror 8: 0x734d03ca (stale) Mirror 10: 0x734d03ca (stale) Mirror 11: 0x734d03ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d37a): Mirror 8: 0x734d03ca (stale) Mirror 10: 0x734d03ca (stale) Mirror 11: 0x734d03ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d37a: skipped 000635 1786828248.523982 read 0x00fc99 thru 0x012c83 (0x2feb bytes) 000636 1786828248.547717 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d37a): Mirror 10: 0x734d03ca (stale) Mirror 11: 0x734d03ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d37a): Mirror 10: 0x734d03ca (stale) Mirror 11: 0x734d03ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d37a: skipped 000637 1786828249.175693 trunc from 0x02d37a to 0x005dc8 (0x275b1 bytes) 000638 1786828249.233089 mapread 0x0041cd thru 0x005dc8 (0x1bfc bytes) 000639 1786828249.259659 trunc from 0x002efb to 0x005dc8 (0x2ece bytes) 000641 1786828249.283103 trunc from 0x004b37 to 0x005dc8 (0x1292 bytes) 000642 1786828249.301710 MIRROR_RESYNC 3 mirrors 000643 1786828250.763981 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000644 1786828251.646663 mapread 0x003f76 thru 0x005dc8 (0x1e53 bytes) 000645 1786828251.686599 trunc from 0x005dc9 to 0x01e3c5 (0x185fd bytes) 000646 1786828251.808552 punch from 0x0093fb to 0x00d12a (0x3d30 bytes) 000647 1786828251.817006 punch from 0x01083c to 0x013729 (0x2eee bytes) 000648 1786828251.828298 fallocate 0x01a9e3 thru 0x026b86 (0xc1a4 bytes) 000649 1786828251.836537 trunc from 0x0008b8 to 0x00f1cb (0xe914 bytes) 000650 1786828251.844397 mapwrite 0x021095 thru 0x02e5f7 (0xd563 bytes) 000651 1786828251.928853 write 0x00b37a thru 0x00f581 (0x4208 bytes) 000652 1786828251.951309 fallocate 0x02d04b thru 0x0338b6 (0x686c bytes) 000653 1786828251.963929 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e5f8): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e5f8): Mirror 10: 0x1146b3fe (stale) CRC-32 checksum value for chunk [0, 0x2e5f8): Mirror 1: 0x4e2ec39c Mirror 11: 0x4e2ec39c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e5f8: skipped 000654 1786828252.852757 mapwrite 0x00f4c1 thru 0x01887c (0x93bc bytes) 000655 1786828252.943434 fallocate 0x011737 thru 0x019ef1 (0x87bb bytes) 000656 1786828252.949555 mapwrite 0x006f92 thru 0x009dd0 (0x2e3f bytes) 000658 1786828252.974306 trunc from 0x012355 to 0x01c68f (0xa33b bytes) 000661 1786828252.980841 write 0x010000 thru 0x012ade (0x2adf bytes) 000662 1786828252.997316 trunc from 0x02b893 to 0x02e5f7 (0x2d65 bytes) 000663 1786828253.007570 read 0x023dde thru 0x02de00 (0xa023 bytes) 000664 1786828253.019899 punch from 0x023d12 to 0x026239 (0x2528 bytes) 000665 1786828253.030756 punch from 0x02293f to 0x029b83 (0x7245 bytes) 000666 1786828253.039234 trunc from 0x02e5f8 to 0x02b5ea (0x300d bytes) 000668 1786828253.077351 trunc from 0x024226 to 0x02b5ea (0x73c5 bytes) 000669 1786828253.086984 fallocate 0x029b14 thru 0x038588 (0xea75 bytes) 000670 1786828253.094836 write 0x009e02 thru 0x00d4cb (0x36ca bytes) 000671 1786828253.096722 fallocate 0x0333d6 thru 0x034368 (0xf93 bytes) 000672 1786828253.101573 punch from 0x02a7e5 to 0x034368 (0x9b84 bytes) 000673 1786828253.115172 mapread 0x02e4e4 thru 0x0332e7 (0x4e04 bytes) 000675 1786828253.132437 write 0x010c3a thru 0x0111d9 (0x5a0 bytes) 000676 1786828253.136424 mapread 0x008276 thru 0x00ecbd (0x6a48 bytes) 000677 1786828253.139522 mapwrite 0x003683 thru 0x0121f0 (0xeb6e bytes) 000678 1786828253.202281 mapwrite 0x00fb1e thru 0x018618 (0x8afb bytes) 000679 1786828253.274197 write 0x02867e thru 0x03713f (0xeac2 bytes) 000680 1786828253.278597 punch from 0x01dedb to 0x02a990 (0xcab6 bytes) 000681 1786828253.303217 mapwrite 0x025a32 thru 0x030767 (0xad36 bytes) 000682 1786828253.341622 trunc from 0x026bca to 0x02902e (0x2465 bytes) 000683 1786828253.348053 punch from 0x02b4c8 to 0x03713f (0xbc78 bytes) 000684 1786828253.354474 trunc from 0x0049ed to 0x00c625 (0x7c39 bytes) 000685 1786828253.360001 trunc from 0x02a56e to 0x02ce16 (0x28a9 bytes) 000686 1786828253.370484 MIRROR_RESYNC 3 mirrors 000687 1786828254.785329 read 0x03088a thru 0x034caa (0x4421 bytes) 000688 1786828254.811326 write 0x01e22f thru 0x01fc5b (0x1a2d bytes) 000689 1786828254.886077 trunc from 0x02b3c0 to 0x02e117 (0x2d58 bytes) 000690 1786828254.892105 trunc from 0x037140 to 0x02449b (0x12ca4 bytes) 000691 1786828254.924419 mapread 0x01957c thru 0x01f5e0 (0x6065 bytes) 000692 1786828254.959568 read 0x01e762 thru 0x022473 (0x3d12 bytes) 000693 1786828254.961884 fallocate 0x039e72 thru 0x03ffff (0x618e bytes) 000694 1786828254.970411 read 0x0133bf thru 0x016c40 (0x3882 bytes) 000695 1786828255.002353 read 0x029788 thru 0x038d5c (0xf5d5 bytes) 000696 1786828255.003479 trunc from 0x03880f to 0x03ffff (0x77f1 bytes) 000697 1786828255.011130 read 0x015014 thru 0x021b82 (0xcb6f bytes) 000698 1786828255.013278 write 0x02285f thru 0x0248a9 (0x204b bytes) 000699 1786828255.015163 mapwrite 0x02a466 thru 0x034aee (0xa689 bytes) 000700 1786828255.053332 read 0x031c95 thru 0x03e3df (0xc74b bytes) 000701 1786828255.073579 mapwrite 0x0384a9 thru 0x03b23f (0x2d97 bytes) 000702 1786828255.096564 trunc from 0x03fb48 to 0x03ffff (0x4b8 bytes) 000703 1786828255.108375 read 0x019fcf thru 0x024e19 (0xae4b bytes) 000704 1786828255.111539 punch from 0x036878 to 0x0369c9 (0x152 bytes) 000705 1786828255.119493 mapwrite 0x00ce16 thru 0x011ffb (0x51e6 bytes) 000706 1786828255.136580 mapread 0x01165a thru 0x01d4e7 (0xbe8e bytes) 000707 1786828255.138215 mapread 0x022567 thru 0x02e709 (0xc1a3 bytes) 000708 1786828255.140403 mapread 0x007f91 thru 0x009077 (0x10e7 bytes) 000709 1786828255.142185 fallocate 0x02f4a2 thru 0x039a57 (0xa5b6 bytes) 000710 1786828255.147931 mapwrite 0x020033 thru 0x020748 (0x716 bytes) 000711 1786828255.160516 fallocate 0x02257d thru 0x02a708 (0x818c bytes) 000712 1786828255.179635 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xfa7a49f3 (stale) Mirror 11: 0xfa7a49f3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb3f19631 Mirror 12: 0xb3f19631 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000713 1786828256.470481 trunc from 0x040000 to 0x02479a (0x1b865 bytes) 000714 1786828256.547534 trunc from 0x002f1b to 0x0095c4 (0x66aa bytes) 000716 1786828256.553673 mapwrite 0x004315 thru 0x0046f3 (0x3df bytes) 000717 1786828256.574477 punch from 0x007d82 to 0x00c850 (0x4acf bytes) 000718 1786828256.582192 read 0x01f3ac thru 0x023095 (0x3cea bytes) 000719 1786828256.597318 mapread 0x014a05 thru 0x01bda9 (0x73a5 bytes) 000720 1786828256.627102 mapread 0x00fa1a thru 0x010395 (0x97c bytes) 000721 1786828256.628171 mapwrite 0x02e38f thru 0x031026 (0x2c98 bytes) 000722 1786828256.696479 mapwrite 0x02810b thru 0x03241d (0xa313 bytes) 000723 1786828256.763400 punch from 0x017e62 to 0x01fa1e (0x7bbd bytes) 000726 1786828256.771370 write 0x00991f thru 0x00e8ff (0x4fe1 bytes) 000727 1786828256.781525 mapwrite 0x020544 thru 0x0252ce (0x4d8b bytes) 000728 1786828256.818829 write 0x01ecba thru 0x02abca (0xbf11 bytes) 000729 1786828256.825123 mapwrite 0x03efb4 thru 0x03ffff (0x104c bytes) 000730 1786828256.882548 read 0x01244b thru 0x0188dc (0x6492 bytes) 000731 1786828256.894688 trunc from 0x040000 to 0x009580 (0x36a7f bytes) 000732 1786828256.911876 write 0x03abb4 thru 0x03acfa (0x147 bytes) 000733 1786828256.913315 trunc from 0x0038f7 to 0x00cc25 (0x932f bytes) 000734 1786828256.922664 punch from 0x009cb3 to 0x00f530 (0x587e bytes) 000735 1786828256.931334 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3acfb): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3acfb): Mirror 10: 0x486bf786 (stale) Mirror 11: 0x486bf786 (stale) Mirror 12: 0x7854ab6e (stale) CRC-32 checksum value for chunk [0, 0x3acfb): Mirror 1: 0x488f6b26 Mirror 13: 0x488f6b26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3acfb: skipped 000736 1786828258.044741 trunc from 0x03acfb to 0x027c9e (0x1305c bytes) 000737 1786828258.151953 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27c9f): Mirror 10: 0xe49b54d (stale) Mirror 11: 0xe49b54d (stale) Mirror 13: 0x201a01bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27c9f): Mirror 10: 0xe49b54d (stale) Mirror 11: 0xe49b54d (stale) Mirror 13: 0x201a01bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27c9f: skipped 000738 1786828258.641282 mapread 0x005fab thru 0x01180e (0xb864 bytes) 000739 1786828258.675069 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x27c9f): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27c9f): Mirror 10: 0xe49b54d (stale) Mirror 11: 0xe49b54d (stale) Mirror 13: 0x201a01bb (stale) CRC-32 checksum value for chunk [0, 0x27c9f): Mirror 1: 0x201a01bb Mirror 14: 0x201a01bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27c9f: skipped 000740 1786828259.677378 MIRROR_RESYNC 5 mirrors 000741 1786828260.602939 fallocate 0x035b0b thru 0x03b4e3 (0x59d9 bytes) 000742 1786828260.645384 mapread 0x028b15 thru 0x032fc3 (0xa4af bytes) 000743 1786828260.676172 MIRROR_RESYNC 5 mirrors 000744 1786828261.622666 trunc from 0x03b4e4 to 0x01e1aa (0x1d339 bytes) 000745 1786828261.662563 read 0x01245a thru 0x012a6c (0x613 bytes) 000746 1786828261.669202 mapwrite 0x01f852 thru 0x026797 (0x6f46 bytes) 000747 1786828261.733594 read 0x026481 thru 0x026797 (0x317 bytes) 000748 1786828261.735548 trunc from 0x0260bc to 0x026797 (0x6dc bytes) 000749 1786828261.742025 punch from 0x01de34 to 0x020a8f (0x2c5c bytes) 000750 1786828261.748645 write 0x01aabb thru 0x0232f0 (0x8836 bytes) 000751 1786828261.751406 punch from 0x024687 to 0x026797 (0x2111 bytes) 000752 1786828261.757993 fallocate 0x01d465 thru 0x023d25 (0x68c1 bytes) 000757 1786828261.766613 trunc from 0x01de76 to 0x021098 (0x3223 bytes) 000758 1786828261.781838 trunc from 0x026798 to 0x035928 (0xf191 bytes) 000759 1786828261.800652 punch from 0x029d3e to 0x035928 (0xbbeb bytes) 000760 1786828261.810035 MIRROR_RESYNC 5 mirrors 000761 1786828263.421047 mapwrite 0x03af02 thru 0x03ffff (0x50fe bytes) 000762 1786828263.516278 mapread 0x013cc0 thru 0x01c124 (0x8465 bytes) 000763 1786828263.546143 trunc from 0x040000 to 0x002780 (0x3d87f bytes) 000764 1786828263.573530 read 0x0011f5 thru 0x002780 (0x158c bytes) 000767 1786828263.584263 trunc from 0x0019d3 to 0x002780 (0xdae bytes) 000768 1786828263.591003 trunc from 0x00268b to 0x002780 (0xf6 bytes) 000771 1786828263.598255 write 0x034a05 thru 0x03ffff (0xb5fb bytes) 000772 1786828263.603639 fallocate 0x01b140 thru 0x01b373 (0x234 bytes) 000773 1786828263.631258 MIRROR_RESYNC 5 mirrors 000774 1786828264.788066 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000775 1786828265.298978 trunc from 0x040000 to 0x01f101 (0x20efe bytes) 000777 1786828265.390610 mapwrite 0x0313e9 thru 0x03e993 (0xd5ab bytes) 000778 1786828265.522232 read 0x00eae3 thru 0x01b45c (0xc97a bytes) 000779 1786828265.542266 trunc from 0x03e994 to 0x016a70 (0x27f23 bytes) 000780 1786828265.565396 trunc from 0x00a503 to 0x0152f1 (0xadef bytes) 000781 1786828265.578502 trunc from 0x00f8c0 to 0x016a70 (0x71b1 bytes) 000783 1786828265.587561 mapwrite 0x027d35 thru 0x027e37 (0x103 bytes) 000784 1786828265.641569 fallocate 0x01b5d7 thru 0x0200ef (0x4b19 bytes) 000785 1786828265.646969 punch from 0x0269cb to 0x027e37 (0x146d bytes) 000786 1786828265.652651 trunc from 0x027e38 to 0x018714 (0xf723 bytes) 000787 1786828265.681929 punch from 0x000559 to 0x00db35 (0xd5dd bytes) 000788 1786828265.695996 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x18715): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18715): Mirror 10: 0x1d861de6 (stale) Mirror 11: 0x1d861de6 (stale) Mirror 13: 0x1d861de6 (stale) CRC-32 checksum value for chunk [0, 0x18715): Mirror 1: 0x6e7c62fd Mirror 14: 0x6e7c62fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18715: skipped 000789 1786828266.761907 write 0x011696 thru 0x01ca3d (0xb3a8 bytes) 000790 1786828266.810841 trunc from 0x0145d3 to 0x01ca3d (0x846b bytes) 000791 1786828266.829279 read 0x00b6ae thru 0x00cef1 (0x1844 bytes) 000792 1786828266.843299 trunc from 0x01ca3e to 0x01c30b (0x732 bytes) 000793 1786828266.878954 mapread 0x007cc6 thru 0x00f0d5 (0x7410 bytes) 000794 1786828266.905176 mapread 0x001808 thru 0x00ef18 (0xd711 bytes) 000795 1786828266.917013 mapwrite 0x0084c8 thru 0x00eb7e (0x66b7 bytes) 000796 1786828266.938236 write 0x01b84f thru 0x026465 (0xac17 bytes) 000797 1786828266.946995 read 0x012fa3 thru 0x01c27f (0x92dd bytes) 000798 1786828266.957572 trunc from 0x01b2b4 to 0x026465 (0xb1b2 bytes) 000799 1786828266.973176 trunc from 0x004300 to 0x01009f (0xbda0 bytes) 000800 1786828266.979542 read 0x0116d5 thru 0x01aad9 (0x9405 bytes) 000801 1786828266.980332 write 0x017c42 thru 0x021596 (0x9955 bytes) 000802 1786828266.992164 fallocate 0x028735 thru 0x02dd6f (0x563b bytes) 000803 1786828266.997257 fallocate 0x01344b thru 0x0138f8 (0x4ae bytes) 000804 1786828267.002446 trunc from 0x02dd70 to 0x036521 (0x87b2 bytes) 000805 1786828267.023483 trunc from 0x014ed5 to 0x01a93b (0x5a67 bytes) 000806 1786828267.048547 read 0x02d8ea thru 0x036521 (0x8c38 bytes) 000808 1786828267.062489 mapwrite 0x01f44a thru 0x02ea5a (0xf611 bytes) 000809 1786828267.108935 trunc from 0x004b31 to 0x006e70 (0x2340 bytes) 000810 1786828267.119693 mapwrite 0x00712f thru 0x0155fb (0xe4cd bytes) 000811 1786828267.184931 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36522): Mirror 11: 0x4ccdb2f8 (stale) Mirror 13: 0x4ccdb2f8 (stale) Mirror 14: 0xb968405a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36522): Mirror 11: 0x4ccdb2f8 (stale) Mirror 13: 0x4ccdb2f8 (stale) Mirror 14: 0xb968405a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36522: skipped 000812 1786828267.772130 MIRROR_RESYNC 4 mirrors 000813 1786828269.301105 read 0x01b47e thru 0x02665f (0xb1e2 bytes) 000814 1786828269.312071 punch from 0x01920c to 0x021b20 (0x8915 bytes) 000815 1786828269.338159 trunc from 0x02f782 to 0x031d09 (0x2588 bytes) 000816 1786828269.343532 fallocate 0x000b04 thru 0x00259b (0x1a98 bytes) 000817 1786828269.349517 fallocate 0x0326c2 thru 0x03a16f (0x7aae bytes) 000818 1786828269.359990 fallocate 0x02a848 thru 0x02c029 (0x17e2 bytes) 000819 1786828269.368326 write 0x02b29c thru 0x0300bf (0x4e24 bytes) 000820 1786828269.381288 trunc from 0x036522 to 0x0390a8 (0x2b87 bytes) 000821 1786828269.408259 punch from 0x0314f1 to 0x03771d (0x622d bytes) 000822 1786828269.415004 trunc from 0x02c36c to 0x036558 (0xa1ed bytes) 000823 1786828269.430830 fallocate 0x028ad6 thru 0x03174c (0x8c77 bytes) 000824 1786828269.437104 mapwrite 0x0155b6 thru 0x023bd3 (0xe61e bytes) 000825 1786828269.515839 trunc from 0x0390a9 to 0x0150ec (0x23fbc bytes) 000826 1786828269.539133 fallocate 0x00cbbe thru 0x01a22b (0xd66e bytes) 000827 1786828269.551237 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a22c): Mirror 13: 0x850b3253 (stale) Mirror 14: 0x850b3253 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a22c): Mirror 13: 0x850b3253 (stale) Mirror 14: 0x850b3253 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a22c: skipped 000828 1786828269.874088 fallocate 0x00d624 thru 0x019c2c (0xc609 bytes) 000830 1786828269.888145 mapwrite 0x0307f5 thru 0x0360ac (0x58b8 bytes) 000831 1786828269.947985 punch from 0x00761d to 0x00d2dc (0x5cc0 bytes) 000833 1786828269.955317 punch from 0x003340 to 0x006b30 (0x37f1 bytes) 000834 1786828269.961177 write 0x0212f3 thru 0x022e3a (0x1b48 bytes) 000835 1786828269.981367 fallocate 0x008cad thru 0x00f589 (0x68dd bytes) 000837 1786828269.989370 mapread 0x025c17 thru 0x032a22 (0xce0c bytes) 000838 1786828270.008249 fallocate 0x012426 thru 0x021bd7 (0xf7b2 bytes) 000839 1786828270.024029 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x360ad): Mirror 13: 0x708e6e7f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x360ad): Mirror 13: 0x708e6e7f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x360ad: skipped 000840 1786828270.377010 mapread 0x034523 thru 0x0360ac (0x1b8a bytes) 000841 1786828270.408644 mapread 0x015ff9 thru 0x01b6bc (0x56c4 bytes) 000842 1786828270.430134 trunc from 0x0360ad to 0x01fd13 (0x16399 bytes) 000844 1786828270.486008 trunc from 0x01fd14 to 0x03b14d (0x1b43a bytes) 000845 1786828270.528551 trunc from 0x0255be to 0x025e84 (0x8c7 bytes) 000846 1786828270.536251 mapread 0x01fbbb thru 0x0270da (0x7520 bytes) 000847 1786828270.561869 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000848 1786828270.792502 fallocate 0x00a8e2 thru 0x00b563 (0xc82 bytes) 000849 1786828270.806941 punch from 0x037901 to 0x03b14d (0x384d bytes) 000851 1786828270.815219 mapread 0x02af53 thru 0x02d6d5 (0x2783 bytes) 000852 1786828270.854819 write 0x03f95c thru 0x03ffff (0x6a4 bytes) 000854 1786828270.868519 punch from 0x027e85 to 0x0379f7 (0xfb73 bytes) 000855 1786828270.878632 punch from 0x02b682 to 0x0387e4 (0xd163 bytes) 000856 1786828270.890219 fallocate 0x02255e thru 0x02947e (0x6f21 bytes) 000857 1786828270.899040 mapwrite 0x020551 thru 0x0258d1 (0x5381 bytes) 000858 1786828270.928915 mapread 0x01b252 thru 0x02606d (0xae1c bytes) 000859 1786828270.930640 fallocate 0x03a5f5 thru 0x03fa34 (0x5440 bytes) 000860 1786828270.937424 fallocate 0x00c3ef thru 0x017209 (0xae1b bytes) 000861 1786828270.943415 write 0x00cf03 thru 0x01bd0b (0xee09 bytes) 000864 1786828270.953239 punch from 0x00ebb1 to 0x015272 (0x66c2 bytes) 000868 1786828270.976132 MIRROR_EXTEND to 2 mirrors 000869 1786828272.039361 mapread 0x026d25 thru 0x028e2b (0x2107 bytes) 000871 1786828272.138387 trunc from 0x040000 to 0x01630c (0x29cf3 bytes) 000872 1786828272.213935 trunc from 0x01278d to 0x01630c (0x3b80 bytes) 000873 1786828272.220100 trunc from 0x004647 to 0x006505 (0x1ebf bytes) 000874 1786828272.225887 write 0x0196ad thru 0x02300e (0x9962 bytes) 000875 1786828272.235530 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000876 1786828272.410601 read 0x011e64 thru 0x0129ce (0xb6b bytes) 000877 1786828272.419748 trunc from 0x02300f to 0x020b77 (0x2497 bytes) 000878 1786828272.458805 trunc from 0x009d8c to 0x017721 (0xd996 bytes) 000879 1786828272.472429 read 0x01a5db thru 0x01e0d4 (0x3afa bytes) 000882 1786828272.491941 punch from 0x019021 to 0x01cf5c (0x3f3c bytes) 000883 1786828272.499207 mapread 0x01233f thru 0x020b77 (0xe839 bytes) 000884 1786828272.520368 mapwrite 0x00f301 thru 0x01a954 (0xb654 bytes) 000886 1786828272.540811 trunc from 0x020b78 to 0x01d7fe (0x3379 bytes) 000887 1786828272.556358 fallocate 0x02a1c4 thru 0x03880c (0xe649 bytes) 000890 1786828272.565149 mapread 0x01b7ba thru 0x02acb4 (0xf4fb bytes) 000891 1786828272.595428 trunc from 0x020c8a to 0x025169 (0x44e0 bytes) 000892 1786828272.603747 write 0x00e3f9 thru 0x0114c8 (0x30d0 bytes) 000893 1786828272.606256 trunc from 0x006aea to 0x00760d (0xb24 bytes) 000894 1786828272.612893 mapwrite 0x00976b thru 0x0143b9 (0xac4f bytes) 000897 1786828272.652747 mapwrite 0x03a544 thru 0x03ac4a (0x707 bytes) 000898 1786828272.693842 mapwrite 0x00fcf8 thru 0x011109 (0x1412 bytes) 000899 1786828272.715738 trunc from 0x025a12 to 0x0344d7 (0xeac6 bytes) 000900 1786828272.727002 read 0x02489e thru 0x028c74 (0x43d7 bytes) 000901 1786828272.738813 mapwrite 0x011685 thru 0x020a16 (0xf392 bytes) 000902 1786828272.767818 fallocate 0x031644 thru 0x03578d (0x414a bytes) 000903 1786828272.775470 write 0x036fcf thru 0x03a0c6 (0x30f8 bytes) 000905 1786828272.779483 fallocate 0x0214c2 thru 0x0297b3 (0x82f2 bytes) 000906 1786828272.792748 fallocate 0x02eeb1 thru 0x031f96 (0x30e6 bytes) 000907 1786828272.800918 trunc from 0x00e124 to 0x016cac (0x8b89 bytes) 000909 1786828272.808999 write 0x032add thru 0x03b6f6 (0x8c1a bytes) 000910 1786828272.816694 mapwrite 0x00070c thru 0x00d114 (0xca09 bytes) 000911 1786828272.854566 read 0x023725 thru 0x028e30 (0x570c bytes) 000912 1786828272.855914 punch from 0x02942a to 0x02ff81 (0x6b58 bytes) 000913 1786828272.871431 mapwrite 0x00f1a6 thru 0x019005 (0x9e60 bytes) 000914 1786828272.932275 punch from 0x010088 to 0x0154b5 (0x542e bytes) 000915 1786828272.943190 mapwrite 0x010630 thru 0x018c70 (0x8641 bytes) 000916 1786828273.038139 mapwrite 0x01a0ef thru 0x023d7b (0x9c8d bytes) 000918 1786828273.068052 trunc from 0x000e38 to 0x00ecfa (0xdec3 bytes) 000919 1786828273.075047 trunc from 0x02266e to 0x02fd2d (0xd6c0 bytes) 000920 1786828273.083504 mapread 0x0045ee thru 0x008dbf (0x47d2 bytes) 000921 1786828273.115444 trunc from 0x03b6f7 to 0x001844 (0x39eb2 bytes) 000922 1786828273.139471 mapwrite 0x00fc80 thru 0x01c5ba (0xc93b bytes) 000924 1786828273.290056 trunc from 0x01c5bb to 0x032606 (0x1604c bytes) 000925 1786828273.310388 write 0x03fe4f thru 0x03ffff (0x1b1 bytes) 000926 1786828273.311702 write 0x0047a7 thru 0x00e29f (0x9af9 bytes) 000927 1786828273.328527 mapwrite 0x00608d thru 0x0071b0 (0x1124 bytes) 000928 1786828273.383173 MIRROR_EXTEND to 2 mirrors 000929 1786828274.107465 read 0x0310b1 thru 0x039c67 (0x8bb7 bytes) 000930 1786828274.123995 punch from 0x034d9c to 0x03f040 (0xa2a5 bytes) 000931 1786828274.171805 mapread 0x00f208 thru 0x0159fb (0x67f4 bytes) 000932 1786828274.216668 trunc from 0x01af7d to 0x02213b (0x71bf bytes) 000933 1786828274.232094 write 0x01e186 thru 0x02e0b8 (0xff33 bytes) 000934 1786828274.273345 mapread 0x03c37d thru 0x03ffff (0x3c83 bytes) 000935 1786828274.299268 mapread 0x016c96 thru 0x01701a (0x385 bytes) 000937 1786828274.307054 mapread 0x0210f0 thru 0x0235ef (0x2500 bytes) 000938 1786828274.308135 punch from 0x038757 to 0x03ffff (0x78a9 bytes) 000940 1786828274.319970 trunc from 0x040000 to 0x009e06 (0x361f9 bytes) 000942 1786828274.416226 trunc from 0x007aa3 to 0x0089d8 (0xf36 bytes) 000943 1786828274.434827 write 0x0007ad thru 0x006e0c (0x6660 bytes) 000944 1786828274.455143 write 0x0133fa thru 0x01a438 (0x703f bytes) 000945 1786828274.458962 trunc from 0x01a439 to 0x017ee1 (0x2557 bytes) 000947 1786828274.496498 punch from 0x008d64 to 0x00a094 (0x1331 bytes) 000948 1786828274.516690 read 0x0038de thru 0x00db24 (0xa247 bytes) 000949 1786828274.536819 mapwrite 0x02c98a thru 0x03bfc4 (0xf63b bytes) 000950 1786828274.694040 punch from 0x006dea to 0x015734 (0xe94b bytes) 000951 1786828274.729349 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2c:0x0]: Invalid argument (22) 000952 1786828274.968690 trunc from 0x01ba1e to 0x02ac86 (0xf269 bytes) 000953 1786828274.975401 write 0x004839 thru 0x0124f0 (0xdcb8 bytes) 000954 1786828274.996424 mapwrite 0x02d410 thru 0x02f399 (0x1f8a bytes) 000955 1786828275.038945 trunc from 0x03bfc5 to 0x018cbd (0x23307 bytes) 000956 1786828275.061277 mapread 0x004514 thru 0x013384 (0xee71 bytes) 000958 1786828275.090382 trunc from 0x018cbe to 0x01ad32 (0x2075 bytes) 000959 1786828275.108522 trunc from 0x01ad33 to 0x01993e (0x13f4 bytes) 000961 1786828275.130993 read 0x00a740 thru 0x015e74 (0xb735 bytes) 000963 1786828275.133976 read 0x018d78 thru 0x01993e (0xbc7 bytes) 000964 1786828275.149152 trunc from 0x005e86 to 0x00cf71 (0x70ec bytes) 000965 1786828275.156836 fallocate 0x01e7bd thru 0x024470 (0x5cb4 bytes) 000966 1786828275.167913 trunc from 0x024471 to 0x026173 (0x1d03 bytes) 000969 1786828275.202025 read 0x011000 thru 0x016713 (0x5714 bytes) 000970 1786828275.202849 read 0x007ab1 thru 0x008822 (0xd72 bytes) 000971 1786828275.267621 mapread 0x010797 thru 0x013f26 (0x3790 bytes) 000973 1786828275.281207 MIRROR_EXTEND to 2 mirrors 000974 1786828276.352325 read 0x01ea60 thru 0x026173 (0x7714 bytes) 000975 1786828276.372324 fallocate 0x02ae54 thru 0x037132 (0xc2df bytes) 000976 1786828276.410885 trunc from 0x037133 to 0x02ac07 (0xc52b bytes) 000978 1786828276.448156 mapread 0x01070b thru 0x01b479 (0xad6f bytes) 000979 1786828276.480722 mapread 0x00a7f5 thru 0x0168d6 (0xc0e2 bytes) 000980 1786828276.496272 write 0x03e5fd thru 0x03ffff (0x1a03 bytes) 000981 1786828276.497564 trunc from 0x040000 to 0x0270df (0x18f20 bytes) 000982 1786828276.527899 MIRROR_RESYNC 2 mirrors 000984 1786828277.340444 mapwrite 0x037e58 thru 0x03b9d1 (0x3b7a bytes) 000985 1786828277.496272 mapwrite 0x015c9b thru 0x0212ea (0xb650 bytes) 000986 1786828277.638600 fallocate 0x007caf thru 0x0117e1 (0x9b33 bytes) 000987 1786828277.654907 punch from 0x01a0a5 to 0x027fb0 (0xdf0c bytes) 000988 1786828277.694569 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b9d2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9d2): Mirror 2: 0xc59a9cb6 (stale) CRC-32 checksum value for chunk [0, 0x3b9d2): Mirror 1: 0xbe33eb7c Mirror 3: 0xbe33eb7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9d2: skipped 000989 1786828279.164130 punch from 0x0103d6 to 0x013a39 (0x3664 bytes) 000990 1786828279.221755 mapread 0x00bf0f thru 0x00f6e3 (0x37d5 bytes) 000991 1786828279.259160 punch from 0x030fe3 to 0x03b9d1 (0xa9ef bytes) 000992 1786828279.267172 write 0x00932c thru 0x01692d (0xd602 bytes) 000993 1786828279.282933 trunc from 0x03b9d2 to 0x03e718 (0x2d47 bytes) 000995 1786828279.309008 fallocate 0x026461 thru 0x02bdc2 (0x5962 bytes) 000996 1786828279.315413 mapwrite 0x006a5f thru 0x010a67 (0xa009 bytes) 000999 1786828279.346903 read 0x01c295 thru 0x01c793 (0x4ff bytes) 001000 1786828279.356290 fallocate 0x0360ff thru 0x039506 (0x3408 bytes) All operations completed A-OK! PASS 70a (105s) == sanity-flr test 71: check mirror extend parent fid ==== 17:11:25 (1786828285) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x61:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (15s) == sanity-flr test 72: check append on FLR file ========== 17:11:40 (1786828300) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x129:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191674 s, 10.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x129:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203641 s, 5.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x129:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x129:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (13s) == sanity-flr test 200a: stress test ===================== 17:11:53 (1786828313) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1984704 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 705824 .. Extending file size to 6854592 .. Extending file size to 9248896 .. Extending file size to 7046656 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7225408 .. Extending file size to 7310848 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8658144 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61279 61281 61282 61285 61286 done 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) 192.168.204.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:13:17 (1786828397) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2538816 .. Extending file size to 6176352 .. done Extending file size to 8352352 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7843616 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8220704 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8772640 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8845248 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62754 62755 62756 62758 62760 done 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) 192.168.204.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:14:43 (1786828483) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2376 1268928 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2064 1269240 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25196 3581824 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17508 3589512 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42704 7171336 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2376 1268928 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2064 1269240 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25196 3581824 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17508 3589512 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42704 7171336 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0896971 s, 23.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.223109 s, 9.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x50b135e6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:15:04 (1786828504) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0802808 s, 26.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 17:15:12 (1786828512) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x140:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x140:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x141:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x13f:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:15:20 (1786828520) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118698 s, 17.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:15:29 (1786828529) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0875832 s, 23.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x142:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x143:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x140:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x142:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x143:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x140:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x141:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:15:38 (1786828538) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127315 s, 16.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f4:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:16:14 (1786828574) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134505 s, 7.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f3:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:17:06 (1786828626) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x142:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x143:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100407 s, 20.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x142:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x143:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x144:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x142:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x143:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x144:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:17:59 (1786828679) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg419-server: Pool lustre.test_204f created Adding targets to pool oleg419-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg419-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136852 s, 15.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x145:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x145:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg419-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg419-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg419-server: Pool lustre.test_204f destroyed PASS 204f (69s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:19:08 (1786828748) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:19:17 (1786828757) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:19:26 (1786828766) oleg419-server: Pool lustre.test_206 created Adding targets to pool oleg419-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg419-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg419-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg419-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg419-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:19:53 (1786828793) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505383 s, 20.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281075 s, 37.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x756ba8a Mirror 3: 0x756ba8a PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:20:01 (1786828801) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:20:04 (1786828804) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:20:08 (1786828808) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0692326 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1a50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2150000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2150000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2150000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x23a0000 Read this (4 bytes): BBBB 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:20:43 (1786828843) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0346569 s, 236 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:20:52 (1786828852) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577759 s, 18.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x8f lmm_fid: [0x200000405:0x8f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x8f lmm_fid: [0x200000405:0x8f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611279 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-15 17:20:53.000000000 -0400 Modify: 2026-08-15 17:20:53.000000000 -0400 Change: 2026-08-15 17:20:55.000000000 -0400 Birth: 2026-08-15 17:20:54.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:21:01 (1786828861) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03586 s, 29.2 MB/s oleg419-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg419-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg419-server: O/280000401/d14/14e: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:21:21 (1786828881) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.243713 s, 168 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0214931 s, 191 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 15 17:21 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x94:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 15 17:21 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:21:30 (1786828890) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:21:38 (1786828898) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1951 sec =========== 17:21:46 (1786828906) === sanity-flr: start cleanup 17:21:48 (1786828908) === === sanity-flr: finish cleanup 17:21:56 (1786828916) ===