-----============= acceptance-small: sanity-flr ============----- Fri Aug 21 22:18:14 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg451-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:18:36 (1787365116) === oleg451-client.virtnet: executing check_config_client /mnt/lustre oleg451-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg451-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff953151394000.idle_timeout=debug osc.lustre-OST0001-osc-ffff953151394000.idle_timeout=debug disable quota as required oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 22:19:19 (1787365159) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8232] == sanity-flr test 0a: lfs mirror create with -N option == 22:19:21 (1787365161) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:19:42 (1787365182) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:19:45 (1787365185) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:19:49 (1787365189) PASS 0d (31s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:20:21 (1787365221) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:20:25 (1787365225) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 22:20:29 (1787365229) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff953151394000.stats=clear osc.lustre-OST0001-osc-ffff953151394000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:20:37 (1787365237) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:20:49 (1787365249) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:20:58 (1787365258) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:21:02 (1787365262) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:21:10 (1787365270) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 22:21:20 (1787365280) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:21:28 (1787365288) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:21:36 (1787365296) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:21:44 (1787365304) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:21:55 (1787365315) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:22:02 (1787365322) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.609106 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162788 s, 6.4 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:22:12 (1787365332) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.619895 s, 25.4 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:22:20 (1787365340) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:22:31 (1787365351) Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff953151394000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff953151394000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg451-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff953151394000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff953151394000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg451-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (136s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:24:47 (1787365487) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143366 s, 14.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff953151394000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff953151394000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:25:36 (1787365536) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff953151394000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff953151394000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff953151394000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff953151394000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg451-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:27:05 (1787365625) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.02437 s, 17.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 21 22:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.109s user 0m0.001s sys 0m0.169s Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff953151394000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff953151394000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.081s user 0m0.001s sys 0m0.194s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff953151394000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff953151394000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.118s user 0m0.001s sys 0m0.169s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg451-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg451-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff953151394000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff953151394000.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:28:45 (1787365725) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:28:48 (1787365728) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:28:51 (1787365731) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 22:28:54 (1787365734) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 22:29:03 (1787365743) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.617659 s, 20.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:29:16 (1787365756) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344557 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321461 s, 32.6 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:29:23 (1787365763) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:29:31 (1787365771) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00077009 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00269297 s, 380 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00149009 s, 687 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00274117 s, 374 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 22:29:42 (1787365782) create test files with size 15735147 .. 08d60a8d3fe7752847f661b46879569f - 18345a896873c344b1472dd473852c5b - 1ccfd2e04a1961a38e3088e19ba447d8 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg451-client.virtnet /mnt/lustre (opts:) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 22:30:06 (1787365806) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:30:23 (1787365823) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:30:31 (1787365831) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104525 s, 10.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278825 s, 37.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 22:30:43 (1787365843) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120092 s, 17.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.339145 s, 12.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.196653 s, 10.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 22:31:03 (1787365863) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 22:31:06 (1787365866) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:31:10 (1787365870) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605777 s, 17.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 22:31:19 (1787365879) llite.lustre-ffff953168731800.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 22:31:27 (1787365887) llite.lustre-ffff953168731800.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326978 s, 32.1 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773723 s, 13.6 MB/s **verify components PASS 43d (10s) == sanity-flr test 44a: lfs mirror split check =========== 22:31:37 (1787365897) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.373191 s, 8.4 MB/s PASS 44a (13s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:31:50 (1787365910) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:32:02 (1787365922) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.220354 s, 47.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg451-client.virtnet /mnt/lustre (opts:) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:32:15 (1787365935) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff953146045000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.32035 s, 32.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg451-client.virtnet /mnt/lustre (opts:) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff953168717000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:32:33 (1787365953) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:33:15 (1787365995) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:33:55 (1787366035) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 22:34:00 (1787366040) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.329362 s, 9.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.145177 s, 21.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4275b0c7 (nosync,stale) Mirror 2: 0x4275b0c7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: d8c2bc5c2da3b671d1cc888b34266dd2 - mirror 1 checksum: 74cde9ba2f90b1f46a4a9acb22ea8b07 - mirror 2 checksum: d8c2bc5c2da3b671d1cc888b34266dd2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-21T22:34:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-21T22:34:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd8:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d8c2bc5c2da3b671d1cc888b34266dd2 - mirror 1 checksum: 74cde9ba2f90b1f46a4a9acb22ea8b07 - mirror 2 checksum: 74cde9ba2f90b1f46a4a9acb22ea8b07 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-21T22:34:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-21T22:34:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd8:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 22:34:12 (1787366052) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465248 s, 22.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:34:20 (1787366060) Failing mds1 on oleg451-server Stopping /mnt/lustre-mds1 (opts:) on oleg451-server 22:34:33 (1787366073) shut down facet: mds1 facet_host: oleg451-server facet_failover_host: oleg451-server Failover mds1 to oleg451-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 22:35:05 (1787366105) targets are mounted 22:35:05 (1787366105) facet_failover done oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03058 s, 34.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xda:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:35:21 (1787366121) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.047736 s, 429 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0234387 s, 874 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:35:32 (1787366132) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0487673 s, 420 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00712924 s, 2.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:35:42 (1787366142) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00588741 s, 2.8 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:35:50 (1787366150) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0644602 s, 318 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:35:59 (1787366159) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0223353 s, 183 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0360486 s, 114 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:36:07 (1787366167) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:36:31 (1787366191) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:36:59 (1787366219) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:37:23 (1787366243) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:37:48 (1787366268) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 22:37:57 (1787366277) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 58877 mirror split pid 58878 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58877 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58878 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 22:39:05 (1787366345) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6347 000002 1787366347.613624 MIRROR_EXTEND to 2 mirrors skipping zero size read 000004 1787366348.073580 write 0x032ac8 thru 0x038c02 (0x613b bytes) 000005 1787366348.114441 write 0x03f212 thru 0x03ffff (0xdee bytes) 000006 1787366348.115793 read 0x00495a thru 0x01012a (0xb7d1 bytes) fallocating to largest ever: 0x384b9 000007 1787366348.125903 fallocate 0x02d694 thru 0x0384b8 (0xae25 bytes) 000008 1787366348.132279 read 0x02a74e thru 0x03890c (0xe1bf bytes) 000009 1787366348.149633 read 0x02b899 thru 0x030d59 (0x54c1 bytes) 000010 1787366348.150836 trunc from 0x040000 to 0x01e582 (0x21a7d bytes) 000011 1787366348.222728 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000013 1787366348.414943 trunc from 0x00b29f to 0x00b6ec (0x44e bytes) 000014 1787366348.427549 MIRROR_EXTEND to 2 mirrors 000015 1787366349.138774 read 0x00306f thru 0x00e1ec (0xb17e bytes) 000016 1787366349.167329 mapread 0x00f561 thru 0x01aee2 (0xb982 bytes) 000017 1787366349.225715 trunc from 0x01e583 to 0x001d79 (0x1c809 bytes) 000018 1787366349.318514 MIRROR_RESYNC 2 mirrors 000019 1787366349.881440 mapread 0x001b8c thru 0x001d79 (0x1ee bytes) 000021 1787366349.898533 write 0x007b68 thru 0x013a39 (0xbed2 bytes) 000022 1787366349.952251 fallocate 0x0039e0 thru 0x00720c (0x382d bytes) 000024 1787366349.960802 write 0x004adb thru 0x01481b (0xfd41 bytes) 000025 1787366349.967796 fallocate 0x0250eb thru 0x027f48 (0x2e5e bytes) 000026 1787366349.973646 read 0x00942c thru 0x00b2ca (0x1e9f bytes) 000027 1787366349.974789 read 0x005124 thru 0x008119 (0x2ff6 bytes) 000028 1787366349.976142 mapread 0x00cfa7 thru 0x0118c0 (0x491a bytes) 000029 1787366349.977773 mapwrite 0x00a7de thru 0x00c9d1 (0x21f4 bytes) 000030 1787366349.994219 mapwrite 0x0326a2 thru 0x03ffff (0xd95e bytes) 000031 1787366350.049619 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb0b3ca4 Mirror 3: 0xb0b3ca4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped fallocating to largest ever: 0x3c0af 000032 1787366351.305223 fallocate 0x03a936 thru 0x03c0ae (0x1779 bytes) fallocating to largest ever: 0x3e636 000033 1787366351.345923 fallocate 0x031aba thru 0x03e635 (0xcb7c bytes) 000034 1787366351.360750 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0b3ca4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0b3ca4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000035 1787366351.823625 mapread 0x0155f7 thru 0x01f424 (0x9e2e bytes) 000036 1787366351.903662 read 0x00e815 thru 0x017317 (0x8b03 bytes) 000037 1787366351.939169 trunc from 0x037694 to 0x039164 (0x1ad1 bytes) 000038 1787366351.950685 mapread 0x036ad0 thru 0x037607 (0xb38 bytes) 000039 1787366351.973065 mapwrite 0x002c46 thru 0x00adff (0x81ba bytes) 000040 1787366352.005038 mapread 0x02fe7d thru 0x03338f (0x3513 bytes) 000042 1787366352.011766 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb0b3ca4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa53f30e7 Mirror 4: 0xa53f30e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000043 1787366353.338504 mapread 0x0379fb thru 0x03d91f (0x5f25 bytes) 000044 1787366353.360185 trunc from 0x032933 to 0x03380f (0xedd bytes) 000045 1787366353.416885 trunc from 0x040000 to 0x002f47 (0x3d0b8 bytes) 000046 1787366353.483667 trunc from 0x002f48 to 0x021445 (0x1e4fe bytes) 000047 1787366353.504154 fallocate 0x001cfe thru 0x00d675 (0xb978 bytes) 000048 1787366353.510622 trunc from 0x021446 to 0x001eb8 (0x1f58d bytes) 000049 1787366353.534502 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1eb9): Mirror 4: 0x7f10a2f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eb9): Mirror 4: 0x7f10a2f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eb9: skipped 000050 1787366353.916734 punch from 0x0000d9 to 0x001eb8 (0x1de0 bytes) 000051 1787366353.924749 fallocate 0x01d689 thru 0x02cf8b (0xf903 bytes) 000052 1787366353.929328 punch from 0x01e3b6 to 0x0257c7 (0x7412 bytes) 000053 1787366353.936195 write 0x00f22d thru 0x0185d5 (0x93a9 bytes) 000054 1787366353.956528 write 0x0093db thru 0x01017d (0x6da3 bytes) 000055 1787366353.962992 trunc from 0x009f6e to 0x019160 (0xf1f3 bytes) 000056 1787366353.997355 write 0x01011d thru 0x018749 (0x862d bytes) 000057 1787366354.026490 read 0x01a19f thru 0x02979e (0xf600 bytes) 000058 1787366354.042062 trunc from 0x02ce8f to 0x02cf8b (0xfd bytes) 000059 1787366354.047292 punch from 0x01c041 to 0x0289ca (0xc98a bytes) 000060 1787366354.053573 trunc from 0x01e9b7 to 0x021662 (0x2cac bytes) 000062 1787366354.069593 MIRROR_RESYNC 2 mirrors 000063 1787366354.948141 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000065 1787366355.146598 trunc from 0x01d9f2 to 0x02450c (0x6b1b bytes) 000066 1787366355.166076 trunc from 0x02cf8c to 0x01546a (0x17b21 bytes) 000067 1787366355.247616 write 0x03c160 thru 0x03ffff (0x3ea0 bytes) 000069 1787366355.264626 MIRROR_EXTEND to 2 mirrors 000070 1787366356.440015 trunc from 0x040000 to 0x02a4ee (0x15b11 bytes) 000071 1787366356.508269 trunc from 0x018d94 to 0x026778 (0xd9e5 bytes) 000072 1787366356.518680 punch from 0x003c50 to 0x011410 (0xd7c1 bytes) fallocating to largest ever: 0x3f2b4 000074 1787366356.531882 fallocate 0x036309 thru 0x03f2b3 (0x8fab bytes) 000075 1787366356.540476 trunc from 0x011dc0 to 0x012764 (0x9a5 bytes) fallocating to largest ever: 0x40000 000076 1787366356.551652 fallocate 0x038ace thru 0x03ffff (0x7532 bytes) 000078 1787366356.568506 read 0x00ba7c thru 0x01026c (0x47f1 bytes) 000079 1787366356.580765 trunc from 0x0247c3 to 0x02e6d8 (0x9f16 bytes) 000080 1787366356.591476 mapwrite 0x01f761 thru 0x02c7cb (0xd06b bytes) 000081 1787366356.672917 write 0x0314e8 thru 0x033aeb (0x2604 bytes) 000082 1787366356.680149 fallocate 0x02b2f7 thru 0x038576 (0xd280 bytes) 000083 1787366356.691672 fallocate 0x008658 thru 0x00a170 (0x1b19 bytes) 000084 1787366356.711487 read 0x005f45 thru 0x013d5f (0xde1b bytes) 000085 1787366356.778070 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000086 1787366357.039919 fallocate 0x02915a thru 0x03239c (0x9243 bytes) 000087 1787366357.045583 punch from 0x02d12c to 0x032ed1 (0x5da6 bytes) 000088 1787366357.054388 trunc from 0x03f2b4 to 0x01aa18 (0x2489b bytes) 000089 1787366357.077915 MIRROR_EXTEND to 2 mirrors 000091 1787366357.928303 trunc from 0x01aa19 to 0x0280c6 (0xd6ae bytes) 000092 1787366357.979404 MIRROR_RESYNC 2 mirrors 000093 1787366358.509184 trunc from 0x0280c7 to 0x0216cb (0x69fb bytes) 000094 1787366358.585006 trunc from 0x013e22 to 0x01e64f (0xa82e bytes) 000095 1787366358.595772 trunc from 0x00fd35 to 0x01c138 (0xc404 bytes) 000096 1787366358.608117 trunc from 0x003e6b to 0x006979 (0x2b0f bytes) 000097 1787366358.615555 mapread 0x0074a9 thru 0x013f06 (0xca5e bytes) 000098 1787366358.661437 trunc from 0x0216cc to 0x01b008 (0x66c3 bytes) 000099 1787366358.689662 write 0x011410 thru 0x0134b0 (0x20a1 bytes) 000104 1787366358.695833 punch from 0x0164ce to 0x01b008 (0x4b3b bytes) 000105 1787366358.702911 fallocate 0x021475 thru 0x02419d (0x2d29 bytes) 000106 1787366358.725166 trunc from 0x01b009 to 0x004490 (0x16b78 bytes) 000107 1787366358.741237 trunc from 0x0002c9 to 0x0003c1 (0xf9 bytes) 000108 1787366358.747453 mapread 0x002d1d thru 0x003eb4 (0x1198 bytes) 000109 1787366358.748569 read 0x003877 thru 0x004490 (0xc1a bytes) 000111 1787366358.750904 mapread 0x001667 thru 0x004490 (0x2e2a bytes) 000112 1787366358.758513 write 0x00e09d thru 0x0128a2 (0x4806 bytes) 000113 1787366358.763427 write 0x0365fa thru 0x03b21f (0x4c26 bytes) 000114 1787366358.767213 punch from 0x039947 to 0x03b21f (0x18d9 bytes) 000115 1787366358.819906 fallocate 0x00d184 thru 0x0150ce (0x7f4b bytes) 000116 1787366358.831708 trunc from 0x03b220 to 0x022ae2 (0x1873d bytes) 000117 1787366358.851602 punch from 0x0099ee to 0x00fbb7 (0x61ca bytes) 000118 1787366358.857520 fallocate 0x02070e thru 0x02a6bd (0x9fb0 bytes) 000119 1787366358.862679 mapread 0x0165b1 thru 0x017884 (0x12d4 bytes) 000120 1787366358.881533 mapwrite 0x02dfd0 thru 0x03c632 (0xe663 bytes) 000121 1787366358.971436 punch from 0x00527f to 0x00c1c6 (0x6f48 bytes) 000122 1787366358.976861 mapread 0x02ddf0 thru 0x02e2ff (0x510 bytes) 000123 1787366358.977933 write 0x02b7e2 thru 0x0302d7 (0x4af6 bytes) 000125 1787366358.980193 mapwrite 0x02f11a thru 0x03e64b (0xf532 bytes) 000126 1787366359.050758 trunc from 0x032c92 to 0x03e64b (0xb9ba bytes) 000127 1787366359.075348 mapwrite 0x00a40e thru 0x00c2cd (0x1ec0 bytes) 000128 1787366359.126909 fallocate 0x00fb48 thru 0x01f54b (0xfa04 bytes) 000129 1787366359.136656 mapread 0x0105d8 thru 0x015c0d (0x5636 bytes) 000130 1787366359.140009 write 0x01e011 thru 0x024362 (0x6352 bytes) 000131 1787366359.155296 mapread 0x03c291 thru 0x03e64b (0x23bb bytes) 000132 1787366359.177611 trunc from 0x03e64c to 0x015846 (0x28e05 bytes) 000133 1787366359.224466 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000134 1787366359.470273 fallocate 0x03f72c thru 0x03ffff (0x8d4 bytes) 000136 1787366359.484411 write 0x0382c4 thru 0x03ffff (0x7d3c bytes) 000137 1787366359.493814 mapwrite 0x0222e3 thru 0x02a601 (0x831f bytes) 000138 1787366359.575612 read 0x035f6d thru 0x03e7ed (0x8881 bytes) 000139 1787366359.585489 read 0x03ebd3 thru 0x03ffff (0x142d bytes) 000140 1787366359.591222 trunc from 0x040000 to 0x02c4ca (0x13b35 bytes) 000141 1787366359.624517 mapread 0x00e12e thru 0x01df44 (0xfe17 bytes) 000142 1787366359.655670 MIRROR_EXTEND to 2 mirrors 000143 1787366360.709119 fallocate 0x00df91 thru 0x017d5a (0x9dca bytes) 000144 1787366360.790917 fallocate 0x0000b8 thru 0x00f8b1 (0xf7fa bytes) 000145 1787366360.800735 mapread 0x0116c5 thru 0x01dd53 (0xc68f bytes) 000146 1787366360.842867 fallocate 0x013151 thru 0x018d2c (0x5bdc bytes) 000147 1787366360.848429 read 0x00664b thru 0x00ab2c (0x44e2 bytes) 000148 1787366360.861315 trunc from 0x02c4cb to 0x014f1a (0x175b0 bytes) 000149 1787366360.892195 read 0x00528d thru 0x00e488 (0x91fc bytes) 000150 1787366360.906404 mapwrite 0x02eff4 thru 0x03e920 (0xf92d bytes) 000152 1787366360.986052 fallocate 0x027e92 thru 0x029b09 (0x1c78 bytes) 000153 1787366361.002805 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000154 1787366361.243082 trunc from 0x0070ae to 0x013acf (0xca22 bytes) 000155 1787366361.257809 read 0x00f8cd thru 0x01e9e0 (0xf114 bytes) 000157 1787366361.276483 fallocate 0x036587 thru 0x037322 (0xd9c bytes) 000158 1787366361.294764 trunc from 0x03e0d1 to 0x03e920 (0x850 bytes) 000159 1787366361.309313 read 0x025a50 thru 0x02f031 (0x95e2 bytes) 000161 1787366361.343329 write 0x03aa7c thru 0x03f36f (0x48f4 bytes) 000162 1787366361.351315 fallocate 0x01feed thru 0x02fa45 (0xfb59 bytes) 000164 1787366361.358457 mapread 0x01838d thru 0x019e04 (0x1a78 bytes) 000165 1787366361.359863 read 0x0380d8 thru 0x03f36f (0x7298 bytes) 000166 1787366361.360867 trunc from 0x03f370 to 0x009a77 (0x358f8 bytes) 000168 1787366361.394847 fallocate 0x0269ab thru 0x030ceb (0xa341 bytes) 000169 1787366361.406073 read 0x007698 thru 0x009a77 (0x23e0 bytes) 000170 1787366361.407526 trunc from 0x008c94 to 0x009a77 (0xde4 bytes) 000171 1787366361.417525 trunc from 0x009a78 to 0x01073b (0x6cc4 bytes) 000174 1787366361.464913 trunc from 0x01073c to 0x02b7c7 (0x1b08c bytes) 000175 1787366361.496762 punch from 0x005135 to 0x00a79f (0x566b bytes) 000176 1787366361.505889 mapwrite 0x027dea thru 0x037b74 (0xfd8b bytes) 000178 1787366361.612404 MIRROR_EXTEND to 2 mirrors 000179 1787366362.513233 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000181 1787366362.707353 trunc from 0x037b75 to 0x03809f (0x52b bytes) 000182 1787366362.749767 mapread 0x02283f thru 0x0248dc (0x209e bytes) 000183 1787366362.769911 read 0x015f2f thru 0x01952f (0x3601 bytes) 000184 1787366362.777904 fallocate 0x033ae1 thru 0x03ffff (0xc51f bytes) 000185 1787366362.783097 write 0x007025 thru 0x008561 (0x153d bytes) 000186 1787366362.795135 punch from 0x002691 to 0x00ff0f (0xd87f bytes) 000187 1787366362.805881 fallocate 0x014835 thru 0x01a4e6 (0x5cb2 bytes) 000188 1787366362.811382 fallocate 0x02be93 thru 0x02d76a (0x18d8 bytes) 000189 1787366362.816160 write 0x01637e thru 0x01faa1 (0x9724 bytes) 000190 1787366362.818280 write 0x029a21 thru 0x033f22 (0xa502 bytes) 000191 1787366362.830354 mapread 0x036222 thru 0x03809f (0x1e7e bytes) 000192 1787366362.841832 trunc from 0x0380a0 to 0x038f47 (0xea8 bytes) 000193 1787366362.878907 punch from 0x01de0e to 0x026d69 (0x8f5c bytes) 000197 1787366362.899111 mapwrite 0x0183d4 thru 0x0253f9 (0xd026 bytes) 000198 1787366362.931938 fallocate 0x012d13 thru 0x0198da (0x6bc8 bytes) 000199 1787366362.938135 mapread 0x013d01 thru 0x01d148 (0x9448 bytes) 000200 1787366362.953651 mapread 0x028ab4 thru 0x03160f (0x8b5c bytes) 000203 1787366362.956783 mapwrite 0x034abe thru 0x03fd08 (0xb24b bytes) 000204 1787366363.020658 mapread 0x03151c thru 0x032e54 (0x1939 bytes) 000205 1787366363.021563 mapread 0x01f03a thru 0x02a257 (0xb21e bytes) 000207 1787366363.031547 mapwrite 0x034112 thru 0x03ffff (0xbeee bytes) 000208 1787366363.071229 mapread 0x0356ff thru 0x03bd98 (0x669a bytes) 000209 1787366363.072744 write 0x02c9bd thru 0x02d3e6 (0xa2a bytes) 000211 1787366363.074721 mapwrite 0x02f960 thru 0x0309fd (0x109e bytes) 000212 1787366363.092414 trunc from 0x040000 to 0x03bc08 (0x43f7 bytes) 000214 1787366363.110442 trunc from 0x03bc09 to 0x01c9c6 (0x1f242 bytes) 000215 1787366363.138781 mapwrite 0x019323 thru 0x022e46 (0x9b24 bytes) 000216 1787366363.190718 mapwrite 0x0354dd thru 0x03ffff (0xab23 bytes) 000217 1787366363.269518 mapread 0x00376d thru 0x0073ce (0x3c62 bytes) 000218 1787366363.279265 mapwrite 0x01dfa7 thru 0x01e831 (0x88b bytes) 000219 1787366363.303504 punch from 0x012722 to 0x018021 (0x5900 bytes) 000220 1787366363.309875 mapread 0x01606d thru 0x01ef05 (0x8e99 bytes) 000221 1787366363.319885 fallocate 0x038a3a thru 0x03ed8b (0x6352 bytes) 000223 1787366363.325912 fallocate 0x01fa80 thru 0x02b5ee (0xbb6f bytes) 000225 1787366363.332341 write 0x03a95f thru 0x03ffff (0x56a1 bytes) 000226 1787366363.333604 mapwrite 0x009192 thru 0x014d6b (0xbbda bytes) 000230 1787366363.358367 read 0x02a49e thru 0x03a1b6 (0xfd19 bytes) 000231 1787366363.372310 write 0x003c38 thru 0x011df7 (0xe1c0 bytes) 000232 1787366363.383871 MIRROR_EXTEND to 2 mirrors 000233 1787366364.066875 MIRROR_EXTEND to 3 mirrors 000234 1787366364.989668 punch from 0x03965d to 0x03ffff (0x69a3 bytes) 000235 1787366365.032404 mapwrite 0x0324b9 thru 0x03ffff (0xdb47 bytes) 000236 1787366365.084397 read 0x029175 thru 0x02fe84 (0x6d10 bytes) 000237 1787366365.092703 trunc from 0x005c9f to 0x00d11a (0x747c bytes) 000238 1787366365.097843 write 0x00cece thru 0x0151f8 (0x832b bytes) 000239 1787366365.116944 trunc from 0x040000 to 0x03ebbe (0x1441 bytes) 000240 1787366365.146441 write 0x00f0b1 thru 0x011e81 (0x2dd1 bytes) 000243 1787366365.149851 trunc from 0x03e0c9 to 0x03ebbe (0xaf6 bytes) 000244 1787366365.156146 punch from 0x027134 to 0x0352bb (0xe188 bytes) 000245 1787366365.162362 read 0x034208 thru 0x0394f4 (0x52ed bytes) 000246 1787366365.197350 write 0x009b1f thru 0x019618 (0xfafa bytes) 000247 1787366365.199015 write 0x002971 thru 0x0070af (0x473f bytes) 000248 1787366365.205408 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ebbf): Mirror 2: 0x9413ca91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ebbf): Mirror 2: 0x9413ca91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ebbf: skipped 000251 1787366365.634563 trunc from 0x00c49b to 0x0145e7 (0x814d bytes) 000252 1787366365.646908 read 0x02b4ed thru 0x037d4e (0xc862 bytes) 000253 1787366365.670220 punch from 0x03a9ec to 0x03e7db (0x3df0 bytes) 000255 1787366365.681915 read 0x0020ca thru 0x00e4e5 (0xc41c bytes) 000256 1787366365.712407 fallocate 0x02d030 thru 0x0310d0 (0x40a1 bytes) 000257 1787366365.721029 punch from 0x03d6a0 to 0x03ebbe (0x151f bytes) 000258 1787366365.739532 trunc from 0x03ebbf to 0x00304f (0x3bb6f bytes) 000259 1787366365.767423 write 0x03d486 thru 0x03fe3c (0x29b7 bytes) 000260 1787366365.768608 read 0x02adea thru 0x037aba (0xccd1 bytes) 000261 1787366365.783201 fallocate 0x00d858 thru 0x013941 (0x60ea bytes) 000262 1787366365.792233 punch from 0x02c143 to 0x02de51 (0x1d0f bytes) 000263 1787366365.804381 mapwrite 0x02e419 thru 0x0381e6 (0x9dce bytes) 000264 1787366365.875197 MIRROR_RESYNC 2 mirrors 000265 1787366366.830872 punch from 0x026048 to 0x031142 (0xb0fb bytes) 000266 1787366366.880893 mapwrite 0x00bbbb thru 0x00d938 (0x1d7e bytes) 000267 1787366366.952884 trunc from 0x02e130 to 0x03923b (0xb10c bytes) 000268 1787366366.966811 trunc from 0x03fe3d to 0x03cb86 (0x32b6 bytes) 000269 1787366367.018365 read 0x006be0 thru 0x01636e (0xf78f bytes) 000272 1787366367.106751 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cb87): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb87): Mirror 2: 0x50a5520b (stale) CRC-32 checksum value for chunk [0, 0x3cb87): Mirror 1: 0x2b10020c Mirror 3: 0x2b10020c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb87: skipped 000273 1787366368.295694 read 0x0378ad thru 0x03cb86 (0x52da bytes) 000274 1787366368.304565 punch from 0x0076e4 to 0x0106c5 (0x8fe2 bytes) 000275 1787366368.344171 mapwrite 0x01f972 thru 0x022436 (0x2ac5 bytes) 000276 1787366368.406852 trunc from 0x0199e5 to 0x0270ce (0xd6ea bytes) 000277 1787366368.419186 punch from 0x03042f to 0x038213 (0x7de5 bytes) 000278 1787366368.426194 punch from 0x021aac to 0x0229c0 (0xf15 bytes) 000279 1787366368.437761 read 0x013f75 thru 0x019f00 (0x5f8c bytes) 000280 1787366368.471087 fallocate 0x0355a1 thru 0x03f81c (0xa27c bytes) 000281 1787366368.477267 mapread 0x006f7c thru 0x014871 (0xd8f6 bytes) 000282 1787366368.478828 punch from 0x0359a6 to 0x039923 (0x3f7e bytes) 000283 1787366368.484385 mapwrite 0x010b8b thru 0x01d930 (0xcda6 bytes) 000284 1787366368.508798 mapwrite 0x0302c7 thru 0x034a28 (0x4762 bytes) 000285 1787366368.540659 write 0x037a18 thru 0x03ffff (0x85e8 bytes) 000286 1787366368.558839 write 0x03b7fa thru 0x03ffff (0x4806 bytes) 000287 1787366368.573708 trunc from 0x0180d7 to 0x0237b0 (0xb6da bytes) 000288 1787366368.579159 read 0x036a9b thru 0x03ffff (0x9565 bytes) 000289 1787366368.598090 trunc from 0x040000 to 0x03c98e (0x3671 bytes) 000290 1787366368.644839 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c98f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c98f): Mirror 2: 0xdef24a7 (stale) Mirror 3: 0x79485d41 (stale) CRC-32 checksum value for chunk [0, 0x3c98f): Mirror 1: 0x52c8196 Mirror 4: 0x52c8196 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c98f: skipped 000291 1787366369.949297 trunc from 0x02ab8b to 0x02c3f3 (0x1869 bytes) 000292 1787366369.992994 mapread 0x034ce0 thru 0x03c98e (0x7caf bytes) 000293 1787366370.011358 mapwrite 0x00b0f2 thru 0x00e0b1 (0x2fc0 bytes) 000295 1787366370.036239 trunc from 0x03c98f to 0x016cf1 (0x25c9d bytes) 000297 1787366370.062757 punch from 0x005f1d to 0x00bc88 (0x5d6c bytes) 000298 1787366370.068624 punch from 0x0098c5 to 0x009eda (0x616 bytes) 000299 1787366370.074787 read 0x00166e thru 0x00f20c (0xdb9f bytes) 000300 1787366370.087129 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x16cf2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16cf2): Mirror 2: 0x35f9c6d3 (stale) Mirror 3: 0xf4a4828c (stale) Mirror 4: 0xce16fa1d (stale) CRC-32 checksum value for chunk [0, 0x16cf2): Mirror 1: 0xb18b21f3 Mirror 5: 0xb18b21f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16cf2: skipped 000301 1787366371.068206 trunc from 0x016508 to 0x016cf1 (0x7ea bytes) 000302 1787366371.135491 trunc from 0x0100a1 to 0x016cf1 (0x6c51 bytes) 000303 1787366371.149136 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16cf2): Mirror 2: 0x35f9c6d3 (stale) Mirror 3: 0xf4a4828c (stale) Mirror 5: 0xb18b21f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16cf2): Mirror 2: 0x35f9c6d3 (stale) Mirror 3: 0xf4a4828c (stale) Mirror 5: 0xb18b21f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16cf2: skipped 000304 1787366371.687607 punch from 0x009962 to 0x016cf1 (0xd390 bytes) 000305 1787366371.695213 mapwrite 0x01d9ed thru 0x02cb48 (0xf15c bytes) 000306 1787366371.874196 fallocate 0x032f47 thru 0x03cd33 (0x9ded bytes) 000307 1787366371.880662 read 0x026b53 thru 0x02d921 (0x6dcf bytes) 000308 1787366371.892878 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3cd34): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd34): Mirror 2: 0xccb075cd (stale) Mirror 3: 0x69a5756 (stale) Mirror 5: 0x6f7bd798 (stale) CRC-32 checksum value for chunk [0, 0x3cd34): Mirror 1: 0xc6162155 Mirror 6: 0xc6162155 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd34: skipped 000309 1787366373.534426 trunc from 0x03cd34 to 0x00dd47 (0x2efec bytes) 000310 1787366373.671678 trunc from 0x0004bd to 0x004e16 (0x495a bytes) 000311 1787366373.680026 mapread 0x008aff thru 0x00dd47 (0x5249 bytes) 000312 1787366373.737724 mapread 0x0003bb thru 0x00abba (0xa800 bytes) 000313 1787366373.766452 read 0x002f3d thru 0x00a955 (0x7a19 bytes) 000314 1787366373.769220 mapwrite 0x0210bc thru 0x021817 (0x75c bytes) 000315 1787366373.874552 trunc from 0x021818 to 0x0111ad (0x1066a bytes) 000316 1787366373.939522 mapread 0x00d2a1 thru 0x0111ad (0x3f0d bytes) 000317 1787366373.959681 mapread 0x00beb6 thru 0x0111ad (0x52f8 bytes) 000318 1787366373.965658 trunc from 0x007c2c to 0x0111ad (0x9582 bytes) 000319 1787366373.982499 trunc from 0x00782a to 0x007d36 (0x50d bytes) 000321 1787366373.997511 write 0x03c72b thru 0x03ffff (0x38d5 bytes) 000323 1787366374.011114 trunc from 0x00a311 to 0x018f8c (0xec7c bytes) 000324 1787366374.028550 mapread 0x010d09 thru 0x01d991 (0xcc89 bytes) 000325 1787366374.090104 trunc from 0x004167 to 0x00d768 (0x9602 bytes) 000326 1787366374.098631 write 0x02b28c thru 0x0327e5 (0x755a bytes) 000327 1787366374.121716 read 0x0381e7 thru 0x03ffff (0x7e19 bytes) 000328 1787366374.151521 trunc from 0x0320cb to 0x03ffff (0xdf35 bytes) 000329 1787366374.184481 mapread 0x028846 thru 0x02ec12 (0x63cd bytes) 000330 1787366374.189284 fallocate 0x02aa94 thru 0x039267 (0xe7d4 bytes) 000331 1787366374.225841 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb50fe4c1 (stale) Mirror 5: 0x1cc53a97 (stale) Mirror 6: 0xd90b726 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb50fe4c1 (stale) Mirror 5: 0x1cc53a97 (stale) Mirror 6: 0xd90b726 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000332 1787366374.707048 punch from 0x00045e to 0x006307 (0x5eaa bytes) 000333 1787366374.712250 trunc from 0x01b44b to 0x01de0a (0x29c0 bytes) 000334 1787366374.717277 fallocate 0x00f6e1 thru 0x01d795 (0xe0b5 bytes) 000335 1787366374.723625 fallocate 0x036977 thru 0x03ffff (0x9689 bytes) 000336 1787366374.735720 fallocate 0x02fe8a thru 0x03f736 (0xf8ad bytes) 000337 1787366374.742620 mapread 0x00fe4d thru 0x01de02 (0xdfb6 bytes) 000338 1787366374.772300 punch from 0x01afd7 to 0x01e688 (0x36b2 bytes) 000339 1787366374.777129 trunc from 0x040000 to 0x02d7d9 (0x12826 bytes) 000340 1787366374.799321 write 0x00c54b thru 0x01b792 (0xf248 bytes) 000341 1787366374.806649 trunc from 0x02d7da to 0x002963 (0x2ae76 bytes) 000342 1787366374.829107 mapwrite 0x027ee8 thru 0x02f54c (0x7665 bytes) 000343 1787366374.881754 trunc from 0x023980 to 0x02f54c (0xbbcd bytes) 000344 1787366374.888148 fallocate 0x001926 thru 0x00501f (0x36fa bytes) 000345 1787366374.894009 write 0x01b2b8 thru 0x01ec09 (0x3952 bytes) 000346 1787366374.905463 read 0x02e15a thru 0x02f54c (0x13f3 bytes) 000347 1787366374.917550 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2f54d): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f54d): Mirror 3: 0xf9be0e65 (stale) Mirror 5: 0x1a081982 (stale) Mirror 6: 0x272436e5 (stale) CRC-32 checksum value for chunk [0, 0x2f54d): Mirror 1: 0x4938c787 Mirror 7: 0x4938c787 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f54d: skipped 000349 1787366376.399561 mapwrite 0x02518d thru 0x02daf4 (0x8968 bytes) 000350 1787366376.528312 mapwrite 0x01fdf8 thru 0x020ccb (0xed4 bytes) 000351 1787366376.575670 MIRROR_RESYNC 5 mirrors 000353 1787366378.720500 write 0x01a791 thru 0x01acc7 (0x537 bytes) 000354 1787366378.771424 read 0x016368 thru 0x01caeb (0x6784 bytes) 000356 1787366378.789530 mapread 0x01f3ca thru 0x026fa4 (0x7bdb bytes) 000357 1787366378.808757 write 0x027d96 thru 0x032b57 (0xadc2 bytes) 000358 1787366378.810461 trunc from 0x032b58 to 0x03fab1 (0xcf5a bytes) 000359 1787366378.835158 read 0x031ec7 thru 0x032e5c (0xf96 bytes) 000360 1787366378.841642 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3fab2): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fab2): Mirror 3: 0x6c8ef40b (stale) Mirror 5: 0x6c8ef40b (stale) Mirror 6: 0x6c8ef40b (stale) Mirror 7: 0x6c8ef40b (stale) CRC-32 checksum value for chunk [0, 0x3fab2): Mirror 1: 0xedd131e4 Mirror 8: 0xedd131e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fab2: skipped 000362 1787366380.092892 mapwrite 0x01b8ba thru 0x027bf1 (0xc338 bytes) 000363 1787366380.285481 fallocate 0x0021de thru 0x002369 (0x18c bytes) 000364 1787366380.296946 mapwrite 0x02c04a thru 0x0310f3 (0x50aa bytes) 000365 1787366380.335327 punch from 0x02a8da to 0x039841 (0xef68 bytes) 000366 1787366380.344629 punch from 0x0068ad to 0x014bdf (0xe333 bytes) 000367 1787366380.359890 read 0x01d9f8 thru 0x027cb0 (0xa2b9 bytes) 000368 1787366380.372092 trunc from 0x026318 to 0x031c60 (0xb949 bytes) 000369 1787366380.380003 read 0x00c8c9 thru 0x00d4ac (0xbe4 bytes) 000370 1787366380.395761 trunc from 0x03fab2 to 0x03a454 (0x565d bytes) 000371 1787366380.445291 mapread 0x0156e5 thru 0x01898e (0x32aa bytes) 000372 1787366380.455038 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a455): Mirror 3: 0x5392017c (stale) Mirror 5: 0x5392017c (stale) Mirror 7: 0x5392017c (stale) Mirror 8: 0xd6d9e40d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a455): Mirror 3: 0x5392017c (stale) Mirror 5: 0x5392017c (stale) Mirror 7: 0x5392017c (stale) Mirror 8: 0xd6d9e40d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a455: skipped 000373 1787366381.210603 trunc from 0x02b64e to 0x03a1bb (0xeb6e bytes) 000374 1787366381.225796 mapwrite 0x039d83 thru 0x03ffff (0x627d bytes) 000376 1787366381.331283 read 0x007412 thru 0x00dc4b (0x683a bytes) 000379 1787366381.345624 trunc from 0x040000 to 0x02463a (0x1b9c5 bytes) 000380 1787366381.363254 trunc from 0x016215 to 0x01e8ee (0x86da bytes) 000381 1787366381.369889 mapread 0x024349 thru 0x02463a (0x2f2 bytes) 000382 1787366381.380590 mapread 0x00751d thru 0x00c517 (0x4ffb bytes) 000383 1787366381.382670 trunc from 0x02463b to 0x01a833 (0x9e07 bytes) 000384 1787366381.405588 trunc from 0x01a834 to 0x015bfd (0x4c36 bytes) 000385 1787366381.423623 punch from 0x00dd10 to 0x01256c (0x485d bytes) 000386 1787366381.448509 read 0x005337 thru 0x008a29 (0x36f3 bytes) 000387 1787366381.465535 trunc from 0x013f85 to 0x014e87 (0xf03 bytes) 000388 1787366381.472022 read 0x014ff8 thru 0x015bfd (0xc06 bytes) 000389 1787366381.480464 mapwrite 0x01df52 thru 0x0259fe (0x7aad bytes) 000390 1787366381.599497 fallocate 0x005606 thru 0x007b6f (0x256a bytes) 000391 1787366381.607636 punch from 0x01f195 to 0x023974 (0x47e0 bytes) 000392 1787366381.622945 mapread 0x00317e thru 0x00b450 (0x82d3 bytes) 000393 1787366381.632513 mapwrite 0x007958 thru 0x00dcd4 (0x637d bytes) 000394 1787366381.678682 trunc from 0x0259ff to 0x0258f0 (0x10e bytes) 000395 1787366381.709857 punch from 0x01c8a4 to 0x023e78 (0x75d5 bytes) 000396 1787366381.727798 read 0x007011 thru 0x011b24 (0xab14 bytes) 000398 1787366381.729127 mapwrite 0x01154b thru 0x013df1 (0x28a7 bytes) 000399 1787366381.754290 punch from 0x001a0f to 0x00f99a (0xdf8c bytes) 000400 1787366381.762575 fallocate 0x023790 thru 0x0275ea (0x3e5b bytes) 000401 1787366381.772340 punch from 0x01c720 to 0x01cec2 (0x7a3 bytes) 000402 1787366381.779402 mapwrite 0x018924 thru 0x0193e2 (0xabf bytes) 000403 1787366381.800631 mapwrite 0x03d9c7 thru 0x03ffff (0x2639 bytes) 000404 1787366381.895878 fallocate 0x02f1d3 thru 0x02f278 (0xa6 bytes) 000405 1787366381.905467 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x13f49004 (stale) Mirror 7: 0x13f49004 (stale) Mirror 8: 0x7df31bb2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x13f49004 (stale) Mirror 7: 0x13f49004 (stale) Mirror 8: 0x7df31bb2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000406 1787366382.322901 mapread 0x01a613 thru 0x028c67 (0xe655 bytes) 000407 1787366382.355464 trunc from 0x040000 to 0x014c0e (0x2b3f1 bytes) 000408 1787366382.399090 fallocate 0x01ceb0 thru 0x025205 (0x8356 bytes) 000409 1787366382.409081 fallocate 0x005eee thru 0x00e25a (0x836d bytes) 000410 1787366382.419840 punch from 0x00f526 to 0x01a0da (0xabb5 bytes) 000411 1787366382.428791 fallocate 0x035fd6 thru 0x03fee4 (0x9f0f bytes) 000412 1787366382.436587 fallocate 0x0102ab thru 0x01579c (0x54f2 bytes) 000413 1787366382.443761 read 0x001eba thru 0x009acb (0x7c12 bytes) 000414 1787366382.462644 MIRROR_RESYNC 4 mirrors 000415 1787366383.758281 fallocate 0x030437 thru 0x03164b (0x1215 bytes) 000416 1787366383.842295 read 0x035fbb thru 0x03ba5c (0x5aa2 bytes) 000417 1787366383.884822 read 0x00c8cc thru 0x00ecff (0x2434 bytes) 000418 1787366383.904333 trunc from 0x004484 to 0x006303 (0x1e80 bytes) 000419 1787366383.913644 read 0x017c9a thru 0x021221 (0x9588 bytes) 000420 1787366383.914361 read 0x010f51 thru 0x01c735 (0xb7e5 bytes) 000421 1787366383.924548 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fee5): Mirror 7: 0x192b1eef (stale) Mirror 8: 0x192b1eef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fee5): Mirror 7: 0x192b1eef (stale) Mirror 8: 0x192b1eef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fee5: skipped 000422 1787366384.231896 fallocate 0x03e842 thru 0x03ffff (0x17be bytes) 000423 1787366384.238448 trunc from 0x040000 to 0x010ff4 (0x2f00b bytes) 000424 1787366384.271921 write 0x0172f9 thru 0x023d91 (0xca99 bytes) 000426 1787366384.274318 trunc from 0x023d92 to 0x000f25 (0x22e6c bytes) 000427 1787366384.303707 punch from 0x000868 to 0x000f25 (0x6be bytes) 000428 1787366384.312118 fallocate 0x00b120 thru 0x00c34a (0x122b bytes) 000429 1787366384.318075 mapwrite 0x03d193 thru 0x03ffff (0x2e6d bytes) 000430 1787366384.404174 mapwrite 0x00b940 thru 0x00be18 (0x4d9 bytes) 000432 1787366384.465224 fallocate 0x0058d2 thru 0x006557 (0xc86 bytes) 000433 1787366384.489929 punch from 0x01c924 to 0x01f376 (0x2a53 bytes) 000434 1787366384.500083 punch from 0x01a99f to 0x029c07 (0xf269 bytes) 000435 1787366384.516899 trunc from 0x01a118 to 0x028f44 (0xee2d bytes) 000436 1787366384.534438 mapwrite 0x0144a4 thru 0x018d8a (0x48e7 bytes) 000437 1787366384.613538 trunc from 0x040000 to 0x0249e0 (0x1b61f bytes) 000438 1787366384.665254 trunc from 0x01d461 to 0x01f2e2 (0x1e82 bytes) 000439 1787366384.675126 read 0x0057e3 thru 0x009be0 (0x43fe bytes) 000440 1787366384.675748 punch from 0x0146c8 to 0x019741 (0x507a bytes) 000441 1787366384.693594 punch from 0x00f84e to 0x01abfd (0xb3b0 bytes) 000442 1787366384.715422 trunc from 0x0138c9 to 0x01ed16 (0xb44e bytes) 000444 1787366384.736497 write 0x0052ca thru 0x006ecb (0x1c02 bytes) 000445 1787366384.737568 mapwrite 0x0156a4 thru 0x016504 (0xe61 bytes) 000446 1787366384.816599 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x249e1): Mirror 8: 0x45a59b7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x249e1): Mirror 8: 0x45a59b7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x249e1: skipped 000447 1787366385.258927 trunc from 0x0036a8 to 0x01288c (0xf1e5 bytes) 000448 1787366385.264383 punch from 0x0016e9 to 0x001930 (0x248 bytes) 000449 1787366385.270560 punch from 0x02403c to 0x0249e0 (0x9a5 bytes) 000450 1787366385.274841 punch from 0x01a4e8 to 0x0249e0 (0xa4f9 bytes) 000451 1787366385.279839 write 0x03a400 thru 0x03d3ee (0x2fef bytes) 000452 1787366385.287741 fallocate 0x016ea9 thru 0x019a3b (0x2b93 bytes) 000453 1787366385.295742 mapread 0x02cb73 thru 0x032e27 (0x62b5 bytes) 000455 1787366385.319875 trunc from 0x03d3ef to 0x016966 (0x26a88 bytes) 000456 1787366385.353115 punch from 0x005f0f to 0x01106f (0xb161 bytes) 000457 1787366385.358667 trunc from 0x00c2ea to 0x016966 (0xa67d bytes) 000458 1787366385.364136 trunc from 0x016967 to 0x024b14 (0xe1ae bytes) 000459 1787366385.390326 fallocate 0x016587 thru 0x020e2c (0xa8a6 bytes) 000460 1787366385.402706 read 0x01ac77 thru 0x01f352 (0x46dc bytes) 000462 1787366385.426307 write 0x025c9a thru 0x029e33 (0x419a bytes) 000463 1787366385.429715 trunc from 0x023cf4 to 0x029e33 (0x6140 bytes) 000465 1787366385.472696 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000466 1787366385.725307 trunc from 0x0018cb to 0x011158 (0xf88e bytes) 000467 1787366385.738822 trunc from 0x01142b to 0x0147cf (0x33a5 bytes) 000468 1787366385.742677 punch from 0x022810 to 0x029e33 (0x7624 bytes) 000470 1787366385.749374 read 0x024684 thru 0x02958c (0x4f09 bytes) 000471 1787366385.768246 fallocate 0x00f9a7 thru 0x019555 (0x9baf bytes) 000472 1787366385.781093 mapwrite 0x027bbb thru 0x02a64c (0x2a92 bytes) 000473 1787366385.859809 trunc from 0x02a64d to 0x023428 (0x7224 bytes) 000475 1787366385.877451 read 0x006674 thru 0x011a94 (0xb421 bytes) 000476 1787366385.892001 mapread 0x0163a9 thru 0x01b7e9 (0x5441 bytes) 000477 1787366385.904329 MIRROR_EXTEND to 2 mirrors 000478 1787366386.910810 read 0x002d53 thru 0x004ad0 (0x1d7e bytes) 000479 1787366386.929070 trunc from 0x023429 to 0x0048f1 (0x1eb37 bytes) 000480 1787366387.031829 mapwrite 0x01f5da thru 0x02608f (0x6ab6 bytes) 000481 1787366387.144723 mapwrite 0x007d6c thru 0x00c45f (0x46f4 bytes) 000482 1787366387.209845 fallocate 0x02f9f5 thru 0x036dc3 (0x73cf bytes) 000483 1787366387.223485 read 0x009cc0 thru 0x01714c (0xd48d bytes) 000484 1787366387.254791 fallocate 0x023da0 thru 0x02fb0b (0xbd6c bytes) 000485 1787366387.269882 trunc from 0x026090 to 0x0123b4 (0x13cdb bytes) 000486 1787366387.298871 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000487 1787366387.470596 write 0x034cf2 thru 0x038b36 (0x3e45 bytes) 000488 1787366387.476942 mapwrite 0x02d409 thru 0x031bbb (0x47b3 bytes) 000489 1787366387.510231 mapwrite 0x014927 thru 0x0242ff (0xf9d9 bytes) 000490 1787366387.553536 mapread 0x00eecd thru 0x014cff (0x5e33 bytes) 000492 1787366387.556896 mapwrite 0x02bdde thru 0x02c630 (0x853 bytes) 000493 1787366387.569036 trunc from 0x00f821 to 0x01ca2e (0xd20e bytes) 000494 1787366387.575525 trunc from 0x00003b to 0x00d75f (0xd725 bytes) 000495 1787366387.582669 punch from 0x0170ef to 0x024a35 (0xd947 bytes) 000496 1787366387.589368 trunc from 0x038b37 to 0x02e750 (0xa3e6 bytes) 000497 1787366387.610873 mapread 0x00bdf0 thru 0x011745 (0x5956 bytes) 000498 1787366387.622499 write 0x0165f7 thru 0x016fa5 (0x9af bytes) 000499 1787366387.627988 mapwrite 0x0046c1 thru 0x00f83d (0xb17d bytes) 000500 1787366387.652814 mapwrite 0x01e32a thru 0x020ebd (0x2b94 bytes) 000501 1787366387.678534 trunc from 0x02e751 to 0x010636 (0x1e11a bytes) 000502 1787366387.696534 trunc from 0x010637 to 0x023e0a (0x137d4 bytes) 000503 1787366387.717925 trunc from 0x023e0b to 0x01ee39 (0x4fd1 bytes) 000505 1787366387.760425 punch from 0x00c60d to 0x019214 (0xcc08 bytes) 000507 1787366387.778544 read 0x019264 thru 0x01a613 (0x13b0 bytes) 000508 1787366387.795299 write 0x002127 thru 0x004b4c (0x2a26 bytes) 000509 1787366387.804009 mapwrite 0x026443 thru 0x027431 (0xfef bytes) 000510 1787366387.879856 fallocate 0x01542b thru 0x016bf1 (0x17c7 bytes) 000511 1787366387.892439 read 0x0163f3 thru 0x02542f (0xf03d bytes) 000512 1787366387.936333 write 0x02e96e thru 0x03dc5b (0xf2ee bytes) 000513 1787366387.940676 write 0x025402 thru 0x028236 (0x2e35 bytes) 000515 1787366387.973560 mapread 0x014513 thru 0x01866b (0x4159 bytes) 000516 1787366387.979505 mapwrite 0x033124 thru 0x03d206 (0xa0e3 bytes) 000517 1787366388.029062 punch from 0x006c65 to 0x012bce (0xbf6a bytes) 000518 1787366388.045669 trunc from 0x03dc5c to 0x01bcb2 (0x21fa9 bytes) 000520 1787366388.080434 write 0x029045 thru 0x02e3ed (0x53a9 bytes) 000521 1787366388.084052 read 0x000795 thru 0x00901e (0x888a bytes) 000522 1787366388.099442 mapwrite 0x0021ed thru 0x00de77 (0xbc8b bytes) 000523 1787366388.130184 trunc from 0x02ad89 to 0x02dc91 (0x2f09 bytes) 000524 1787366388.145841 mapwrite 0x01e844 thru 0x024eef (0x66ac bytes) 000525 1787366388.176742 mapwrite 0x01741d thru 0x0253c8 (0xdfac bytes) 000526 1787366388.207449 write 0x003679 thru 0x003f18 (0x8a0 bytes) 000527 1787366388.210002 trunc from 0x02e3ee to 0x03c123 (0xdd36 bytes) 000528 1787366388.228749 write 0x004bda thru 0x00e2c0 (0x96e7 bytes) 000529 1787366388.232782 read 0x01ed08 thru 0x0299f8 (0xacf1 bytes) 000530 1787366388.234435 read 0x01a213 thru 0x0276a0 (0xd48e bytes) 000531 1787366388.237892 write 0x015cac thru 0x01ac7b (0x4fd0 bytes) 000533 1787366388.245044 write 0x02ffda thru 0x03ccfe (0xcd25 bytes) 000534 1787366388.286686 fallocate 0x008056 thru 0x017095 (0xf040 bytes) 000535 1787366388.308756 mapread 0x01b834 thru 0x02b7bb (0xff88 bytes) 000537 1787366388.310838 mapread 0x010f0d thru 0x01f626 (0xe71a bytes) 000538 1787366388.316725 mapwrite 0x039729 thru 0x03c168 (0x2a40 bytes) 000539 1787366388.349029 fallocate 0x00c822 thru 0x01ac36 (0xe415 bytes) 000540 1787366388.354532 fallocate 0x012d59 thru 0x01b94a (0x8bf2 bytes) 000542 1787366388.361730 punch from 0x01e39c to 0x02b3c0 (0xd025 bytes) 000543 1787366388.370419 trunc from 0x03ccff to 0x01bf7d (0x20d81 bytes) 000545 1787366388.399007 fallocate 0x01da45 thru 0x024b01 (0x70bd bytes) 000547 1787366388.408028 punch from 0x008148 to 0x00a8c4 (0x277d bytes) 000550 1787366388.417656 fallocate 0x03485e thru 0x03ffff (0xb7a2 bytes) 000551 1787366388.424423 punch from 0x0032d1 to 0x004e87 (0x1bb7 bytes) 000552 1787366388.431297 mapread 0x00e72a thru 0x01aa24 (0xc2fb bytes) 000554 1787366388.435818 punch from 0x016685 to 0x01b4ff (0x4e7b bytes) 000555 1787366388.442682 read 0x00d20c thru 0x01bf7d (0xed72 bytes) 000556 1787366388.463333 MIRROR_EXTEND to 2 mirrors 000557 1787366389.552367 mapread 0x01252c thru 0x01bf7d (0x9a52 bytes) 000558 1787366389.589515 read 0x009575 thru 0x00b0c5 (0x1b51 bytes) 000559 1787366389.602469 mapwrite 0x01e948 thru 0x02be43 (0xd4fc bytes) 000560 1787366389.718924 write 0x0251bc thru 0x029ed8 (0x4d1d bytes) 000561 1787366389.720657 mapwrite 0x02176a thru 0x02e458 (0xccef bytes) 000562 1787366389.769485 read 0x024e78 thru 0x029383 (0x450c bytes) 000563 1787366389.771427 write 0x00ecdc thru 0x01927c (0xa5a1 bytes) 000565 1787366389.783595 trunc from 0x02820b to 0x02e458 (0x624e bytes) 000566 1787366389.789454 trunc from 0x02e459 to 0x026be2 (0x7876 bytes) 000567 1787366389.802390 mapwrite 0x0057c9 thru 0x00a5ba (0x4df2 bytes) 000568 1787366389.820992 mapwrite 0x003125 thru 0x00a2e2 (0x71be bytes) 000569 1787366389.838294 read 0x00f424 thru 0x01ed36 (0xf913 bytes) 000570 1787366389.839401 trunc from 0x00e426 to 0x01ae65 (0xca40 bytes) 000571 1787366389.850229 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000573 1787366390.002444 mapwrite 0x01bc56 thru 0x02255d (0x6908 bytes) 000574 1787366390.044361 mapwrite 0x03b634 thru 0x03ffff (0x49cc bytes) 000575 1787366390.095568 trunc from 0x040000 to 0x01939d (0x26c62 bytes) 000576 1787366390.116443 mapread 0x011b04 thru 0x01939d (0x789a bytes) 000577 1787366390.127658 write 0x01018b thru 0x011359 (0x11cf bytes) 000578 1787366390.129112 mapwrite 0x013c35 thru 0x0153b7 (0x1783 bytes) 000579 1787366390.149936 trunc from 0x01939e to 0x01d940 (0x45a3 bytes) 000580 1787366390.166775 write 0x0227a3 thru 0x0239f6 (0x1254 bytes) 000581 1787366390.168860 write 0x018def thru 0x0244c5 (0xb6d7 bytes) 000582 1787366390.173861 fallocate 0x020ed0 thru 0x022ed0 (0x2001 bytes) 000583 1787366390.180278 trunc from 0x0244c6 to 0x012693 (0x11e32 bytes) 000585 1787366390.204947 mapwrite 0x0271b8 thru 0x02e4fe (0x7347 bytes) 000586 1787366390.267938 trunc from 0x021e96 to 0x0274ab (0x5616 bytes) 000587 1787366390.277821 write 0x011cb9 thru 0x0124f2 (0x83a bytes) 000588 1787366390.279164 trunc from 0x0284ec to 0x02d092 (0x4ba7 bytes) 000589 1787366390.287843 trunc from 0x02e4ff to 0x011150 (0x1d3ae bytes) 000590 1787366390.328428 mapwrite 0x03598e thru 0x03f837 (0x9eaa bytes) 000592 1787366390.441081 MIRROR_EXTEND to 2 mirrors 000593 1787366391.551339 trunc from 0x012a47 to 0x01ea78 (0xc032 bytes) 000594 1787366391.628939 punch from 0x01d5e2 to 0x027310 (0x9d2f bytes) 000597 1787366391.639255 trunc from 0x03f838 to 0x0315ae (0xe289 bytes) 000598 1787366391.701606 trunc from 0x024407 to 0x02b66a (0x7264 bytes) 000599 1787366391.727089 write 0x01a312 thru 0x020c68 (0x6957 bytes) 000600 1787366391.774127 read 0x02205b thru 0x02a42a (0x83d0 bytes) 000601 1787366391.790272 read 0x02211f thru 0x02416f (0x2051 bytes) 000602 1787366391.791099 mapread 0x02d55a thru 0x0315ae (0x4055 bytes) 000603 1787366391.802209 mapwrite 0x03019e thru 0x032ebe (0x2d21 bytes) 000604 1787366391.864104 punch from 0x01b362 to 0x026d67 (0xba06 bytes) 000605 1787366391.871777 mapread 0x004cad thru 0x00a206 (0x555a bytes) 000608 1787366391.875132 read 0x023f2f thru 0x025c04 (0x1cd6 bytes) 000609 1787366391.926893 read 0x00041b thru 0x0053d9 (0x4fbf bytes) 000610 1787366391.927894 mapread 0x01bbda thru 0x023d97 (0x81be bytes) 000611 1787366391.935355 fallocate 0x0029cd thru 0x00e37e (0xb9b2 bytes) 000612 1787366391.945917 trunc from 0x0093f6 to 0x017a78 (0xe683 bytes) 000613 1787366391.952598 read 0x027e33 thru 0x02bbbd (0x3d8b bytes) 000614 1787366391.953831 write 0x00e14c thru 0x010e4c (0x2d01 bytes) 000615 1787366391.966411 trunc from 0x011dad to 0x016fcc (0x5220 bytes) 000617 1787366391.987900 trunc from 0x00e735 to 0x0172d4 (0x8ba0 bytes) 000618 1787366392.001755 trunc from 0x017005 to 0x026257 (0xf253 bytes) 000619 1787366392.015841 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32ebf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ebf): Mirror 2: 0x5dfcea8 (stale) CRC-32 checksum value for chunk [0, 0x32ebf): Mirror 1: 0x5c31904c Mirror 3: 0x5c31904c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ebf: skipped 000620 1787366393.522542 mapread 0x01b5df thru 0x02122a (0x5c4c bytes) 000621 1787366393.547281 mapwrite 0x03007b thru 0x036af2 (0x6a78 bytes) 000622 1787366393.662523 trunc from 0x036af3 to 0x03cb3e (0x604c bytes) 000624 1787366393.702899 write 0x038deb thru 0x03ffff (0x7215 bytes) 000626 1787366393.713511 trunc from 0x03f783 to 0x03ffff (0x87d bytes) 000627 1787366393.732210 fallocate 0x002dff thru 0x004d91 (0x1f93 bytes) 000628 1787366393.737383 fallocate 0x022452 thru 0x031f5f (0xfb0e bytes) 000629 1787366393.744708 mapread 0x037786 thru 0x03be62 (0x46dd bytes) 000630 1787366393.756386 punch from 0x009375 to 0x00d86b (0x44f7 bytes) 000631 1787366393.762101 punch from 0x025102 to 0x02de01 (0x8d00 bytes) 000633 1787366393.767412 read 0x028c5d thru 0x029ce5 (0x1089 bytes) 000634 1787366393.773551 trunc from 0x01f542 to 0x021a80 (0x253f bytes) 000635 1787366393.777833 trunc from 0x040000 to 0x02193e (0x1e6c1 bytes) 000636 1787366393.799578 punch from 0x0026a5 to 0x006dc2 (0x471e bytes) 000637 1787366393.807625 trunc from 0x02193f to 0x01ea26 (0x2f18 bytes) 000638 1787366393.848176 trunc from 0x01ea27 to 0x01342b (0xb5fb bytes) 000639 1787366393.881783 trunc from 0x0050cc to 0x0130ff (0xe034 bytes) 000640 1787366393.899285 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1342c): Mirror 3: 0x4422daf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1342c): Mirror 3: 0x4422daf4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1342c: skipped 000641 1787366394.172273 mapwrite 0x0082de thru 0x00b61d (0x3340 bytes) 000642 1787366394.224553 fallocate 0x004958 thru 0x00d15e (0x8807 bytes) 000643 1787366394.234409 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1342c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1342c): Mirror 3: 0x4422daf4 (stale) CRC-32 checksum value for chunk [0, 0x1342c): Mirror 1: 0xdd903a1e Mirror 4: 0xdd903a1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1342c: skipped 000644 1787366395.345790 write 0x03ff1c thru 0x03ffff (0xe4 bytes) 000645 1787366395.398151 write 0x00f751 thru 0x015641 (0x5ef1 bytes) 000646 1787366395.413490 mapwrite 0x02087c thru 0x025946 (0x50cb bytes) 000647 1787366395.504792 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8ca17ac8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8ca17ac8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000648 1787366395.834487 trunc from 0x0014fd to 0x003375 (0x1e79 bytes) 000649 1787366395.847688 trunc from 0x040000 to 0x00554c (0x3aab3 bytes) 000650 1787366395.923357 trunc from 0x002949 to 0x00554c (0x2c04 bytes) 000651 1787366395.928383 write 0x004247 thru 0x011ed0 (0xdc8a bytes) 000652 1787366395.938862 fallocate 0x020ec1 thru 0x030821 (0xf961 bytes) 000653 1787366395.946540 trunc from 0x003999 to 0x005de3 (0x244b bytes) 000654 1787366395.983387 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30822): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30822): Mirror 4: 0x8cc3af3e (stale) CRC-32 checksum value for chunk [0, 0x30822): Mirror 1: 0xd4c67194 Mirror 5: 0xd4c67194 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30822: skipped 000655 1787366397.369173 punch from 0x01e7d3 to 0x024715 (0x5f43 bytes) 000656 1787366397.451865 mapread 0x019115 thru 0x028fc4 (0xfeb0 bytes) 000657 1787366397.514090 trunc from 0x030822 to 0x022854 (0xdfcd bytes) 000658 1787366397.546595 trunc from 0x001b53 to 0x00796c (0x5e1a bytes) 000659 1787366397.552051 fallocate 0x000578 thru 0x0088de (0x8367 bytes) 000660 1787366397.571709 trunc from 0x022855 to 0x00691a (0x1bf3a bytes) 000661 1787366397.622368 punch from 0x004559 to 0x00691a (0x23c2 bytes) 000662 1787366397.630839 trunc from 0x00691b to 0x00487e (0x209c bytes) 000663 1787366397.653025 fallocate 0x02d31f thru 0x03bedb (0xebbd bytes) 000664 1787366397.658416 punch from 0x003e9f to 0x0099d4 (0x5b36 bytes) 000666 1787366397.663633 write 0x03a52a thru 0x03faa0 (0x5577 bytes) 000668 1787366397.669623 write 0x03e131 thru 0x03ffff (0x1ecf bytes) 000669 1787366397.671076 mapwrite 0x00fb6c thru 0x01d0c9 (0xd55e bytes) 000670 1787366397.714361 fallocate 0x03273c thru 0x03387c (0x1141 bytes) 000672 1787366397.721875 fallocate 0x02d94d thru 0x03d445 (0xfaf9 bytes) 000673 1787366397.727149 trunc from 0x040000 to 0x00dea2 (0x3215d bytes) 000674 1787366397.757796 MIRROR_RESYNC 3 mirrors 000675 1787366398.497487 mapwrite 0x00a94e thru 0x011eea (0x759d bytes) 000676 1787366398.633689 trunc from 0x011eeb to 0x038d22 (0x26e38 bytes) 000677 1787366398.664984 punch from 0x032040 to 0x038d22 (0x6ce3 bytes) 000679 1787366398.672327 fallocate 0x010058 thru 0x017724 (0x76cd bytes) 000680 1787366398.678946 read 0x02b768 thru 0x02ec6e (0x3507 bytes) 000681 1787366398.690562 punch from 0x033131 to 0x0388dd (0x57ad bytes) 000683 1787366398.697683 trunc from 0x02ea76 to 0x03677f (0x7d0a bytes) 000684 1787366398.704458 mapread 0x00746c thru 0x00ea47 (0x75dc bytes) 000687 1787366398.707750 punch from 0x011b12 to 0x020548 (0xea37 bytes) 000689 1787366398.716482 fallocate 0x022f38 thru 0x032e42 (0xff0b bytes) 000690 1787366398.722653 trunc from 0x02f9ac to 0x037ed9 (0x852e bytes) 000691 1787366398.738807 MIRROR_RESYNC 3 mirrors 000692 1787366399.829824 MIRROR_RESYNC 3 mirrors 000693 1787366400.300575 punch from 0x01a091 to 0x024aeb (0xaa5b bytes) 000694 1787366400.341749 mapread 0x007eb0 thru 0x015e1d (0xdf6e bytes) 000695 1787366400.401388 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38d23): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d23): Mirror 4: 0x9af172ee (stale) Mirror 5: 0x9af172ee (stale) CRC-32 checksum value for chunk [0, 0x38d23): Mirror 1: 0x9af172ee Mirror 6: 0x9af172ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d23: skipped 000696 1787366401.980414 mapwrite 0x01e71d thru 0x02509e (0x6982 bytes) 000698 1787366402.113195 MIRROR_RESYNC 4 mirrors 000699 1787366403.379063 MIRROR_RESYNC 4 mirrors 000700 1787366403.846617 punch from 0x01b7da to 0x020edd (0x5704 bytes) 000701 1787366403.892237 write 0x03399e thru 0x039d0c (0x636f bytes) 000702 1787366403.919186 MIRROR_RESYNC 4 mirrors 000703 1787366405.334613 mapwrite 0x025d35 thru 0x02d81c (0x7ae8 bytes) 000704 1787366405.494184 read 0x01cd6b thru 0x0227e9 (0x5a7f bytes) 000705 1787366405.512301 mapwrite 0x010623 thru 0x017e12 (0x77f0 bytes) 000706 1787366405.587049 read 0x01af51 thru 0x02ace5 (0xfd95 bytes) 000707 1787366405.617434 punch from 0x03196d to 0x039893 (0x7f27 bytes) 000708 1787366405.641928 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39d0d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d0d): Mirror 4: 0x152db807 (stale) Mirror 5: 0x152db807 (stale) Mirror 6: 0x152db807 (stale) CRC-32 checksum value for chunk [0, 0x39d0d): Mirror 1: 0x63102368 Mirror 7: 0x63102368 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d0d: skipped 000709 1787366406.940178 mapwrite 0x02d5ed thru 0x03325f (0x5c73 bytes) 000710 1787366407.085810 mapwrite 0x00858d thru 0x015c07 (0xd67b bytes) 000711 1787366407.112919 mapread 0x012252 thru 0x01374e (0x14fd bytes) 000712 1787366407.114136 write 0x01234b thru 0x014ff1 (0x2ca7 bytes) 000713 1787366407.119922 punch from 0x01a0f3 to 0x01d8ae (0x37bc bytes) 000714 1787366407.130634 trunc from 0x039d0d to 0x036c9c (0x3070 bytes) 000715 1787366407.157687 mapread 0x01d78c thru 0x02d169 (0xf9de bytes) 000717 1787366407.184315 trunc from 0x036c9d to 0x026506 (0x10796 bytes) 000718 1787366407.200162 trunc from 0x009ced to 0x00a413 (0x727 bytes) 000719 1787366407.210499 trunc from 0x00331c to 0x01020f (0xcef4 bytes) 000720 1787366407.221120 trunc from 0x009585 to 0x00f5c1 (0x603d bytes) 000721 1787366407.233444 trunc from 0x00573a to 0x014cea (0xf5b1 bytes) 000722 1787366407.252107 write 0x02fa53 thru 0x035b11 (0x60bf bytes) 000723 1787366407.257144 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35b12): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b12): Mirror 4: 0x50dace08 (stale) Mirror 5: 0x50dace08 (stale) Mirror 6: 0x50dace08 (stale) Mirror 7: 0x354bc225 (stale) CRC-32 checksum value for chunk [0, 0x35b12): Mirror 1: 0xb9332b1c Mirror 8: 0xb9332b1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b12: skipped 000725 1787366408.498326 write 0x00b88e thru 0x0119bf (0x6132 bytes) 000726 1787366408.543845 punch from 0x003c0b to 0x005141 (0x1537 bytes) 000727 1787366408.548238 trunc from 0x026016 to 0x027fd3 (0x1fbe bytes) 000728 1787366408.555511 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x35b12): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b12): Mirror 4: 0x50dace08 (stale) Mirror 5: 0x50dace08 (stale) Mirror 6: 0x50dace08 (stale) Mirror 7: 0x354bc225 (stale) Mirror 8: 0xb9332b1c (stale) CRC-32 checksum value for chunk [0, 0x35b12): Mirror 1: 0xd8a8a85b Mirror 9: 0xd8a8a85b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b12: skipped 000729 1787366410.333836 MIRROR_RESYNC 7 mirrors 000732 1787366413.086894 mapread 0x005335 thru 0x013b9b (0xe867 bytes) 000733 1787366413.169912 trunc from 0x035b12 to 0x00ea59 (0x270b8 bytes) 000734 1787366413.279507 mapread 0x005813 thru 0x007445 (0x1c33 bytes) 000735 1787366413.291369 trunc from 0x00ea5a to 0x01ad24 (0xc2cb bytes) 000736 1787366413.329561 trunc from 0x0032ec to 0x00d650 (0xa365 bytes) 000737 1787366413.342997 mapwrite 0x002d17 thru 0x00c559 (0x9843 bytes) 000739 1787366413.447018 mapwrite 0x038570 thru 0x03fe27 (0x78b8 bytes) 000740 1787366413.567712 punch from 0x01aa8e to 0x01ab7d (0xf0 bytes) 000742 1787366413.580095 punch from 0x031563 to 0x03c902 (0xb3a0 bytes) 000743 1787366413.597994 trunc from 0x02f0a3 to 0x03e7d2 (0xf730 bytes) 000744 1787366413.611430 write 0x017df5 thru 0x020fa4 (0x91b0 bytes) 000746 1787366413.636944 MIRROR_RESYNC 7 mirrors 000747 1787366416.286797 trunc from 0x03fe28 to 0x0384fa (0x792d bytes) 000748 1787366416.360724 fallocate 0x006a12 thru 0x00a349 (0x3938 bytes) 000749 1787366416.372857 punch from 0x0295f2 to 0x0361e2 (0xcbf1 bytes) 000750 1787366416.397381 mapwrite 0x01f892 thru 0x02f301 (0xfa70 bytes) 000751 1787366416.520698 mapread 0x0355f9 thru 0x0384fa (0x2f02 bytes) 000752 1787366416.542429 MIRROR_RESYNC 7 mirrors 000753 1787366418.825095 mapwrite 0x03fad7 thru 0x03ffff (0x529 bytes) 000754 1787366418.911711 write 0x013ee5 thru 0x01d234 (0x9350 bytes) 000755 1787366418.926778 trunc from 0x02351d to 0x030828 (0xd30c bytes) 000757 1787366418.935306 fallocate 0x039938 thru 0x03a514 (0xbdd bytes) 000758 1787366418.940406 punch from 0x032105 to 0x039a8e (0x798a bytes) 000759 1787366418.945593 mapwrite 0x03139d thru 0x03ffff (0xec63 bytes) 000760 1787366419.018830 mapwrite 0x0001ec thru 0x00938e (0x91a3 bytes) 000761 1787366419.080999 trunc from 0x030310 to 0x03277e (0x246f bytes) 000762 1787366419.090281 read 0x02b0f3 thru 0x0308a9 (0x57b7 bytes) 000763 1787366419.113348 MIRROR_RESYNC 7 mirrors 000764 1787366421.200995 trunc from 0x040000 to 0x03fd81 (0x27e bytes) 000765 1787366421.282815 write 0x0104e5 thru 0x01c40f (0xbf2b bytes) 000766 1787366421.299815 MIRROR_RESYNC 7 mirrors 000767 1787366423.427422 mapwrite 0x03b98f thru 0x03c56d (0xbdf bytes) 000768 1787366423.558089 mapread 0x03ecf2 thru 0x03fd81 (0x1090 bytes) 000769 1787366423.559075 trunc from 0x019128 to 0x021846 (0x871f bytes) 000770 1787366423.585843 trunc from 0x03fd82 to 0x03fab0 (0x2d1 bytes) 000771 1787366423.629314 write 0x021156 thru 0x021a05 (0x8b0 bytes) 000772 1787366423.640274 write 0x021413 thru 0x025ef3 (0x4ae1 bytes) 000773 1787366423.649766 punch from 0x00b6a8 to 0x00e265 (0x2bbe bytes) 000774 1787366423.656835 trunc from 0x00dceb to 0x01928a (0xb5a0 bytes) 000775 1787366423.670644 mapread 0x01e1cb thru 0x0254e6 (0x731c bytes) 000776 1787366423.698899 punch from 0x01e845 to 0x0276ba (0x8e76 bytes) 000777 1787366423.736585 mapread 0x003a1c thru 0x00b30b (0x78f0 bytes) 000778 1787366423.751846 read 0x00e694 thru 0x012607 (0x3f74 bytes) 000779 1787366423.757354 read 0x003292 thru 0x0055f6 (0x2365 bytes) 000780 1787366423.758440 trunc from 0x030d1d to 0x0368a7 (0x5b8b bytes) 000781 1787366423.772440 punch from 0x0077d5 to 0x00be7b (0x46a7 bytes) 000782 1787366423.779722 fallocate 0x01a625 thru 0x0222dd (0x7cb9 bytes) 000783 1787366423.786143 trunc from 0x000188 to 0x0025cb (0x2444 bytes) 000784 1787366423.796154 write 0x03e7c0 thru 0x03ffff (0x1840 bytes) 000785 1787366423.811455 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x558b8916 (stale) Mirror 5: 0x558b8916 (stale) Mirror 6: 0x558b8916 (stale) Mirror 7: 0x558b8916 (stale) Mirror 8: 0x558b8916 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x558b8916 (stale) Mirror 5: 0x558b8916 (stale) Mirror 6: 0x558b8916 (stale) Mirror 7: 0x558b8916 (stale) Mirror 8: 0x558b8916 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000786 1787366424.399719 read 0x0324ac thru 0x039904 (0x7459 bytes) 000787 1787366424.411494 trunc from 0x01d367 to 0x0270a3 (0x9d3d bytes) 000788 1787366424.416708 mapread 0x0044ff thru 0x013e5a (0xf95c bytes) 000789 1787366424.463048 punch from 0x03ec35 to 0x03ffff (0x13cb bytes) 000790 1787366424.470533 trunc from 0x038679 to 0x03da7c (0x5404 bytes) 000791 1787366424.477347 write 0x004aad thru 0x009f61 (0x54b5 bytes) 000792 1787366424.484454 trunc from 0x00a864 to 0x00b027 (0x7c4 bytes) 000793 1787366424.489822 mapwrite 0x000e4d thru 0x004514 (0x36c8 bytes) 000794 1787366424.540798 write 0x018779 thru 0x022c5f (0xa4e7 bytes) 000795 1787366424.547798 mapread 0x0349bc thru 0x03ffff (0xb644 bytes) 000796 1787366424.563598 trunc from 0x040000 to 0x03b151 (0x4eae bytes) 000797 1787366424.594697 MIRROR_RESYNC 6 mirrors 000798 1787366426.654935 read 0x0130db thru 0x0164d3 (0x33f9 bytes) 000799 1787366426.663658 punch from 0x0354d0 to 0x0372ea (0x1e1b bytes) 000801 1787366426.716166 read 0x025a19 thru 0x028ed6 (0x34be bytes) 000802 1787366426.729935 punch from 0x03014a to 0x03b151 (0xb008 bytes) 000803 1787366426.749766 mapread 0x0130d6 thru 0x017c6f (0x4b9a bytes) 000804 1787366426.752492 mapread 0x01e1cb thru 0x0225d7 (0x440d bytes) 000805 1787366426.753391 mapwrite 0x0179a3 thru 0x01cbd1 (0x522f bytes) 000806 1787366426.789381 trunc from 0x03b152 to 0x03b0d9 (0x78 bytes) 000807 1787366426.820709 read 0x02ce5c thru 0x03b0d9 (0xe27e bytes) 000808 1787366426.832169 read 0x02a5af thru 0x02d956 (0x33a8 bytes) 000809 1787366426.833545 trunc from 0x024498 to 0x02d9c0 (0x9529 bytes) 000810 1787366426.839938 mapwrite 0x006f8e thru 0x00db51 (0x6bc4 bytes) 000811 1787366426.865280 write 0x0051de thru 0x011204 (0xc027 bytes) 000812 1787366426.869189 read 0x002480 thru 0x003c30 (0x17b1 bytes) 000814 1787366426.871505 trunc from 0x02b86a to 0x0350ed (0x9884 bytes) 000815 1787366426.878455 punch from 0x03549c to 0x03b0d9 (0x5c3e bytes) 000816 1787366426.888860 MIRROR_RESYNC 6 mirrors 000817 1787366428.306748 write 0x01ff5c thru 0x023fe5 (0x408a bytes) 000818 1787366428.386226 fallocate 0x005555 thru 0x013959 (0xe405 bytes) 000819 1787366428.395666 trunc from 0x03b0da to 0x03a3fc (0xcdd bytes) 000820 1787366428.463696 write 0x0258f5 thru 0x030c51 (0xb35d bytes) 000821 1787366428.511429 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a3fd): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3fd): Mirror 4: 0xa866e513 (stale) Mirror 5: 0xa866e513 (stale) Mirror 6: 0xa866e513 (stale) Mirror 7: 0xa866e513 (stale) Mirror 8: 0xa866e513 (stale) CRC-32 checksum value for chunk [0, 0x3a3fd): Mirror 1: 0xc9479e95 Mirror 9: 0xc9479e95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3fd: skipped 000822 1787366429.607416 mapread 0x02c6e9 thru 0x02cb8c (0x4a4 bytes) 000823 1787366429.617938 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a3fd): Mirror 4: 0xa866e513 (stale) Mirror 5: 0xa866e513 (stale) Mirror 6: 0xa866e513 (stale) Mirror 7: 0xa866e513 (stale) Mirror 8: 0xa866e513 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3fd): Mirror 4: 0xa866e513 (stale) Mirror 5: 0xa866e513 (stale) Mirror 6: 0xa866e513 (stale) Mirror 7: 0xa866e513 (stale) Mirror 8: 0xa866e513 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3fd: skipped 000824 1787366430.117488 write 0x022231 thru 0x02694d (0x471d bytes) 000825 1787366430.168780 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a3fd): Mirror 4: 0xa866e513 (stale) Mirror 6: 0xa866e513 (stale) Mirror 7: 0xa866e513 (stale) Mirror 8: 0xa866e513 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3fd): Mirror 4: 0xa866e513 (stale) Mirror 6: 0xa866e513 (stale) Mirror 7: 0xa866e513 (stale) Mirror 8: 0xa866e513 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3fd: skipped 000826 1787366430.579076 mapwrite 0x01a381 thru 0x023381 (0x9001 bytes) 000827 1787366430.633632 punch from 0x000b4d to 0x006fcc (0x6480 bytes) 000828 1787366430.641754 punch from 0x001776 to 0x011397 (0xfc22 bytes) 000829 1787366430.657822 trunc from 0x02314c to 0x02eef5 (0xbdaa bytes) 000830 1787366430.669609 write 0x03f0bc thru 0x03ffff (0xf44 bytes) 000831 1787366430.681019 read 0x02ddc1 thru 0x037c2f (0x9e6f bytes) 000832 1787366430.705689 fallocate 0x003d6d thru 0x00befe (0x8192 bytes) 000833 1787366430.712590 trunc from 0x040000 to 0x020d55 (0x1f2aa bytes) 000834 1787366430.736682 punch from 0x0163b9 to 0x020d55 (0xa99d bytes) 000836 1787366430.743421 trunc from 0x020d56 to 0x0050e8 (0x1bc6d bytes) 000837 1787366430.764644 read 0x00199d thru 0x0050e8 (0x374c bytes) 000838 1787366430.770682 trunc from 0x0050e9 to 0x031554 (0x2c46c bytes) 000839 1787366430.786333 write 0x02d02d thru 0x033259 (0x622d bytes) 000840 1787366430.805438 punch from 0x01a1f4 to 0x020641 (0x644e bytes) 000841 1787366430.811466 trunc from 0x03325a to 0x0102de (0x22f7b bytes) 000842 1787366430.831288 mapwrite 0x019204 thru 0x01f651 (0x644e bytes) 000843 1787366430.912552 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f652): Mirror 4: 0x95a344fa (stale) Mirror 6: 0x95a344fa (stale) Mirror 7: 0x95a344fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f652): Mirror 4: 0x95a344fa (stale) Mirror 6: 0x95a344fa (stale) Mirror 7: 0x95a344fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f652: skipped 000844 1787366431.350309 mapwrite 0x022826 thru 0x02d48e (0xac69 bytes) 000845 1787366431.454795 read 0x002971 thru 0x00c758 (0x9de8 bytes) 000846 1787366431.467321 fallocate 0x02e86d thru 0x037869 (0x8ffd bytes) 000847 1787366431.475494 mapread 0x014b42 thru 0x01cf24 (0x83e3 bytes) 000848 1787366431.491298 punch from 0x01e66b to 0x029d50 (0xb6e6 bytes) 000849 1787366431.505085 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3786a): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3786a): Mirror 4: 0xa999c623 (stale) Mirror 6: 0xa999c623 (stale) Mirror 7: 0xa999c623 (stale) CRC-32 checksum value for chunk [0, 0x3786a): Mirror 1: 0xea264ff4 Mirror 8: 0xea264ff4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3786a: skipped 000850 1787366432.809721 mapread 0x02e1c7 thru 0x03569d (0x74d7 bytes) 000851 1787366432.830861 trunc from 0x030638 to 0x0364df (0x5ea8 bytes) 000852 1787366432.877154 read 0x00352a thru 0x00e7c6 (0xb29d bytes) 000853 1787366432.890356 write 0x03c718 thru 0x03c98f (0x278 bytes) 000854 1787366432.896917 mapwrite 0x00df69 thru 0x01b427 (0xd4bf bytes) 000855 1787366432.964434 trunc from 0x03c990 to 0x01c13a (0x20855 bytes) 000856 1787366432.988588 trunc from 0x009127 to 0x018489 (0xf363 bytes) 000857 1787366432.995611 punch from 0x016f36 to 0x0189a9 (0x1a74 bytes) 000858 1787366433.001147 mapwrite 0x0395f1 thru 0x03ed58 (0x5768 bytes) 000859 1787366433.059823 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ed59): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed59): Mirror 4: 0xee6dd5d7 (stale) Mirror 6: 0xee6dd5d7 (stale) Mirror 7: 0xee6dd5d7 (stale) Mirror 8: 0xf4116dac (stale) CRC-32 checksum value for chunk [0, 0x3ed59): Mirror 1: 0x78c77e6d Mirror 9: 0x78c77e6d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed59: skipped 000860 1787366434.430108 fallocate 0x02e3d2 thru 0x032723 (0x4352 bytes) 000861 1787366434.499524 mapwrite 0x0386af thru 0x03a588 (0x1eda bytes) 000862 1787366434.576119 write 0x01861d thru 0x021afc (0x94e0 bytes) 000863 1787366434.631908 mapwrite 0x029b67 thru 0x03555c (0xb9f6 bytes) 000864 1787366434.693598 trunc from 0x03ed59 to 0x030ed1 (0xde87 bytes) 000865 1787366434.753598 trunc from 0x030ed2 to 0x020d6d (0x10164 bytes) 000866 1787366434.775985 trunc from 0x005099 to 0x0099c5 (0x492d bytes) 000867 1787366434.781661 punch from 0x00c843 to 0x0164a3 (0x9c61 bytes) 000868 1787366434.786435 mapwrite 0x02641d thru 0x029e84 (0x3a68 bytes) 000869 1787366434.838586 trunc from 0x029e85 to 0x026d52 (0x3132 bytes) 000870 1787366434.861590 fallocate 0x039aa9 thru 0x03ddf0 (0x4348 bytes) 000871 1787366434.869864 mapread 0x01a469 thru 0x01cadf (0x2677 bytes) 000872 1787366434.871575 mapwrite 0x0347b2 thru 0x03ebef (0xa43e bytes) 000873 1787366434.919933 write 0x002153 thru 0x00d759 (0xb607 bytes) 000874 1787366434.935508 MIRROR_RESYNC 6 mirrors 000875 1787366437.413570 trunc from 0x023dd7 to 0x02d37d (0x95a7 bytes) 000876 1787366437.463433 write 0x00b5e4 thru 0x01b1c3 (0xfbe0 bytes) 000877 1787366437.479849 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ebf0): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ebf0): Mirror 4: 0x7f0d59ce (stale) Mirror 6: 0x7f0d59ce (stale) Mirror 7: 0x7f0d59ce (stale) Mirror 8: 0x7f0d59ce (stale) Mirror 9: 0x7f0d59ce (stale) CRC-32 checksum value for chunk [0, 0x3ebf0): Mirror 1: 0x399dc163 Mirror 10: 0x399dc163 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ebf0: skipped 000878 1787366438.682918 write 0x022cff thru 0x02a486 (0x7788 bytes) 000879 1787366438.757103 punch from 0x01de43 to 0x02b4d8 (0xd696 bytes) 000880 1787366438.778425 read 0x023dd5 thru 0x02ab00 (0x6d2c bytes) 000881 1787366438.802235 write 0x02313a thru 0x0281f6 (0x50bd bytes) 000884 1787366438.812440 write 0x01e0dd thru 0x0269a5 (0x88c9 bytes) 000885 1787366438.822548 mapwrite 0x0033bf thru 0x00e5ee (0xb230 bytes) 000886 1787366438.892288 write 0x03d461 thru 0x03fe1b (0x29bb bytes) 000887 1787366438.897561 fallocate 0x026794 thru 0x02e4c8 (0x7d35 bytes) 000888 1787366438.922717 MIRROR_RESYNC 7 mirrors 000889 1787366441.535159 mapread 0x02434b thru 0x0309b1 (0xc667 bytes) 000890 1787366441.597897 trunc from 0x03fe1c to 0x00cf73 (0x32ea8 bytes) 000891 1787366441.676110 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xcf74): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcf74): Mirror 4: 0x522c4154 (stale) Mirror 6: 0x522c4154 (stale) Mirror 7: 0x522c4154 (stale) Mirror 8: 0x522c4154 (stale) Mirror 9: 0x522c4154 (stale) Mirror 10: 0x522c4154 (stale) CRC-32 checksum value for chunk [0, 0xcf74): Mirror 1: 0x522c4154 Mirror 11: 0x522c4154 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcf74: skipped 000892 1787366442.972165 trunc from 0x00cf74 to 0x03109c (0x24129 bytes) 000893 1787366443.054285 trunc from 0x00af7e to 0x011821 (0x68a4 bytes) 000895 1787366443.060782 trunc from 0x03109d to 0x035472 (0x43d6 bytes) 000896 1787366443.086236 punch from 0x02aace to 0x02bfcb (0x14fe bytes) 000897 1787366443.098625 MIRROR_RESYNC 8 mirrors 000898 1787366445.320934 fallocate 0x00ae7d thru 0x0124a9 (0x762d bytes) 000899 1787366445.364721 punch from 0x02ea8e to 0x035472 (0x69e5 bytes) 000900 1787366445.370382 fallocate 0x016426 thru 0x01c10f (0x5cea bytes) 000901 1787366445.376552 mapwrite 0x02bd72 thru 0x03bb4e (0xfddd bytes) 000902 1787366445.441787 read 0x01ec02 thru 0x028818 (0x9c17 bytes) 000903 1787366445.450371 mapwrite 0x01ce1a thru 0x02b7a4 (0xe98b bytes) 000904 1787366445.485736 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb4f): Mirror 6: 0xf66a9fc5 (stale) Mirror 7: 0xf66a9fc5 (stale) Mirror 8: 0xf66a9fc5 (stale) Mirror 9: 0xf66a9fc5 (stale) Mirror 10: 0xf66a9fc5 (stale) Mirror 11: 0xf66a9fc5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb4f): Mirror 6: 0xf66a9fc5 (stale) Mirror 7: 0xf66a9fc5 (stale) Mirror 8: 0xf66a9fc5 (stale) Mirror 9: 0xf66a9fc5 (stale) Mirror 10: 0xf66a9fc5 (stale) Mirror 11: 0xf66a9fc5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb4f: skipped 000905 1787366446.131751 MIRROR_RESYNC 7 mirrors 000906 1787366448.008713 punch from 0x00ce29 to 0x0172f0 (0xa4c8 bytes) 000907 1787366448.063776 fallocate 0x020ee9 thru 0x02f151 (0xe269 bytes) 000908 1787366448.080135 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb4f): Mirror 7: 0x729b5f73 (stale) Mirror 8: 0x729b5f73 (stale) Mirror 9: 0x729b5f73 (stale) Mirror 10: 0x729b5f73 (stale) Mirror 11: 0x729b5f73 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb4f): Mirror 7: 0x729b5f73 (stale) Mirror 8: 0x729b5f73 (stale) Mirror 9: 0x729b5f73 (stale) Mirror 10: 0x729b5f73 (stale) Mirror 11: 0x729b5f73 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb4f: skipped 000909 1787366448.592222 trunc from 0x0227c1 to 0x02ecfb (0xc53b bytes) 000912 1787366448.598903 punch from 0x006c12 to 0x00d970 (0x6d5f bytes) 000913 1787366448.605514 fallocate 0x019cd4 thru 0x02865e (0xe98b bytes) 000914 1787366448.619729 trunc from 0x03bb4f to 0x005c12 (0x35f3c bytes) 000915 1787366448.648258 fallocate 0x03d3e6 thru 0x03da64 (0x67f bytes) 000916 1787366448.671307 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3da65): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da65): Mirror 7: 0x3e6d2d65 (stale) Mirror 8: 0x3e6d2d65 (stale) Mirror 9: 0x3e6d2d65 (stale) Mirror 10: 0x3e6d2d65 (stale) Mirror 11: 0x3e6d2d65 (stale) CRC-32 checksum value for chunk [0, 0x3da65): Mirror 1: 0x2b065498 Mirror 12: 0x2b065498 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da65: skipped 000917 1787366449.926980 write 0x03655e thru 0x03d8d2 (0x7375 bytes) 000918 1787366449.983438 trunc from 0x0184b4 to 0x0247df (0xc32c bytes) 000921 1787366450.002401 MIRROR_RESYNC 7 mirrors 000922 1787366451.755520 write 0x036650 thru 0x03ffff (0x99b0 bytes) 000923 1787366451.800381 trunc from 0x0194e2 to 0x02598f (0xc4ae bytes) 000924 1787366451.806185 punch from 0x03102d to 0x032ded (0x1dc1 bytes) 000925 1787366451.812115 mapread 0x0235ec thru 0x02be2e (0x8843 bytes) 000927 1787366451.834575 fallocate 0x016910 thru 0x026390 (0xfa81 bytes) 000929 1787366451.844271 write 0x015a81 thru 0x020daa (0xb32a bytes) 000930 1787366451.854569 punch from 0x01300d to 0x01ec82 (0xbc76 bytes) 000932 1787366451.898005 mapwrite 0x0175f2 thru 0x018fad (0x19bc bytes) 000933 1787366451.924530 mapread 0x028e96 thru 0x0370d4 (0xe23f bytes) 000934 1787366451.936925 mapread 0x005d13 thru 0x00edf4 (0x90e2 bytes) 000936 1787366451.950271 mapwrite 0x029957 thru 0x02ece5 (0x538f bytes) 000937 1787366451.999012 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x7e739962 (stale) Mirror 8: 0x7e739962 (stale) Mirror 9: 0x7e739962 (stale) Mirror 10: 0x7e739962 (stale) Mirror 11: 0x7e739962 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x7e739962 (stale) Mirror 8: 0x7e739962 (stale) Mirror 9: 0x7e739962 (stale) Mirror 10: 0x7e739962 (stale) Mirror 11: 0x7e739962 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000938 1787366452.621404 write 0x024cdf thru 0x02cb11 (0x7e33 bytes) 000939 1787366452.651354 mapwrite 0x006ebb thru 0x009aeb (0x2c31 bytes) 000940 1787366452.732708 punch from 0x017ba1 to 0x01b3f6 (0x3856 bytes) 000943 1787366452.759110 MIRROR_RESYNC 6 mirrors 000944 1787366455.369937 punch from 0x02cdaa to 0x03a03f (0xd296 bytes) 000945 1787366455.407672 mapwrite 0x023ee4 thru 0x02cbe7 (0x8d04 bytes) 000946 1787366455.448389 write 0x03272d thru 0x034af3 (0x23c7 bytes) 000947 1787366455.461003 write 0x01e51b thru 0x025a4b (0x7531 bytes) 000948 1787366455.472020 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xfaa043fb (stale) Mirror 8: 0xfaa043fb (stale) Mirror 9: 0xfaa043fb (stale) Mirror 10: 0xfaa043fb (stale) Mirror 11: 0xfaa043fb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6c7a6ba4 Mirror 12: 0x6c7a6ba4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000949 1787366456.775605 fallocate 0x01051d thru 0x013cac (0x3790 bytes) 000950 1787366456.823445 punch from 0x01b6fe to 0x023a55 (0x8358 bytes) 000951 1787366456.829859 write 0x020d29 thru 0x028adc (0x7db4 bytes) 000952 1787366456.845656 trunc from 0x0017e2 to 0x008aaf (0x72ce bytes) 000953 1787366456.853877 mapwrite 0x027760 thru 0x0289ab (0x124c bytes) 000954 1787366456.910085 write 0x00dfdb thru 0x01a03e (0xc064 bytes) 000955 1787366456.934933 write 0x032d1c thru 0x03aa43 (0x7d28 bytes) 000956 1787366456.957416 trunc from 0x040000 to 0x0161d2 (0x29e2d bytes) 000957 1787366456.990778 mapread 0x003ccd thru 0x0040ca (0x3fe bytes) 000958 1787366457.009514 mapwrite 0x011168 thru 0x01819a (0x7033 bytes) 000960 1787366457.089570 MIRROR_RESYNC 7 mirrors 000961 1787366458.920487 mapread 0x00ef8c thru 0x01819a (0x920f bytes) 000962 1787366458.952784 MIRROR_EXTEND to 8 mirrors 000963 1787366459.812502 mapwrite 0x03471e thru 0x038a95 (0x4378 bytes) 000964 1787366459.939940 trunc from 0x0176e8 to 0x0177d4 (0xed bytes) 000965 1787366459.945845 trunc from 0x0242a4 to 0x0289e5 (0x4742 bytes) 000966 1787366459.954061 punch from 0x00b4cd to 0x00d46a (0x1f9e bytes) 000967 1787366459.959396 fallocate 0x02e31a thru 0x02f96a (0x1651 bytes) 000969 1787366459.969820 write 0x0120d7 thru 0x013b48 (0x1a72 bytes) 000970 1787366459.981447 write 0x002500 thru 0x003c67 (0x1768 bytes) 000971 1787366459.992231 trunc from 0x038a96 to 0x009df9 (0x2ec9c bytes) 000972 1787366460.018705 write 0x026c32 thru 0x02f442 (0x8811 bytes) 000973 1787366460.021172 read 0x00e3cf thru 0x012b05 (0x4737 bytes) 000974 1787366460.043939 write 0x0392bb thru 0x03e7f2 (0x5538 bytes) 000975 1787366460.045590 write 0x01cde4 thru 0x01dc2b (0xe48 bytes) 000976 1787366460.064402 write 0x00e700 thru 0x01bf16 (0xd817 bytes) 000977 1787366460.071658 punch from 0x00e014 to 0x014f1c (0x6f09 bytes) 000978 1787366460.106938 mapread 0x03cd48 thru 0x03e7f2 (0x1aab bytes) 000979 1787366460.109180 punch from 0x0214b4 to 0x027246 (0x5d93 bytes) 000980 1787366460.116099 write 0x00b524 thru 0x0123db (0x6eb8 bytes) 000983 1787366460.159217 trunc from 0x028796 to 0x037c51 (0xf4bc bytes) 000984 1787366460.172566 mapwrite 0x003c4d thru 0x00e68b (0xaa3f bytes) 000985 1787366460.211215 trunc from 0x03e7f3 to 0x01dc43 (0x20baf bytes) 000986 1787366460.232533 mapwrite 0x006113 thru 0x00a7a6 (0x4694 bytes) 000987 1787366460.248453 fallocate 0x0264e2 thru 0x02a193 (0x3cb2 bytes) 000988 1787366460.263510 MIRROR_RESYNC 8 mirrors 000989 1787366462.080129 read 0x01bf44 thru 0x01dc43 (0x1d00 bytes) 000991 1787366462.096400 MIRROR_EXTEND to 9 mirrors 000993 1787366463.008173 read 0x0104d5 thru 0x01a56c (0xa098 bytes) 000995 1787366463.025422 MIRROR_EXTEND to 10 mirrors 000997 1787366463.895542 read 0x00d018 thru 0x00dab8 (0xaa1 bytes) 000998 1787366463.905380 punch from 0x00cc5a to 0x01354d (0x68f4 bytes) 000999 1787366463.970715 mapwrite 0x003013 thru 0x0071e5 (0x41d3 bytes) All operations completed A-OK! PASS 70a (123s) == sanity-flr test 71: check mirror extend parent fid ==== 22:41:08 (1787366468) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x5e:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 22:41:18 (1787366478) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246571 s, 8.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911432 s, 11.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 22:41:27 (1787366487) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8587840 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5527072 .. Extending file size to 8836800 .. Extending file size to 7412800 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8230848 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8332352 .. failed Extending file size to 8642816 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8946656 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8913856 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62639 62640 62643 62644 62645 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) 192.168.204.151@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:42:45 (1787366565) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6169568 .. Extending file size to 4158656 .. done Extending file size to 4872512 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8303552 .. Extending file size to 6707488 .. Extending file size to 7702464 .. Extending file size to 8860064 .. Extending file size to 8169920 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9069952 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9067936 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 64410 64412 64413 64415 64416 failed 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) 192.168.204.151@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:44:02 (1787366642) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2516 1268788 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2024 1269280 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24528 3582492 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18808 3588212 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43336 7170704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2516 1268788 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2024 1269280 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24528 3582492 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18808 3588212 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43336 7170704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0983366 s, 21.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204409 s, 10.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x31d1d7d0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:44:21 (1787366661) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132864 s, 15.8 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 22:44:29 (1787366669) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ab:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e4:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e5:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x2e2:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:44:37 (1787366677) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146749 s, 14.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:44:46 (1787366686) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124996 s, 16.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e4:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:44:56 (1787366696) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139251 s, 15.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:45:31 (1787366731) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg451-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844221 s, 12.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:46:25 (1787366785) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg451-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0998492 s, 21.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:47:16 (1787366836) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg451-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg451-server: Pool lustre.test_204f created Adding targets to pool oleg451-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg451-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160269 s, 13.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg451-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg451-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg451-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:48:31 (1787366911) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:48:39 (1787366919) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:48:47 (1787366927) oleg451-server: Pool lustre.test_206 created Adding targets to pool oleg451-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg451-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg451-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg451-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg451-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:49:16 (1787366956) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100554 s, 10.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305899 s, 34.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfff53253 Mirror 3: 0xfff53253 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:49:29 (1787366969) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:49:33 (1787366973) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:49:37 (1787366977) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0920695 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xea0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x20d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x20d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x20d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1cd0000 Read this (4 bytes): BBBB 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:50:20 (1787367020) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0303353 s, 270 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:50:27 (1787367027) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443759 s, 23.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1cc lmm_fid: [0x200000405:0x1cc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1cc lmm_fid: [0x200000405:0x1cc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f5:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611596 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-21 22:50:29.000000000 -0400 Modify: 2026-08-21 22:50:29.000000000 -0400 Change: 2026-08-21 22:50:31.000000000 -0400 Birth: 2026-08-21 22:50:29.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:50:37 (1787367037) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507433 s, 20.7 MB/s oleg451-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg451-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg451-server: O/280000401/d19/2f3: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:51:01 (1787367061) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.23597 s, 174 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0311044 s, 132 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 21 22:51 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d1:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 21 22:51 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:51:10 (1787367070) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:51:20 (1787367080) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1995 sec =========== 22:51:31 (1787367091) === sanity-flr: start cleanup 22:51:34 (1787367094) === === sanity-flr: finish cleanup 22:51:44 (1787367104) ===