-----============= acceptance-small: sanity-flr ============----- Sat Aug 15 17:44:54 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg621-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 17:45:14 (1786830314) === oleg621-client.virtnet: executing check_config_client /mnt/lustre oleg621-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg621-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9234a03b8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9234a03b8800.idle_timeout=debug disable quota as required oleg621-server: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 17:45:49 (1786830349) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8210] == sanity-flr test 0a: lfs mirror create with -N option == 17:45:52 (1786830352) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:46:08 (1786830368) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:46:10 (1786830370) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:46:13 (1786830373) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:46:36 (1786830396) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:46:40 (1786830400) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 17:46:43 (1786830403) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9234a03b8800.stats=clear osc.lustre-OST0001-osc-ffff9234a03b8800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:46:51 (1786830411) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:47:01 (1786830421) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:47:10 (1786830430) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 17:47:13 (1786830433) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:47:22 (1786830442) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 17:47:31 (1786830451) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:47:37 (1786830457) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:47:44 (1786830464) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:47:50 (1786830470) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:47:59 (1786830479) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:48:06 (1786830486) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.381287 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106388 s, 9.9 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:48:14 (1786830494) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.692064 s, 27.3 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 17:48:22 (1786830502) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:48:34 (1786830514) Stopping /mnt/lustre-ost1 (opts:) on oleg621-server oleg621-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9234a03b8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9234a03b8800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg621-server: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg621-server: oleg621-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg621-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg621-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg621-server oleg621-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9234a03b8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9234a03b8800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg621-server: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg621-server: oleg621-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg621-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (127s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:50:41 (1786830641) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.203397 s, 25.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg621-server oleg621-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9234a03b8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9234a03b8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg621-server: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg621-server: oleg621-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg621-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (31s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:51:13 (1786830673) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg621-server oleg621-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9234a03b8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9234a03b8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg621-server: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg621-server: oleg621-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg621-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg621-server oleg621-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9234a03b8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9234a03b8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg621-server: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg621-server: oleg621-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg621-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (63s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:52:16 (1786830736) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.63855 s, 19.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 15 17:52 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.984s user 0m0.000s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg621-server oleg621-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9234a03b8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9234a03b8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.921s user 0m0.001s sys 0m0.183s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg621-server: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg621-server: oleg621-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg621-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg621-server oleg621-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9234a03b8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9234a03b8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.975s user 0m0.002s sys 0m0.192s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg621-server: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg621-server: oleg621-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg621-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg621-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9234a03b8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9234a03b8800.ost_server_uuid in FULL state after 0 sec PASS 33b (81s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:53:37 (1786830817) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:53:40 (1786830820) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:53:43 (1786830823) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 17:53:46 (1786830826) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 17:53:52 (1786830832) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.803878 s, 15.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:54:05 (1786830845) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235422 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024622 s, 42.6 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:54:12 (1786830852) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:54:20 (1786830860) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111407 s, 919 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00129726 s, 789 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00310858 s, 329 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121691 s, 841 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 17:54:29 (1786830869) create test files with size 15756954 .. 9d0665439f239f9e70641df39ad92c7c - 06cacdcb8d523a361a2d41fcdd37b8ad - f8d479f909fbc10ec0375ffe4cb81f71 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg621-client.virtnet /mnt/lustre (opts:) Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 17:54:53 (1786830893) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:55:06 (1786830906) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:55:15 (1786830915) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114927 s, 9.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027982 s, 37.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 17:55:27 (1786830927) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0812219 s, 25.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.339553 s, 12.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162117 s, 12.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 17:55:42 (1786830942) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 17:55:45 (1786830945) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:55:49 (1786830949) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027498 s, 38.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 17:55:56 (1786830956) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.288115 s, 10.9 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:56:06 (1786830966) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:56:15 (1786830975) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.380354 s, 27.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg621-client.virtnet /mnt/lustre (opts:) Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:56:27 (1786830987) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff92348440a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.303083 s, 34.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg621-client.virtnet /mnt/lustre (opts:) Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff923482ab0800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:56:42 (1786831002) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (33s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:57:15 (1786831035) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:57:55 (1786831075) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 17:57:58 (1786831078) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.241274 s, 13.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.115541 s, 27.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9a957c89 (nosync,stale) Mirror 2: 0x9a957c89 (nosync,stale) ** verify mirror 2 doesn't change original checksum: d66ee5105a765393c0201004cc182aff - mirror 1 checksum: 1f2f7b595e1c69606e5950a17250ff00 - mirror 2 checksum: d66ee5105a765393c0201004cc182aff - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T17:58:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T17:58:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d66ee5105a765393c0201004cc182aff - mirror 1 checksum: 1f2f7b595e1c69606e5950a17250ff00 - mirror 2 checksum: 1f2f7b595e1c69606e5950a17250ff00 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T17:58:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T17:58:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 17:58:09 (1786831089) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269807 s, 38.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:58:16 (1786831096) Failing mds1 on oleg621-server Stopping /mnt/lustre-mds1 (opts:) on oleg621-server 17:58:28 (1786831108) shut down facet: mds1 facet_host: oleg621-server facet_failover_host: oleg621-server Failover mds1 to oleg621-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg621-server: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Started lustre-MDT0000 17:58:48 (1786831128) targets are mounted 17:58:48 (1786831128) facet_failover done oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02896 s, 36.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 50A (45s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:59:01 (1786831141) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0424158 s, 483 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.016506 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:59:10 (1786831150) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0686617 s, 298 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0243436 s, 841 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:59:19 (1786831159) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00569651 s, 2.9 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:59:26 (1786831166) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0436896 s, 469 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:59:32 (1786831172) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0135343 s, 303 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0419815 s, 97.6 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:59:39 (1786831179) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:59:58 (1786831198) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (25s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:00:23 (1786831223) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:00:45 (1786831245) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:01:07 (1786831267) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 18:01:13 (1786831273) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57557 mirror split pid 57558 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57557 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57558 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 18:02:19 (1786831339) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 1341 fallocating to largest ever: 0x26e31 000003 1786831341.890861 fallocate 0x021855 thru 0x026e30 (0x55dc bytes) 000004 1786831341.898572 read 0x013ab5 thru 0x01e2aa (0xa7f6 bytes) 000005 1786831341.910612 mapwrite 0x004e51 thru 0x012547 (0xd6f7 bytes) 000006 1786831341.961559 trunc from 0x00da58 to 0x01576a (0x7d13 bytes) 000007 1786831341.967120 trunc from 0x0069da to 0x013074 (0xc69b bytes) truncating to largest ever: 0x324fb 000008 1786831341.972576 trunc from 0x026e31 to 0x0324fa (0xb6ca bytes) 000009 1786831341.996055 fallocate 0x005e14 thru 0x015cbe (0xfeab bytes) 000010 1786831342.003172 mapread 0x02a5dc thru 0x0324fa (0x7f1f bytes) 000011 1786831342.016875 trunc from 0x0324fb to 0x019184 (0x19376 bytes) 000012 1786831342.033052 fallocate 0x001928 thru 0x00a52e (0x8c07 bytes) 000014 1786831342.045881 MIRROR_EXTEND to 2 mirrors 000015 1786831342.830501 mapwrite 0x03036e thru 0x032069 (0x1cfc bytes) 000016 1786831342.961554 mapread 0x012463 thru 0x01eec4 (0xca62 bytes) 000017 1786831342.995507 MIRROR_RESYNC 2 mirrors 000018 1786831343.806204 write 0x01d5fd thru 0x028432 (0xae36 bytes) 000019 1786831343.846873 mapread 0x029c49 thru 0x032069 (0x8421 bytes) truncating to largest ever: 0x3b7ce 000020 1786831343.866813 trunc from 0x03206a to 0x03b7cd (0x9764 bytes) 000021 1786831343.885789 write 0x01616e thru 0x02246f (0xc302 bytes) 000022 1786831343.893120 read 0x0191fa thru 0x019b0f (0x916 bytes) 000023 1786831343.901174 MIRROR_RESYNC 2 mirrors 000024 1786831344.508296 trunc from 0x000a6b to 0x00a5ea (0x9b80 bytes) 000025 1786831344.538138 read 0x00ca44 thru 0x00f32f (0x28ec bytes) 000026 1786831344.545822 read 0x0339cd thru 0x03b7cd (0x7e01 bytes) 000027 1786831344.559506 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000029 1786831344.728704 trunc from 0x03b7ce to 0x010006 (0x2b7c7 bytes) truncating to largest ever: 0x3d0db 000030 1786831344.750071 trunc from 0x010007 to 0x03d0da (0x2d0d4 bytes) 000031 1786831344.765372 read 0x005239 thru 0x007fa6 (0x2d6e bytes) 000033 1786831344.774656 write 0x02ca49 thru 0x030bbe (0x4176 bytes) 000034 1786831344.788607 trunc from 0x02c9c8 to 0x03060f (0x3c48 bytes) 000036 1786831344.808038 fallocate 0x00130a thru 0x005716 (0x440d bytes) 000037 1786831344.814491 fallocate 0x003f4f thru 0x004cbb (0xd6d bytes) 000038 1786831344.825290 MIRROR_EXTEND to 2 mirrors 000039 1786831345.498907 write 0x03941b thru 0x039ee3 (0xac9 bytes) 000040 1786831345.550359 trunc from 0x03d0db to 0x001b3f (0x3b59b bytes) 000041 1786831345.585478 fallocate 0x007557 thru 0x00e748 (0x71f2 bytes) 000042 1786831345.591442 trunc from 0x0016c2 to 0x001b3f (0x47e bytes) 000043 1786831345.596278 read 0x0009b3 thru 0x001b3f (0x118d bytes) 000044 1786831345.601221 punch from 0x0012ee to 0x001b3f (0x852 bytes) 000045 1786831345.611262 trunc from 0x00143c to 0x001b3f (0x704 bytes) 000046 1786831345.621717 mapread 0x001aab thru 0x001b3f (0x95 bytes) 000047 1786831345.628399 punch from 0x00164c to 0x001b3f (0x4f4 bytes) 000048 1786831345.637129 mapread 0x001547 thru 0x001b3f (0x5f9 bytes) 000049 1786831345.644389 write 0x039a2b thru 0x03ffff (0x65d5 bytes) 000052 1786831345.647821 punch from 0x039a89 to 0x03ffff (0x6577 bytes) 000053 1786831345.665660 punch from 0x001e7c to 0x00f14e (0xd2d3 bytes) 000054 1786831345.672877 mapwrite 0x014087 thru 0x021feb (0xdf65 bytes) 000055 1786831345.719938 trunc from 0x00ee94 to 0x01b1b0 (0xc31d bytes) 000056 1786831345.726613 write 0x0228ae thru 0x023ac9 (0x121c bytes) 000057 1786831345.728391 trunc from 0x040000 to 0x016977 (0x29688 bytes) 000058 1786831345.741152 write 0x038932 thru 0x03a78a (0x1e59 bytes) 000059 1786831345.744145 trunc from 0x03a78b to 0x001f9d (0x387ed bytes) 000060 1786831345.760123 fallocate 0x01d938 thru 0x02bd75 (0xe43e bytes) 000061 1786831345.769464 punch from 0x000c80 to 0x001f9d (0x131e bytes) 000062 1786831345.775996 mapwrite 0x0166e9 thru 0x01a649 (0x3f61 bytes) 000063 1786831345.820241 mapwrite 0x0027b9 thru 0x00f4f0 (0xcd38 bytes) 000064 1786831345.873973 trunc from 0x01a64a to 0x039fc2 (0x1f979 bytes) 000065 1786831345.897086 punch from 0x0103c3 to 0x01a40c (0xa04a bytes) 000066 1786831345.904161 mapwrite 0x01ed5b thru 0x027f32 (0x91d8 bytes) 000067 1786831345.945068 trunc from 0x00912c to 0x01847a (0xf34f bytes) 000068 1786831345.950456 trunc from 0x009b99 to 0x00e7ae (0x4c16 bytes) 000069 1786831345.955953 mapread 0x0054df thru 0x00b89e (0x63c0 bytes) 000070 1786831345.966636 trunc from 0x039fc3 to 0x032a92 (0x7530 bytes) 000071 1786831345.986336 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000072 1786831346.180419 read 0x01b6ae thru 0x021f03 (0x6856 bytes) 000073 1786831346.196574 read 0x030c34 thru 0x032a92 (0x1e5f bytes) 000074 1786831346.219546 trunc from 0x032a93 to 0x016686 (0x1c40c bytes) 000075 1786831346.248104 fallocate 0x002e04 thru 0x00a726 (0x7923 bytes) 000076 1786831346.272609 read 0x00bc3b thru 0x016686 (0xaa4c bytes) 000077 1786831346.273468 write 0x035e96 thru 0x037376 (0x14e1 bytes) 000078 1786831346.282151 trunc from 0x037377 to 0x025568 (0x11e0e bytes) 000079 1786831346.311448 punch from 0x01a232 to 0x01c1bc (0x1f8b bytes) fallocating to largest ever: 0x40000 000080 1786831346.325649 fallocate 0x039f3a thru 0x03ffff (0x60c6 bytes) 000081 1786831346.333423 trunc from 0x03a113 to 0x03df6b (0x3e59 bytes) 000083 1786831346.347061 trunc from 0x040000 to 0x03ae99 (0x5166 bytes) 000084 1786831346.372535 write 0x0076eb thru 0x00dc2b (0x6541 bytes) 000085 1786831346.375597 punch from 0x02ee7f to 0x03151a (0x269c bytes) 000086 1786831346.380153 read 0x030943 thru 0x035f91 (0x564f bytes) 000087 1786831346.392957 trunc from 0x03ae9a to 0x026b70 (0x14329 bytes) 000088 1786831346.407298 write 0x01260f thru 0x01b9bc (0x93ae bytes) 000089 1786831346.408762 punch from 0x01823e to 0x024a1a (0xc7dd bytes) 000090 1786831346.423333 mapwrite 0x00c609 thru 0x01b0e5 (0xeadd bytes) 000091 1786831346.489390 write 0x01065b thru 0x01dfaa (0xd950 bytes) 000092 1786831346.498328 read 0x02457e thru 0x026b70 (0x25f3 bytes) 000093 1786831346.515510 fallocate 0x028ca0 thru 0x0303cd (0x772e bytes) 000094 1786831346.526037 read 0x022ac8 thru 0x02cc84 (0xa1bd bytes) 000095 1786831346.539492 read 0x023000 thru 0x0230ab (0xac bytes) 000097 1786831346.544485 write 0x03e891 thru 0x03ffff (0x176f bytes) 000098 1786831346.551487 MIRROR_EXTEND to 2 mirrors 000099 1786831347.352423 MIRROR_RESYNC 2 mirrors 000100 1786831347.883625 write 0x0129d8 thru 0x022112 (0xf73b bytes) 000101 1786831347.949906 trunc from 0x040000 to 0x02f47d (0x10b82 bytes) 000102 1786831347.973481 trunc from 0x02f47e to 0x013920 (0x1bb5d bytes) 000103 1786831347.997955 fallocate 0x021912 thru 0x0295da (0x7cc9 bytes) 000104 1786831348.006040 write 0x036f9a thru 0x03ffff (0x9066 bytes) 000105 1786831348.011167 fallocate 0x009fe9 thru 0x00c82c (0x2844 bytes) 000106 1786831348.021285 fallocate 0x017b4d thru 0x017e11 (0x2c5 bytes) 000107 1786831348.030862 read 0x01946a thru 0x0218a1 (0x8438 bytes) 000108 1786831348.047443 fallocate 0x001190 thru 0x00b919 (0xa78a bytes) 000109 1786831348.057116 write 0x01808b thru 0x01b423 (0x3399 bytes) 000110 1786831348.064404 mapread 0x000a76 thru 0x0059c4 (0x4f4f bytes) 000111 1786831348.096101 MIRROR_RESYNC 2 mirrors 000112 1786831348.947482 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000113 1786831349.075645 mapwrite 0x029549 thru 0x035c68 (0xc720 bytes) 000114 1786831349.134343 trunc from 0x040000 to 0x03d77b (0x2884 bytes) 000115 1786831349.176177 write 0x02185a thru 0x024dda (0x3581 bytes) 000116 1786831349.183085 mapread 0x03baff thru 0x03d77b (0x1c7d bytes) 000119 1786831349.191538 trunc from 0x003069 to 0x004022 (0xfba bytes) 000120 1786831349.196816 mapread 0x01e8b9 thru 0x01f401 (0xb49 bytes) 000121 1786831349.207854 trunc from 0x03d77c to 0x030de5 (0xc996 bytes) 000122 1786831349.224119 read 0x0136f6 thru 0x01dc82 (0xa58d bytes) 000123 1786831349.230893 mapwrite 0x01a05f thru 0x01a275 (0x217 bytes) 000125 1786831349.258006 fallocate 0x017b71 thru 0x023465 (0xb8f5 bytes) 000126 1786831349.277206 trunc from 0x030de6 to 0x019e82 (0x16f63 bytes) 000127 1786831349.293620 punch from 0x007f36 to 0x0165c1 (0xe68c bytes) 000128 1786831349.304503 read 0x0052ae thru 0x013df4 (0xeb47 bytes) 000129 1786831349.315083 fallocate 0x03ae70 thru 0x03ffff (0x5190 bytes) 000130 1786831349.320199 punch from 0x0200a5 to 0x027180 (0x70dc bytes) 000131 1786831349.324843 punch from 0x00cc80 to 0x014f6f (0x82f0 bytes) 000132 1786831349.330173 fallocate 0x000a91 thru 0x009397 (0x8907 bytes) 000134 1786831349.337531 mapread 0x009f98 thru 0x018a6e (0xead7 bytes) 000135 1786831349.385079 read 0x005d2d thru 0x0081e5 (0x24b9 bytes) 000137 1786831349.386290 punch from 0x00d4b9 to 0x010114 (0x2c5c bytes) 000138 1786831349.392475 mapread 0x001db1 thru 0x004d38 (0x2f88 bytes) 000139 1786831349.394149 trunc from 0x00e001 to 0x015115 (0x7115 bytes) 000140 1786831349.400260 punch from 0x02913a to 0x02afaa (0x1e71 bytes) 000141 1786831349.405353 write 0x022433 thru 0x02ac2a (0x87f8 bytes) 000142 1786831349.421702 read 0x000c09 thru 0x008173 (0x756b bytes) 000143 1786831349.422500 write 0x033fa5 thru 0x03cfee (0x904a bytes) 000144 1786831349.424219 fallocate 0x003098 thru 0x008da3 (0x5d0c bytes) 000146 1786831349.431696 fallocate 0x039132 thru 0x03ffff (0x6ece bytes) 000147 1786831349.437567 mapread 0x037a2b thru 0x03f760 (0x7d36 bytes) 000148 1786831349.439868 fallocate 0x02d2ce thru 0x0312bb (0x3fee bytes) 000149 1786831349.451454 trunc from 0x040000 to 0x03ebca (0x1435 bytes) 000150 1786831349.471902 write 0x01643b thru 0x01b677 (0x523d bytes) 000151 1786831349.473285 read 0x002c90 thru 0x003d9f (0x1110 bytes) 000152 1786831349.474672 mapread 0x028512 thru 0x038437 (0xff26 bytes) 000153 1786831349.476539 punch from 0x006333 to 0x011a8d (0xb75b bytes) 000154 1786831349.495052 fallocate 0x004492 thru 0x0114cc (0xd03b bytes) 000156 1786831349.506227 trunc from 0x03ebcb to 0x01fcc9 (0x1ef01 bytes) 000157 1786831349.530394 read 0x01c715 thru 0x01fcc9 (0x35b5 bytes) 000158 1786831349.532107 read 0x00bbba thru 0x012d00 (0x7147 bytes) 000159 1786831349.542638 fallocate 0x022448 thru 0x0323cf (0xff88 bytes) 000162 1786831349.560716 punch from 0x015d0d to 0x022894 (0xcb88 bytes) 000163 1786831349.583490 trunc from 0x0323d0 to 0x01c7dd (0x15bf2 bytes) 000164 1786831349.611385 punch from 0x00b965 to 0x00e8ab (0x2f47 bytes) 000166 1786831349.619576 punch from 0x01b47f to 0x01c7dd (0x135f bytes) 000167 1786831349.627116 trunc from 0x01bc33 to 0x01c7dd (0xbab bytes) 000168 1786831349.635521 trunc from 0x01c7de to 0x00d30c (0xf4d1 bytes) 000169 1786831349.673122 trunc from 0x000d8e to 0x0015b0 (0x823 bytes) 000170 1786831349.688715 fallocate 0x011f60 thru 0x01e436 (0xc4d7 bytes) 000172 1786831349.698535 read 0x003620 thru 0x0087d0 (0x51b1 bytes) 000173 1786831349.700172 read 0x0009bd thru 0x0077cb (0x6e0f bytes) 000174 1786831349.714501 mapwrite 0x01a231 thru 0x01ea90 (0x4860 bytes) 000175 1786831349.778146 trunc from 0x013eba to 0x01de68 (0x9faf bytes) 000176 1786831349.789370 trunc from 0x01cab8 to 0x01ea90 (0x1fd9 bytes) 000177 1786831349.800061 write 0x031bf6 thru 0x03879f (0x6baa bytes) 000178 1786831349.809671 mapwrite 0x0127ac thru 0x018cb3 (0x6508 bytes) 000179 1786831349.898562 mapwrite 0x02ee50 thru 0x037cf2 (0x8ea3 bytes) 000180 1786831349.928443 mapread 0x027a8e thru 0x02e939 (0x6eac bytes) 000182 1786831349.930643 mapread 0x02ede8 thru 0x033d32 (0x4f4b bytes) 000184 1786831349.932436 mapread 0x037226 thru 0x03879f (0x157a bytes) 000185 1786831349.936034 mapread 0x01336c thru 0x021ff6 (0xec8b bytes) 000186 1786831350.005715 fallocate 0x00d102 thru 0x00fd60 (0x2c5f bytes) 000187 1786831350.011457 punch from 0x01d253 to 0x022e0f (0x5bbd bytes) 000188 1786831350.016509 mapwrite 0x03a168 thru 0x03ffff (0x5e98 bytes) 000190 1786831350.061619 read 0x01ce67 thru 0x024bdc (0x7d76 bytes) 000191 1786831350.086151 mapwrite 0x02f032 thru 0x03ad1f (0xbcee bytes) 000193 1786831350.119104 fallocate 0x0265e5 thru 0x02cd4a (0x6766 bytes) 000195 1786831350.124433 mapread 0x0226d0 thru 0x026172 (0x3aa3 bytes) 000196 1786831350.125977 mapwrite 0x02f345 thru 0x031ac8 (0x2784 bytes) 000197 1786831350.144504 punch from 0x03bde2 to 0x03ffff (0x421e bytes) 000199 1786831350.157252 mapread 0x031c71 thru 0x0344c9 (0x2859 bytes) 000200 1786831350.159539 trunc from 0x00f9e1 to 0x014168 (0x4788 bytes) 000201 1786831350.166936 trunc from 0x0215d3 to 0x0222d0 (0xcfe bytes) 000202 1786831350.174923 trunc from 0x040000 to 0x035c90 (0xa36f bytes) 000204 1786831350.194426 write 0x0076b2 thru 0x007d3d (0x68c bytes) 000205 1786831350.196016 trunc from 0x02f50c to 0x030f00 (0x19f5 bytes) 000207 1786831350.203071 mapread 0x035972 thru 0x035c90 (0x31f bytes) 000208 1786831350.204768 fallocate 0x0077a3 thru 0x0165ed (0xee4b bytes) 000209 1786831350.210493 fallocate 0x029dca thru 0x02c974 (0x2bab bytes) 000210 1786831350.215786 mapwrite 0x026361 thru 0x02d05f (0x6cff bytes) 000211 1786831350.233710 trunc from 0x035c91 to 0x01bcd5 (0x19fbb bytes) 000212 1786831350.249655 write 0x0074cb thru 0x0090b6 (0x1bec bytes) 000213 1786831350.251277 punch from 0x0025a9 to 0x003b5a (0x15b2 bytes) 000214 1786831350.259214 trunc from 0x01bcd6 to 0x00a805 (0x114d0 bytes) 000215 1786831350.283313 punch from 0x00228f to 0x0080aa (0x5e1c bytes) 000216 1786831350.307354 MIRROR_EXTEND to 2 mirrors 000217 1786831350.894066 read 0x000dc5 thru 0x002ded (0x2029 bytes) 000218 1786831350.906044 mapread 0x009171 thru 0x00a51d (0x13ad bytes) 000219 1786831350.940188 mapwrite 0x0228d4 thru 0x024c1e (0x234b bytes) 000220 1786831351.096680 write 0x0290f8 thru 0x031cf3 (0x8bfc bytes) 000221 1786831351.100451 trunc from 0x031cf4 to 0x02766a (0xa689 bytes) 000222 1786831351.147712 trunc from 0x020a9f to 0x0219d0 (0xf32 bytes) 000223 1786831351.153423 write 0x018387 thru 0x028309 (0xff83 bytes) 000224 1786831351.160187 mapwrite 0x02464e thru 0x0274b4 (0x2e67 bytes) 000225 1786831351.197911 write 0x02e2fa thru 0x03b627 (0xd32e bytes) 000226 1786831351.206664 trunc from 0x03b628 to 0x014dc8 (0x2685f bytes) 000227 1786831351.229124 mapwrite 0x0395b4 thru 0x039858 (0x2a5 bytes) 000228 1786831351.357882 trunc from 0x039859 to 0x01a91a (0x1ef3e bytes) 000229 1786831351.408697 mapread 0x016480 thru 0x01a91a (0x449b bytes) 000230 1786831351.422869 mapwrite 0x0227af thru 0x02a1dc (0x7a2e bytes) 000231 1786831351.554481 mapread 0x0037a9 thru 0x00fb41 (0xc399 bytes) 000232 1786831351.572795 mapread 0x00f8e0 thru 0x01311a (0x383b bytes) 000233 1786831351.584243 mapread 0x005759 thru 0x009b41 (0x43e9 bytes) 000234 1786831351.599184 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000235 1786831351.781747 punch from 0x006fae to 0x0087f9 (0x184c bytes) 000236 1786831351.797413 read 0x0225ef thru 0x02a1dc (0x7bee bytes) 000237 1786831351.815143 trunc from 0x02a1dd to 0x029d2f (0x4ad bytes) 000238 1786831351.846716 mapread 0x02501b thru 0x029a02 (0x49e8 bytes) 000239 1786831351.848026 trunc from 0x0068fb to 0x00a93d (0x4043 bytes) 000240 1786831351.856502 trunc from 0x029d30 to 0x037344 (0xd615 bytes) 000241 1786831351.877474 trunc from 0x037345 to 0x01360e (0x23d36 bytes) 000242 1786831351.905848 fallocate 0x0397f6 thru 0x03ffff (0x680a bytes) 000244 1786831351.916969 write 0x0083ab thru 0x013acd (0xb723 bytes) 000245 1786831351.931498 fallocate 0x02f1b3 thru 0x03e72c (0xf57a bytes) 000246 1786831351.936150 fallocate 0x024520 thru 0x02eb77 (0xa658 bytes) 000247 1786831351.940457 mapwrite 0x024947 thru 0x028f11 (0x45cb bytes) 000248 1786831351.975687 punch from 0x00aaa2 to 0x012d74 (0x82d3 bytes) 000249 1786831351.981256 mapread 0x02b352 thru 0x030366 (0x5015 bytes) 000250 1786831351.996088 fallocate 0x03b82b thru 0x03ffff (0x47d5 bytes) 000253 1786831352.027981 MIRROR_EXTEND to 2 mirrors 000254 1786831352.908872 mapread 0x02a217 thru 0x033d30 (0x9b1a bytes) 000255 1786831352.960872 punch from 0x0212d0 to 0x02ad89 (0x9aba bytes) 000256 1786831352.998690 trunc from 0x00e3d2 to 0x01815f (0x9d8e bytes) 000257 1786831353.004047 mapread 0x017852 thru 0x02470c (0xcebb bytes) 000258 1786831353.027340 write 0x00145e thru 0x00e0f8 (0xcc9b bytes) 000259 1786831353.044331 fallocate 0x02fc15 thru 0x03bcb5 (0xc0a1 bytes) 000260 1786831353.049286 mapwrite 0x02f323 thru 0x03e05d (0xed3b bytes) 000261 1786831353.113070 trunc from 0x021617 to 0x027725 (0x610f bytes) 000262 1786831353.118906 read 0x0096c7 thru 0x018900 (0xf23a bytes) 000263 1786831353.124213 read 0x00b5ed thru 0x016da7 (0xb7bb bytes) 000264 1786831353.125013 mapread 0x01aad7 thru 0x026dc3 (0xc2ed bytes) 000265 1786831353.136985 mapwrite 0x03d8eb thru 0x03ffff (0x2715 bytes) 000266 1786831353.155460 mapread 0x019ff7 thru 0x01eb68 (0x4b72 bytes) 000268 1786831353.159472 mapwrite 0x02e517 thru 0x038bf8 (0xa6e2 bytes) 000269 1786831353.184797 mapwrite 0x03b786 thru 0x03ffff (0x487a bytes) 000270 1786831353.204104 mapread 0x028317 thru 0x02f1a4 (0x6e8e bytes) 000273 1786831353.207334 trunc from 0x040000 to 0x02f434 (0x10bcb bytes) 000274 1786831353.227020 fallocate 0x0045e9 thru 0x00b191 (0x6ba9 bytes) 000276 1786831353.233639 write 0x0236fa thru 0x029ee9 (0x67f0 bytes) 000278 1786831353.236783 fallocate 0x029b72 thru 0x037c77 (0xe106 bytes) 000279 1786831353.243524 fallocate 0x029023 thru 0x031a01 (0x89df bytes) 000280 1786831353.251021 read 0x012d5d thru 0x016d05 (0x3fa9 bytes) 000281 1786831353.251917 mapwrite 0x00e03e thru 0x017eff (0x9ec2 bytes) 000283 1786831353.281863 mapwrite 0x008839 thru 0x016ac2 (0xe28a bytes) 000284 1786831353.318058 mapwrite 0x02b661 thru 0x02f307 (0x3ca7 bytes) 000285 1786831353.350983 mapread 0x0021af thru 0x00c9d3 (0xa825 bytes) 000286 1786831353.359500 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000288 1786831353.520664 read 0x016fd4 thru 0x026f29 (0xff56 bytes) 000289 1786831353.541092 trunc from 0x02791a to 0x031678 (0x9d5f bytes) 000290 1786831353.548077 mapwrite 0x02632b thru 0x03584c (0xf522 bytes) 000291 1786831353.645961 trunc from 0x03584d to 0x008048 (0x2d804 bytes) 000292 1786831353.665579 trunc from 0x008049 to 0x008c89 (0xc41 bytes) 000293 1786831353.689536 read 0x008303 thru 0x008c89 (0x987 bytes) 000295 1786831353.701491 write 0x03022f thru 0x03f3f5 (0xf1c7 bytes) 000296 1786831353.704799 fallocate 0x039bd2 thru 0x03ffff (0x642e bytes) 000297 1786831353.709687 mapread 0x0328c4 thru 0x0384b0 (0x5bed bytes) 000298 1786831353.711719 mapwrite 0x02dddd thru 0x036ae8 (0x8d0c bytes) 000301 1786831353.741629 mapread 0x015fc4 thru 0x01e4ad (0x84ea bytes) 000302 1786831353.743627 write 0x003a0f thru 0x00df78 (0xa56a bytes) 000303 1786831353.750775 punch from 0x010614 to 0x011323 (0xd10 bytes) 000306 1786831353.767551 mapwrite 0x0049ac thru 0x006088 (0x16dd bytes) 000307 1786831353.791848 read 0x032dcc thru 0x03f3f5 (0xc62a bytes) 000308 1786831353.792689 trunc from 0x03f3f6 to 0x00abd0 (0x34825 bytes) 000309 1786831353.815529 fallocate 0x03c57e thru 0x03ffff (0x3a82 bytes) 000310 1786831353.831239 fallocate 0x023533 thru 0x02656c (0x303a bytes) 000311 1786831353.844434 trunc from 0x02ec66 to 0x031f71 (0x330c bytes) 000312 1786831353.850394 fallocate 0x000890 thru 0x006c39 (0x63aa bytes) 000313 1786831353.875034 MIRROR_EXTEND to 2 mirrors 000314 1786831354.473164 read 0x0084b6 thru 0x00c5f6 (0x4141 bytes) 000316 1786831354.504314 mapread 0x011bf4 thru 0x0173e5 (0x57f2 bytes) 000317 1786831354.559854 MIRROR_RESYNC 2 mirrors 000318 1786831354.918568 read 0x0055cf thru 0x012c26 (0xd658 bytes) 000319 1786831354.936975 trunc from 0x040000 to 0x01fedd (0x20122 bytes) 000320 1786831354.999663 trunc from 0x01d31f to 0x01fedd (0x2bbf bytes) 000321 1786831355.005759 write 0x037f4f thru 0x03ffff (0x80b1 bytes) 000322 1786831355.010193 punch from 0x039c2d to 0x03ffff (0x63d3 bytes) 000323 1786831355.030123 read 0x02b6ee thru 0x032eaf (0x77c2 bytes) 000324 1786831355.110537 write 0x03782f thru 0x03d957 (0x6129 bytes) 000325 1786831355.117520 read 0x015084 thru 0x017c82 (0x2bff bytes) 000327 1786831355.121223 mapread 0x02a6df thru 0x02f3e5 (0x4d07 bytes) 000329 1786831355.127433 mapread 0x01b37a thru 0x0201da (0x4e61 bytes) 000330 1786831355.130428 mapread 0x00cfde thru 0x01b807 (0xe82a bytes) 000331 1786831355.132184 punch from 0x034c51 to 0x039e56 (0x5206 bytes) 000332 1786831355.154822 MIRROR_RESYNC 2 mirrors 000333 1786831355.901024 trunc from 0x040000 to 0x00c166 (0x33e99 bytes) 000334 1786831355.976268 fallocate 0x01d616 thru 0x02cf67 (0xf952 bytes) 000335 1786831355.995888 MIRROR_RESYNC 2 mirrors 000336 1786831356.567395 fallocate 0x02701e thru 0x02e5f9 (0x75dc bytes) 000337 1786831356.595245 fallocate 0x038399 thru 0x03edbd (0x6a25 bytes) 000338 1786831356.601045 write 0x03d7cf thru 0x03ffff (0x2831 bytes) 000339 1786831356.615935 mapwrite 0x015c02 thru 0x01ec96 (0x9095 bytes) 000340 1786831356.657493 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000341 1786831356.790611 read 0x0229c0 thru 0x02c4be (0x9aff bytes) 000343 1786831356.805582 trunc from 0x0056d9 to 0x0134c9 (0xddf1 bytes) 000344 1786831356.813028 trunc from 0x040000 to 0x036539 (0x9ac6 bytes) 000347 1786831356.845836 mapwrite 0x0260c3 thru 0x0294b9 (0x33f7 bytes) 000348 1786831356.887444 MIRROR_EXTEND to 2 mirrors 000349 1786831357.871301 punch from 0x031240 to 0x036539 (0x52fa bytes) 000350 1786831357.896999 mapwrite 0x0339f2 thru 0x03ffff (0xc60e bytes) 000352 1786831357.999475 mapwrite 0x00dd79 thru 0x013182 (0x540a bytes) 000353 1786831358.068972 fallocate 0x022ac5 thru 0x031636 (0xeb72 bytes) 000354 1786831358.083179 mapwrite 0x002864 thru 0x0108c4 (0xe061 bytes) 000355 1786831358.155542 punch from 0x0057dd to 0x00d9d6 (0x81fa bytes) 000356 1786831358.167282 trunc from 0x040000 to 0x0116ed (0x2e912 bytes) 000357 1786831358.187820 trunc from 0x0001b7 to 0x004aa2 (0x48ec bytes) 000358 1786831358.194610 mapwrite 0x0019fa thru 0x00399c (0x1fa3 bytes) 000359 1786831358.224502 write 0x028f01 thru 0x031fba (0x90ba bytes) 000360 1786831358.227562 fallocate 0x015010 thru 0x015409 (0x3fa bytes) 000361 1786831358.239817 trunc from 0x015433 to 0x01f7a4 (0xa372 bytes) 000363 1786831358.254646 fallocate 0x03f824 thru 0x03ffff (0x7dc bytes) 000364 1786831358.290092 mapwrite 0x005a9a thru 0x00f200 (0x9767 bytes) 000365 1786831358.362042 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000366 1786831358.540967 trunc from 0x02538a to 0x02b8ff (0x6576 bytes) 000367 1786831358.549096 trunc from 0x031fbb to 0x008dba (0x29200 bytes) 000368 1786831358.602932 fallocate 0x033e50 thru 0x039ee2 (0x6093 bytes) 000369 1786831358.616640 mapwrite 0x009e70 thru 0x00c417 (0x25a8 bytes) 000370 1786831358.704594 read 0x001e91 thru 0x00c417 (0xa587 bytes) 000373 1786831358.729097 read 0x00ae17 thru 0x00c417 (0x1601 bytes) 000374 1786831358.733860 mapwrite 0x002142 thru 0x0084ab (0x636a bytes) 000375 1786831358.761769 fallocate 0x01f6ae thru 0x02e16e (0xeac1 bytes) 000377 1786831358.777430 write 0x0160de thru 0x0211c1 (0xb0e4 bytes) 000378 1786831358.780463 mapread 0x01b645 thru 0x0211c1 (0x5b7d bytes) 000380 1786831358.786658 trunc from 0x0102f7 to 0x01f6bf (0xf3c9 bytes) 000381 1786831358.811688 write 0x0373ae thru 0x03d156 (0x5da9 bytes) 000382 1786831358.819350 trunc from 0x03d157 to 0x019593 (0x23bc3 bytes) 000383 1786831358.846395 trunc from 0x00ac63 to 0x013639 (0x89d7 bytes) 000384 1786831358.862625 punch from 0x019430 to 0x019593 (0x164 bytes) 000385 1786831358.876142 fallocate 0x028849 thru 0x032bdc (0xa394 bytes) 000386 1786831358.891473 write 0x036b1c thru 0x03f36d (0x8852 bytes) 000387 1786831358.893023 trunc from 0x03f36e to 0x02f37e (0xffef bytes) 000389 1786831358.922657 fallocate 0x03f2d4 thru 0x03ffff (0xd2c bytes) 000390 1786831358.933296 trunc from 0x040000 to 0x025ad6 (0x1a529 bytes) 000392 1786831358.978058 mapread 0x022600 thru 0x025ad6 (0x34d7 bytes) 000393 1786831359.004376 trunc from 0x025ad7 to 0x02d3bc (0x78e6 bytes) 000394 1786831359.039652 punch from 0x011099 to 0x016964 (0x58cc bytes) 000395 1786831359.049520 mapwrite 0x038801 thru 0x03ffff (0x77ff bytes) 000397 1786831359.096506 fallocate 0x00b04e thru 0x014310 (0x92c3 bytes) 000398 1786831359.105442 mapread 0x00ad73 thru 0x00f686 (0x4914 bytes) 000399 1786831359.107879 trunc from 0x037ce4 to 0x03a75c (0x2a79 bytes) 000400 1786831359.113554 trunc from 0x03d698 to 0x03ffff (0x2968 bytes) 000401 1786831359.119713 mapwrite 0x028ea5 thru 0x02a33b (0x1497 bytes) 000402 1786831359.134596 write 0x02c601 thru 0x03af05 (0xe905 bytes) 000403 1786831359.143135 mapwrite 0x00ba1a thru 0x0121ba (0x67a1 bytes) 000404 1786831359.193873 write 0x0160eb thru 0x01f53d (0x9453 bytes) 000406 1786831359.210405 fallocate 0x02ef68 thru 0x03db66 (0xebff bytes) 000407 1786831359.222974 MIRROR_EXTEND to 2 mirrors 000408 1786831359.937098 mapread 0x0112ed thru 0x01f204 (0xdf18 bytes) 000409 1786831359.983138 trunc from 0x026f2e to 0x0364e5 (0xf5b8 bytes) 000410 1786831360.034056 mapread 0x019745 thru 0x01ff93 (0x684f bytes) 000411 1786831360.065106 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8c0e9975 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x614025d2 Mirror 3: 0x614025d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000412 1786831361.099494 trunc from 0x007b37 to 0x00ba2c (0x3ef6 bytes) 000413 1786831361.142367 fallocate 0x03247e thru 0x03907f (0x6c02 bytes) 000414 1786831361.154003 mapwrite 0x009c34 thru 0x0190cc (0xf499 bytes) 000415 1786831361.218315 mapread 0x00c4bd thru 0x00c7c7 (0x30b bytes) 000416 1786831361.219448 trunc from 0x01493f to 0x01d62b (0x8ced bytes) 000417 1786831361.228655 read 0x03dc0e thru 0x03ffff (0x23f2 bytes) 000418 1786831361.236061 mapread 0x0240cb thru 0x032b28 (0xea5e bytes) 000419 1786831361.260700 mapread 0x02bd08 thru 0x02ca75 (0xd6e bytes) 000420 1786831361.266786 MIRROR_RESYNC 3 mirrors 000421 1786831362.397402 mapwrite 0x0127c0 thru 0x019f20 (0x7761 bytes) 000422 1786831362.519719 fallocate 0x02471d thru 0x02a405 (0x5ce9 bytes) 000423 1786831362.527687 mapwrite 0x017ebe thru 0x026e57 (0xef9a bytes) 000425 1786831362.588142 mapwrite 0x03f2e2 thru 0x03ffff (0xd1e bytes) 000426 1786831362.623200 punch from 0x01472a to 0x019b60 (0x5437 bytes) 000428 1786831362.632115 fallocate 0x014bcc thru 0x01be38 (0x726d bytes) 000429 1786831362.639001 trunc from 0x007126 to 0x011547 (0xa422 bytes) 000430 1786831362.652150 MIRROR_RESYNC 3 mirrors 000433 1786831363.763600 MIRROR_RESYNC 3 mirrors 000434 1786831364.111049 punch from 0x033aab to 0x03fec2 (0xc418 bytes) 000435 1786831364.151204 trunc from 0x00354d to 0x004766 (0x121a bytes) 000436 1786831364.158247 mapwrite 0x00e343 thru 0x00eb17 (0x7d5 bytes) 000438 1786831364.183137 trunc from 0x007df0 to 0x00be88 (0x4099 bytes) 000439 1786831364.191447 fallocate 0x003c0d thru 0x004791 (0xb85 bytes) 000440 1786831364.203067 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe8578f39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe8578f39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000441 1786831364.476123 trunc from 0x0290af to 0x02da98 (0x49ea bytes) 000442 1786831364.481475 punch from 0x00586f to 0x00e2df (0x8a71 bytes) 000443 1786831364.490169 trunc from 0x02bbfe to 0x03259b (0x699e bytes) 000444 1786831364.500288 write 0x0024af thru 0x00a424 (0x7f76 bytes) 000445 1786831364.523349 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000446 1786831364.658349 trunc from 0x040000 to 0x00f6c5 (0x3093a bytes) 000448 1786831364.697125 punch from 0x002b77 to 0x00f186 (0xc610 bytes) 000449 1786831364.703065 punch from 0x00e196 to 0x00f6c5 (0x1530 bytes) 000450 1786831364.709468 mapwrite 0x03ba46 thru 0x03ffff (0x45ba bytes) 000451 1786831364.809133 write 0x0168ef thru 0x019d28 (0x343a bytes) 000452 1786831364.840308 read 0x03d791 thru 0x03ffff (0x286f bytes) 000453 1786831364.841057 write 0x01828e thru 0x0250fe (0xce71 bytes) 000454 1786831364.849843 write 0x03850f thru 0x039a77 (0x1569 bytes) 000455 1786831364.851091 write 0x03040c thru 0x0387fb (0x83f0 bytes) 000456 1786831364.858481 write 0x030c83 thru 0x0366d4 (0x5a52 bytes) 000457 1786831364.860075 mapwrite 0x02b2d0 thru 0x035b1b (0xa84c bytes) 000458 1786831364.911837 trunc from 0x040000 to 0x004baf (0x3b450 bytes) 000459 1786831364.935636 mapread 0x001acc thru 0x004baf (0x30e4 bytes) 000460 1786831364.945203 mapread 0x001a33 thru 0x002d97 (0x1365 bytes) 000462 1786831364.948670 trunc from 0x004bb0 to 0x0324b3 (0x2d904 bytes) 000464 1786831364.965093 trunc from 0x001ec7 to 0x00546a (0x35a4 bytes) 000465 1786831364.972467 trunc from 0x0324b4 to 0x02119c (0x11317 bytes) 000467 1786831364.993467 read 0x00dd30 thru 0x015d7b (0x804c bytes) 000468 1786831365.005979 trunc from 0x02119d to 0x033a62 (0x128c6 bytes) 000471 1786831365.025402 mapread 0x02b3db thru 0x032420 (0x7046 bytes) 000472 1786831365.034579 mapread 0x0180a9 thru 0x01a6e9 (0x2641 bytes) 000473 1786831365.036676 trunc from 0x033a63 to 0x027ec2 (0xbba0 bytes) 000474 1786831365.051674 punch from 0x003c55 to 0x00c357 (0x8703 bytes) 000475 1786831365.057359 punch from 0x006c3f to 0x009f76 (0x3338 bytes) 000476 1786831365.061985 trunc from 0x027ec3 to 0x00b415 (0x1caad bytes) 000477 1786831365.079126 read 0x00949d thru 0x00b415 (0x1f79 bytes) 000478 1786831365.090559 read 0x003a3f thru 0x00b415 (0x79d7 bytes) 000479 1786831365.093427 write 0x03fc58 thru 0x03ffff (0x3a8 bytes) 000480 1786831365.095243 mapread 0x038ebe thru 0x03abb7 (0x1cfa bytes) 000482 1786831365.105934 trunc from 0x040000 to 0x03a27c (0x5d83 bytes) 000483 1786831365.127655 fallocate 0x0265e9 thru 0x0277e5 (0x11fd bytes) 000484 1786831365.134138 write 0x02d9a2 thru 0x039759 (0xbdb8 bytes) 000485 1786831365.136650 trunc from 0x03a27d to 0x023064 (0x17218 bytes) 000488 1786831365.159475 mapwrite 0x038b1e thru 0x03ffff (0x74e2 bytes) 000489 1786831365.250507 fallocate 0x034785 thru 0x03ffff (0xb87b bytes) 000490 1786831365.259306 mapwrite 0x02978c thru 0x0366ea (0xcf5f bytes) 000491 1786831365.360646 read 0x039b79 thru 0x03c92c (0x2db4 bytes) 000492 1786831365.361435 punch from 0x00cdfc to 0x01ad7c (0xdf81 bytes) 000493 1786831365.367076 fallocate 0x0339f5 thru 0x03ffff (0xc60b bytes) 000494 1786831365.372926 write 0x00bd58 thru 0x017fa4 (0xc24d bytes) 000495 1786831365.379443 punch from 0x0002b3 to 0x006ec3 (0x6c11 bytes) 000496 1786831365.385123 mapwrite 0x01aa5a thru 0x020d31 (0x62d8 bytes) 000497 1786831365.419803 punch from 0x00ac9f to 0x01166f (0x69d1 bytes) 000499 1786831365.427029 mapwrite 0x020b77 thru 0x03053c (0xf9c6 bytes) 000500 1786831365.454961 mapread 0x00a2d5 thru 0x01496e (0xa69a bytes) 000501 1786831365.518626 mapread 0x012e60 thru 0x01a755 (0x78f6 bytes) 000502 1786831365.519884 write 0x013c93 thru 0x01a049 (0x63b7 bytes) 000503 1786831365.526162 trunc from 0x01bb34 to 0x01e23e (0x270b bytes) 000505 1786831365.537783 mapwrite 0x03254f thru 0x034a24 (0x24d6 bytes) 000506 1786831365.558139 read 0x03fe0f thru 0x03ffff (0x1f1 bytes) 000508 1786831365.559565 read 0x01953d thru 0x025c95 (0xc759 bytes) 000509 1786831365.579720 fallocate 0x0378ab thru 0x039128 (0x187e bytes) 000510 1786831365.594517 MIRROR_EXTEND to 2 mirrors 000512 1786831366.323322 read 0x03bf79 thru 0x03ffff (0x4087 bytes) 000513 1786831366.338003 punch from 0x02aa43 to 0x02ec33 (0x41f1 bytes) 000514 1786831366.366833 write 0x015c6e thru 0x0197da (0x3b6d bytes) 000515 1786831366.380898 fallocate 0x026497 thru 0x030287 (0x9df1 bytes) 000516 1786831366.385458 punch from 0x021eb6 to 0x0289d5 (0x6b20 bytes) 000517 1786831366.390525 mapread 0x00ba8b thru 0x01af58 (0xf4ce bytes) 000518 1786831366.423369 trunc from 0x02aad4 to 0x031824 (0x6d51 bytes) 000521 1786831366.431993 trunc from 0x0236d0 to 0x029477 (0x5da8 bytes) 000522 1786831366.442084 mapread 0x009258 thru 0x00994d (0x6f6 bytes) 000523 1786831366.443261 read 0x02b23a thru 0x03ad0a (0xfad1 bytes) 000524 1786831366.456451 trunc from 0x040000 to 0x02d07b (0x12f84 bytes) 000525 1786831366.478364 read 0x00a98c thru 0x00f275 (0x48ea bytes) 000526 1786831366.479418 read 0x01a6df thru 0x024ad9 (0xa3fb bytes) 000527 1786831366.481174 trunc from 0x003876 to 0x0082ed (0x4a78 bytes) 000528 1786831366.489052 trunc from 0x02d07c to 0x0286c3 (0x49b8 bytes) 000529 1786831366.520738 write 0x02b032 thru 0x02b5b1 (0x580 bytes) 000530 1786831366.524083 write 0x02e3ca thru 0x02e556 (0x18d bytes) 000531 1786831366.526360 write 0x01256c thru 0x01f86a (0xd2ff bytes) 000532 1786831366.530103 punch from 0x023609 to 0x025f6b (0x2963 bytes) 000533 1786831366.535633 mapwrite 0x03607b thru 0x03eee7 (0x8e6d bytes) 000535 1786831366.586980 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000538 1786831366.759493 read 0x037116 thru 0x03a808 (0x36f3 bytes) 000539 1786831366.773081 trunc from 0x03eee8 to 0x000202 (0x3ece5 bytes) 000540 1786831366.796867 trunc from 0x00013f to 0x000202 (0xc4 bytes) 000541 1786831366.808592 MIRROR_EXTEND to 2 mirrors 000542 1786831367.355338 fallocate 0x02997d thru 0x02a9c4 (0x1048 bytes) 000543 1786831367.397913 mapread 0x017241 thru 0x020065 (0x8e25 bytes) 000544 1786831367.441241 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000545 1786831367.601833 mapread 0x01b631 thru 0x025b1c (0xa4ec bytes) 000546 1786831367.628391 fallocate 0x02be02 thru 0x0368d1 (0xaad0 bytes) 000548 1786831367.639668 trunc from 0x02a9c5 to 0x01b894 (0xf130 bytes) 000549 1786831367.663987 punch from 0x0100dd to 0x010382 (0x2a6 bytes) 000550 1786831367.669083 punch from 0x003ff8 to 0x005401 (0x140a bytes) 000551 1786831367.674487 fallocate 0x027921 thru 0x03398a (0xc06a bytes) 000552 1786831367.680471 read 0x031834 thru 0x032b0e (0x12db bytes) 000553 1786831367.687640 mapwrite 0x0235b0 thru 0x0256e0 (0x2131 bytes) 000554 1786831367.711032 punch from 0x033536 to 0x03398a (0x455 bytes) 000555 1786831367.716848 punch from 0x01129b to 0x01651d (0x5283 bytes) 000556 1786831367.723468 punch from 0x02f9ab to 0x02fce4 (0x33a bytes) 000557 1786831367.731309 punch from 0x02ecb8 to 0x03398a (0x4cd3 bytes) 000558 1786831367.738167 mapread 0x02cd7c thru 0x03398a (0x6c0f bytes) 000560 1786831367.751502 read 0x003f5d thru 0x00e4b9 (0xa55d bytes) 000562 1786831367.774157 punch from 0x029e4f to 0x02f745 (0x58f7 bytes) 000563 1786831367.779634 write 0x00bbe6 thru 0x00bd94 (0x1af bytes) 000564 1786831367.780975 trunc from 0x01b57d to 0x028df5 (0xd879 bytes) 000565 1786831367.788787 mapwrite 0x012301 thru 0x01b038 (0x8d38 bytes) 000566 1786831367.824613 mapwrite 0x0196d7 thru 0x026bc8 (0xd4f2 bytes) 000567 1786831367.924335 read 0x01a9c3 thru 0x0238e0 (0x8f1e bytes) 000569 1786831367.925704 trunc from 0x03398b to 0x03a846 (0x6ebc bytes) 000570 1786831367.945930 mapread 0x00758b thru 0x00ebd8 (0x764e bytes) 000571 1786831367.950650 write 0x00870e thru 0x00d930 (0x5223 bytes) 000572 1786831367.954160 punch from 0x028224 to 0x032836 (0xa613 bytes) 000573 1786831367.966511 mapwrite 0x015175 thru 0x01f559 (0xa3e5 bytes) 000574 1786831367.999813 mapwrite 0x00794e thru 0x0114ce (0x9b81 bytes) 000577 1786831368.021875 mapwrite 0x010535 thru 0x01ad23 (0xa7ef bytes) 000578 1786831368.040682 trunc from 0x0295f2 to 0x0374d5 (0xdee4 bytes) 000579 1786831368.048141 read 0x01e64f thru 0x0228ba (0x426c bytes) 000580 1786831368.049375 fallocate 0x028ccd thru 0x03110c (0x8440 bytes) 000581 1786831368.055109 trunc from 0x016310 to 0x02317b (0xce6c bytes) 000582 1786831368.063718 trunc from 0x02324b to 0x024de9 (0x1b9f bytes) 000583 1786831368.071000 trunc from 0x03a847 to 0x00da50 (0x2cdf6 bytes) 000584 1786831368.086891 write 0x00cd1d thru 0x01496c (0x7c50 bytes) 000585 1786831368.088422 read 0x00973d thru 0x01496c (0xb230 bytes) 000586 1786831368.089289 fallocate 0x032f9b thru 0x03356b (0x5d1 bytes) 000587 1786831368.097788 read 0x0279e1 thru 0x03356b (0xbb8b bytes) 000588 1786831368.110458 punch from 0x00027f to 0x00ef87 (0xed09 bytes) 000589 1786831368.125666 trunc from 0x03356c to 0x0278b7 (0xbcb4 bytes) 000591 1786831368.146381 read 0x016277 thru 0x01b7bd (0x5547 bytes) 000592 1786831368.147680 write 0x0311da thru 0x03872e (0x7555 bytes) 000594 1786831368.152297 write 0x028e42 thru 0x032c08 (0x9dc7 bytes) 000595 1786831368.170468 punch from 0x036941 to 0x03872e (0x1dee bytes) 000596 1786831368.186883 read 0x00437f thru 0x008a8a (0x470c bytes) 000597 1786831368.190043 trunc from 0x0101ee to 0x020175 (0xff88 bytes) 000598 1786831368.197057 trunc from 0x03872f to 0x0030e7 (0x35647 bytes) 000599 1786831368.218439 trunc from 0x0030e8 to 0x02e719 (0x2b632 bytes) 000600 1786831368.237114 read 0x004858 thru 0x01184b (0xcff4 bytes) 000601 1786831368.253475 read 0x024b8a thru 0x02e719 (0x9b90 bytes) 000602 1786831368.255177 mapread 0x0253d0 thru 0x02b8a3 (0x64d4 bytes) 000603 1786831368.258431 read 0x005129 thru 0x00809a (0x2f72 bytes) 000604 1786831368.259183 trunc from 0x024ac7 to 0x029bf2 (0x512c bytes) 000605 1786831368.274928 fallocate 0x0263e2 thru 0x02a573 (0x4192 bytes) 000606 1786831368.286384 MIRROR_EXTEND to 2 mirrors 000607 1786831368.903470 punch from 0x0219e2 to 0x02bd4b (0xa36a bytes) 000608 1786831368.937670 trunc from 0x029bb9 to 0x02e719 (0x4b61 bytes) 000609 1786831368.947509 MIRROR_RESYNC 2 mirrors 000611 1786831369.579097 mapwrite 0x029a7f thru 0x03632a (0xc8ac bytes) 000612 1786831369.682115 read 0x01f82f thru 0x02991f (0xa0f1 bytes) 000613 1786831369.695620 MIRROR_RESYNC 2 mirrors 000614 1786831370.209402 trunc from 0x02ceb2 to 0x035907 (0x8a56 bytes) 000616 1786831370.248631 fallocate 0x031c93 thru 0x03ca24 (0xad92 bytes) 000617 1786831370.257191 write 0x036832 thru 0x03e157 (0x7926 bytes) 000618 1786831370.273463 punch from 0x013c5e to 0x01ab7b (0x6f1e bytes) 000620 1786831370.279022 mapwrite 0x032f79 thru 0x0354bf (0x2547 bytes) 000621 1786831370.331790 read 0x0383fa thru 0x03b2b9 (0x2ec0 bytes) 000623 1786831370.333383 mapread 0x039dae thru 0x03e157 (0x43aa bytes) 000624 1786831370.334718 mapread 0x00fe60 thru 0x018bb6 (0x8d57 bytes) 000625 1786831370.358124 mapread 0x03bd5c thru 0x03ddc0 (0x2065 bytes) 000626 1786831370.368357 punch from 0x025354 to 0x02f25f (0x9f0c bytes) 000627 1786831370.379223 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000628 1786831370.511035 read 0x001755 thru 0x007e67 (0x6713 bytes) 000629 1786831370.528663 read 0x01d80c thru 0x0233cc (0x5bc1 bytes) 000630 1786831370.555789 read 0x00ba53 thru 0x011336 (0x58e4 bytes) 000633 1786831370.568618 trunc from 0x03e158 to 0x00246f (0x3bce8 bytes) 000635 1786831370.637580 read 0x000bd7 thru 0x00246f (0x1899 bytes) 000636 1786831370.638565 fallocate 0x01942a thru 0x022395 (0x8f6c bytes) 000638 1786831370.645768 write 0x036e19 thru 0x03eeb7 (0x809f bytes) 000640 1786831370.652157 punch from 0x028803 to 0x03804c (0xf84a bytes) 000641 1786831370.663720 trunc from 0x00375c to 0x00c6a7 (0x8f4c bytes) 000642 1786831370.672159 punch from 0x02f01b to 0x03417e (0x5164 bytes) 000644 1786831370.678577 trunc from 0x03eeb8 to 0x00d4a7 (0x31a10 bytes) 000646 1786831370.697349 mapwrite 0x01f9bb thru 0x02bea1 (0xc4e7 bytes) 000647 1786831370.771193 mapread 0x01a0d8 thru 0x01c132 (0x205b bytes) 000648 1786831370.772324 trunc from 0x026847 to 0x0292dc (0x2a96 bytes) 000650 1786831370.779306 mapwrite 0x039c79 thru 0x03ec1c (0x4fa4 bytes) 000651 1786831370.832090 trunc from 0x01a303 to 0x0239c8 (0x96c6 bytes) 000652 1786831370.837421 read 0x00bfd5 thru 0x0137df (0x780b bytes) 000653 1786831370.838443 fallocate 0x033e21 thru 0x03ffff (0xc1df bytes) 000654 1786831370.842856 mapwrite 0x02a9e0 thru 0x03a202 (0xf823 bytes) 000655 1786831370.863423 mapread 0x01d03d thru 0x021d52 (0x4d16 bytes) 000657 1786831370.879347 mapwrite 0x02c60e thru 0x031751 (0x5144 bytes) 000658 1786831370.907650 read 0x01760b thru 0x020a77 (0x946d bytes) 000659 1786831370.908983 trunc from 0x020297 to 0x0294ba (0x9224 bytes) 000660 1786831370.914626 mapwrite 0x02d63d thru 0x033184 (0x5b48 bytes) 000661 1786831370.933039 punch from 0x0388de to 0x03a109 (0x182c bytes) 000662 1786831370.938696 write 0x0156ad thru 0x02239a (0xccee bytes) 000663 1786831370.948935 trunc from 0x03ec1d to 0x00ba80 (0x3319c bytes) 000664 1786831370.972081 mapread 0x003871 thru 0x004bcf (0x135f bytes) 000665 1786831370.976707 mapwrite 0x02845c thru 0x02e432 (0x5fd7 bytes) 000666 1786831371.050936 write 0x01f843 thru 0x0221aa (0x2968 bytes) 000667 1786831371.053022 fallocate 0x012442 thru 0x01ed31 (0xc8f0 bytes) 000668 1786831371.062392 write 0x0142ef thru 0x0184fa (0x420c bytes) 000669 1786831371.071313 trunc from 0x02e01b to 0x02e432 (0x418 bytes) 000670 1786831371.078399 trunc from 0x02e433 to 0x00466a (0x29dc8 bytes) 000671 1786831371.096892 trunc from 0x00172f to 0x00466a (0x2f3c bytes) 000672 1786831371.102704 punch from 0x0044ed to 0x00466a (0x17e bytes) 000674 1786831371.113642 read 0x002365 thru 0x00466a (0x2306 bytes) 000675 1786831371.121337 read 0x000147 thru 0x00466a (0x4524 bytes) 000676 1786831371.123432 read 0x0027d6 thru 0x002d1b (0x546 bytes) 000677 1786831371.124226 punch from 0x00434a to 0x00466a (0x321 bytes) 000678 1786831371.128756 read 0x002662 thru 0x00466a (0x2009 bytes) 000679 1786831371.134612 fallocate 0x02ae2a thru 0x02b6b3 (0x88a bytes) 000680 1786831371.138800 trunc from 0x001ced to 0x00466a (0x297e bytes) 000681 1786831371.147077 fallocate 0x03df23 thru 0x03ffff (0x20dd bytes) 000682 1786831371.156949 punch from 0x00047d to 0x00466a (0x41ee bytes) 000683 1786831371.162638 mapread 0x002424 thru 0x00466a (0x2247 bytes) 000684 1786831371.179115 mapread 0x0036bc thru 0x00466a (0xfaf bytes) 000685 1786831371.180369 mapwrite 0x01126e thru 0x01bea5 (0xac38 bytes) 000686 1786831371.229231 mapwrite 0x010b1c thru 0x0139a3 (0x2e88 bytes) 000687 1786831371.251620 read 0x00d29a thru 0x00d630 (0x397 bytes) 000688 1786831371.252407 mapread 0x00151e thru 0x00e113 (0xcbf6 bytes) 000689 1786831371.265477 trunc from 0x014201 to 0x01bea5 (0x7ca5 bytes) 000690 1786831371.272819 trunc from 0x01bea6 to 0x037442 (0x1b59d bytes) 000691 1786831371.295438 read 0x0317c0 thru 0x037442 (0x5c83 bytes) 000692 1786831371.310357 punch from 0x005fb2 to 0x0149ea (0xea39 bytes) 000693 1786831371.317311 mapread 0x011a31 thru 0x01fc18 (0xe1e8 bytes) 000694 1786831371.342582 mapwrite 0x0279bd thru 0x02a4df (0x2b23 bytes) 000695 1786831371.360636 write 0x0188d9 thru 0x0245d7 (0xbcff bytes) 000696 1786831371.362524 mapread 0x034111 thru 0x037442 (0x3332 bytes) 000697 1786831371.365095 mapwrite 0x01ffff thru 0x02d7ee (0xd7f0 bytes) 000698 1786831371.393237 mapread 0x029316 thru 0x02bb95 (0x2880 bytes) 000700 1786831371.394888 trunc from 0x037443 to 0x03a250 (0x2e0e bytes) 000701 1786831371.413367 trunc from 0x03a251 to 0x036f83 (0x32cd bytes) 000702 1786831371.437582 trunc from 0x029ee0 to 0x032eee (0x900f bytes) 000703 1786831371.450239 mapwrite 0x02f3ed thru 0x03bd2f (0xc943 bytes) 000704 1786831371.533136 mapread 0x014377 thru 0x019093 (0x4d1d bytes) 000705 1786831371.534491 mapread 0x01c809 thru 0x026e73 (0xa66b bytes) 000707 1786831371.538222 read 0x00f1bb thru 0x01a57e (0xb3c4 bytes) 000708 1786831371.551471 write 0x00b780 thru 0x00f5c4 (0x3e45 bytes) 000709 1786831371.553463 write 0x02ae14 thru 0x03832a (0xd517 bytes) 000710 1786831371.556589 trunc from 0x031a1b to 0x038558 (0x6b3e bytes) 000711 1786831371.578391 read 0x0242e9 thru 0x029b70 (0x5888 bytes) 000712 1786831371.579768 mapwrite 0x014f25 thru 0x01de75 (0x8f51 bytes) 000713 1786831371.614490 MIRROR_EXTEND to 2 mirrors 000714 1786831372.374138 fallocate 0x039502 thru 0x03ffff (0x6afe bytes) 000715 1786831372.425926 write 0x0102d2 thru 0x01cb94 (0xc8c3 bytes) 000717 1786831372.451944 trunc from 0x03bd30 to 0x00f5f1 (0x2c73e bytes) 000718 1786831372.477801 write 0x019869 thru 0x0260da (0xc872 bytes) 000719 1786831372.484033 punch from 0x002e1a to 0x00d7cd (0xa9b4 bytes) 000720 1786831372.490094 write 0x02f9a8 thru 0x02fd2a (0x383 bytes) 000721 1786831372.491539 trunc from 0x01c263 to 0x01ceed (0xc8b bytes) 000722 1786831372.513057 write 0x00388d thru 0x0049b7 (0x112b bytes) 000723 1786831372.532019 trunc from 0x007848 to 0x00eef8 (0x76b1 bytes) 000724 1786831372.540218 write 0x0166da thru 0x0235ce (0xcef5 bytes) 000725 1786831372.549942 read 0x02a7ef thru 0x02fd2a (0x553c bytes) 000726 1786831372.566134 MIRROR_RESYNC 2 mirrors 000727 1786831373.291856 trunc from 0x0208f1 to 0x02c28c (0xb99c bytes) 000728 1786831373.333284 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000729 1786831373.471925 MIRROR_EXTEND to 2 mirrors 000730 1786831374.120982 fallocate 0x03f072 thru 0x03ffff (0xf8e bytes) 000731 1786831374.150926 trunc from 0x018f74 to 0x01dc3f (0x4ccc bytes) 000732 1786831374.158543 mapread 0x02cf58 thru 0x02ef50 (0x1ff9 bytes) 000733 1786831374.182098 trunc from 0x009f31 to 0x016972 (0xca42 bytes) 000734 1786831374.191625 trunc from 0x025993 to 0x028d55 (0x33c3 bytes) 000735 1786831374.202414 read 0x02486c thru 0x02fd2a (0xb4bf bytes) 000736 1786831374.211489 read 0x026223 thru 0x028d41 (0x2b1f bytes) 000737 1786831374.212322 mapwrite 0x031d20 thru 0x03ffff (0xe2e0 bytes) 000738 1786831374.293068 punch from 0x027c27 to 0x02c5d8 (0x49b2 bytes) 000740 1786831374.299121 read 0x002e0b thru 0x0081a3 (0x5399 bytes) 000741 1786831374.304332 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1d3:0x0]: Invalid argument (22) 000742 1786831374.447877 mapread 0x005083 thru 0x006182 (0x1100 bytes) 000743 1786831374.459451 fallocate 0x036356 thru 0x037717 (0x13c2 bytes) 000744 1786831374.463914 mapread 0x0284ac thru 0x032e65 (0xa9ba bytes) 000745 1786831374.479897 trunc from 0x00444e to 0x00e0d4 (0x9c87 bytes) 000746 1786831374.485247 trunc from 0x0052a3 to 0x012662 (0xd3c0 bytes) 000747 1786831374.490073 mapwrite 0x01e56e thru 0x02430d (0x5da0 bytes) 000749 1786831374.515561 mapread 0x006e0d thru 0x011510 (0xa704 bytes) 000752 1786831374.542340 trunc from 0x040000 to 0x01b851 (0x247ae bytes) 000753 1786831374.570338 write 0x010aa2 thru 0x010f1e (0x47d bytes) 000754 1786831374.573092 mapread 0x00525b thru 0x00ed0d (0x9ab3 bytes) 000756 1786831374.576348 punch from 0x00fb5d to 0x01b851 (0xbcf5 bytes) 000757 1786831374.590852 mapwrite 0x03f915 thru 0x03ffff (0x6eb bytes) 000758 1786831374.632331 trunc from 0x023a3f to 0x030e33 (0xd3f5 bytes) 000760 1786831374.637919 trunc from 0x001eeb to 0x00e6bb (0xc7d1 bytes) 000761 1786831374.650489 MIRROR_EXTEND to 2 mirrors 000762 1786831375.092697 trunc from 0x00b388 to 0x00bfee (0xc67 bytes) 000763 1786831375.145597 mapread 0x038399 thru 0x03f411 (0x7079 bytes) 000764 1786831375.177388 read 0x034ae5 thru 0x03ffff (0xb51b bytes) 000765 1786831375.178133 mapwrite 0x00462a thru 0x00511a (0xaf1 bytes) 000767 1786831375.207792 punch from 0x02dfc3 to 0x033994 (0x59d2 bytes) 000769 1786831375.219399 trunc from 0x040000 to 0x03a673 (0x598c bytes) 000770 1786831375.262949 trunc from 0x02c10b to 0x02dbca (0x1ac0 bytes) 000771 1786831375.267128 trunc from 0x03a674 to 0x02e783 (0xbef0 bytes) 000772 1786831375.284606 write 0x0106dd thru 0x01dcd8 (0xd5fc bytes) 000773 1786831375.292621 trunc from 0x02e784 to 0x0397c3 (0xb040 bytes) 000774 1786831375.312689 trunc from 0x0397c4 to 0x02902d (0x10796 bytes) 000775 1786831375.330235 write 0x02037a thru 0x023ded (0x3a74 bytes) 000776 1786831375.343444 write 0x025ca8 thru 0x02e67e (0x89d7 bytes) 000777 1786831375.353278 read 0x019cba thru 0x0214bf (0x7806 bytes) 000778 1786831375.368946 write 0x00a5f3 thru 0x012d70 (0x877e bytes) 000779 1786831375.371668 mapread 0x028634 thru 0x02e67e (0x604b bytes) 000780 1786831375.373053 trunc from 0x02e67f to 0x0189e4 (0x15c9a bytes) 000781 1786831375.388718 read 0x0105d3 thru 0x014d39 (0x4767 bytes) 000782 1786831375.389971 punch from 0x00658c to 0x00f041 (0x8ab6 bytes) 000783 1786831375.407264 mapwrite 0x02b578 thru 0x032955 (0x73de bytes) 000785 1786831375.463592 mapwrite 0x027dac thru 0x02ecbb (0x6f10 bytes) 000786 1786831375.481946 write 0x029896 thru 0x038fbb (0xf726 bytes) 000787 1786831375.483774 mapread 0x025ac5 thru 0x0352fa (0xf836 bytes) 000788 1786831375.485347 write 0x016064 thru 0x01aba1 (0x4b3e bytes) 000789 1786831375.486506 fallocate 0x027b1c thru 0x02a26a (0x274f bytes) 000790 1786831375.491760 trunc from 0x038fbc to 0x020e68 (0x18153 bytes) 000792 1786831375.507491 mapread 0x00f071 thru 0x0109e9 (0x1979 bytes) 000795 1786831375.510173 trunc from 0x020e69 to 0x03f891 (0x1ea29 bytes) 000796 1786831375.525910 punch from 0x00dc42 to 0x015596 (0x7955 bytes) 000797 1786831375.538390 read 0x005e2d thru 0x0095e1 (0x37b5 bytes) 000798 1786831375.539084 fallocate 0x02e73c thru 0x033532 (0x4df7 bytes) 000800 1786831375.545816 read 0x0327e6 thru 0x0361d0 (0x39eb bytes) 000801 1786831375.562136 fallocate 0x01bdcd thru 0x0277a4 (0xb9d8 bytes) 000802 1786831375.568844 read 0x01eb17 thru 0x028903 (0x9ded bytes) 000803 1786831375.570393 mapread 0x034dd6 thru 0x03a638 (0x5863 bytes) 000805 1786831375.574037 write 0x02e17d thru 0x0320ca (0x3f4e bytes) 000806 1786831375.575645 punch from 0x0272b8 to 0x033b8b (0xc8d4 bytes) 000807 1786831375.602414 trunc from 0x033231 to 0x0363b5 (0x3185 bytes) 000808 1786831375.611932 mapwrite 0x038319 thru 0x03ffff (0x7ce7 bytes) 000809 1786831375.651582 trunc from 0x040000 to 0x02524e (0x1adb1 bytes) 000810 1786831375.674028 read 0x005271 thru 0x00846d (0x31fd bytes) 000811 1786831375.674784 punch from 0x017d17 to 0x023e33 (0xc11d bytes) 000812 1786831375.684341 MIRROR_RESYNC 2 mirrors 000813 1786831376.293424 punch from 0x002ef1 to 0x010c5d (0xdd6d bytes) 000814 1786831376.323281 trunc from 0x02524f to 0x001654 (0x23bfa bytes) 000815 1786831376.349384 mapread 0x000efe thru 0x001654 (0x757 bytes) 000816 1786831376.361579 fallocate 0x02d136 thru 0x030eb2 (0x3d7d bytes) 000819 1786831376.367640 trunc from 0x0126d5 to 0x01812e (0x5a5a bytes) 000820 1786831376.372205 write 0x03a920 thru 0x03ffff (0x56e0 bytes) 000821 1786831376.374891 write 0x00e7fa thru 0x01716b (0x8972 bytes) 000822 1786831376.387293 trunc from 0x02d49d to 0x02fc41 (0x27a5 bytes) 000823 1786831376.400545 trunc from 0x01ba39 to 0x026b22 (0xb0ea bytes) 000824 1786831376.408411 write 0x01b09f thru 0x01f63b (0x459d bytes) 000825 1786831376.427543 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa1a259f7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9f44ca6c Mirror 3: 0x9f44ca6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000826 1786831377.139958 trunc from 0x040000 to 0x009f31 (0x360ce bytes) 000827 1786831377.218412 punch from 0x00377f to 0x003c74 (0x4f6 bytes) 000828 1786831377.223566 mapwrite 0x00c70b thru 0x00c852 (0x148 bytes) 000829 1786831377.273013 trunc from 0x00c1b3 to 0x00c852 (0x6a0 bytes) 000830 1786831377.281958 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xc853): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc853): Mirror 2: 0x8baaba35 (stale) Mirror 3: 0xd431ccd5 (stale) CRC-32 checksum value for chunk [0, 0xc853): Mirror 1: 0xd431ccd5 Mirror 4: 0xd431ccd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc853: skipped 000831 1786831377.982613 trunc from 0x00277f to 0x00c4bd (0x9d3f bytes) 000832 1786831378.016959 fallocate 0x0074ba thru 0x00b865 (0x43ac bytes) 000833 1786831378.023939 mapread 0x000fc2 thru 0x005e68 (0x4ea7 bytes) 000834 1786831378.057115 mapread 0x005724 thru 0x009db6 (0x4693 bytes) 000837 1786831378.062639 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xc853): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc853): Mirror 2: 0x8baaba35 (stale) Mirror 3: 0xd431ccd5 (stale) Mirror 4: 0xd431ccd5 (stale) CRC-32 checksum value for chunk [0, 0xc853): Mirror 1: 0xd431ccd5 Mirror 5: 0xd431ccd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc853: skipped 000839 1786831378.791277 mapwrite 0x001b35 thru 0x010d04 (0xf1d0 bytes) 000840 1786831378.908544 fallocate 0x0346f0 thru 0x03cae2 (0x83f3 bytes) 000841 1786831378.915273 write 0x03c14a thru 0x03ffff (0x3eb6 bytes) 000843 1786831378.926897 trunc from 0x03809b to 0x03913e (0x10a4 bytes) 000844 1786831378.935069 fallocate 0x00713b thru 0x00daa4 (0x696a bytes) 000845 1786831378.939999 read 0x002ef4 thru 0x00d5e2 (0xa6ef bytes) 000846 1786831378.941053 mapwrite 0x038866 thru 0x03ffff (0x779a bytes) 000847 1786831378.981384 read 0x02c991 thru 0x03a753 (0xddc3 bytes) 000848 1786831378.994262 punch from 0x00f410 to 0x015eaa (0x6a9b bytes) 000850 1786831379.001968 trunc from 0x02316a to 0x02d89d (0xa734 bytes) 000851 1786831379.009373 trunc from 0x03f933 to 0x03ffff (0x6cd bytes) 000852 1786831379.021002 read 0x0163f0 thru 0x0167c4 (0x3d5 bytes) 000853 1786831379.021828 mapwrite 0x03b81a thru 0x03ffff (0x47e6 bytes) 000854 1786831379.050401 mapread 0x025755 thru 0x02ad6d (0x5619 bytes) 000855 1786831379.051852 read 0x02c20d thru 0x0377d7 (0xb5cb bytes) 000856 1786831379.054621 fallocate 0x011374 thru 0x02007c (0xed09 bytes) 000857 1786831379.061921 trunc from 0x040000 to 0x014203 (0x2bdfc bytes) 000858 1786831379.088778 fallocate 0x005ac3 thru 0x00fbc0 (0xa0fe bytes) 000859 1786831379.094280 punch from 0x00a7aa to 0x00e0f8 (0x394f bytes) 000860 1786831379.102949 mapwrite 0x00da51 thru 0x00f3c4 (0x1974 bytes) 000861 1786831379.124628 trunc from 0x014204 to 0x03a1f2 (0x25fef bytes) 000862 1786831379.142229 fallocate 0x005da9 thru 0x00daa6 (0x7cfe bytes) 000863 1786831379.149687 trunc from 0x013d8d to 0x017824 (0x3a98 bytes) 000864 1786831379.155000 fallocate 0x03b95a thru 0x03ffff (0x46a6 bytes) 000865 1786831379.167087 trunc from 0x02892b to 0x03559c (0xcc72 bytes) 000867 1786831379.188345 MIRROR_RESYNC 5 mirrors 000868 1786831380.472103 MIRROR_EXTEND to 6 mirrors 000869 1786831381.258640 trunc from 0x00d0ed to 0x0160c4 (0x8fd8 bytes) 000870 1786831381.304068 punch from 0x01565f to 0x01e1ad (0x8b4f bytes) 000871 1786831381.315759 read 0x03534a thru 0x03a1f2 (0x4ea9 bytes) 000872 1786831381.333605 fallocate 0x01ae48 thru 0x01d9d8 (0x2b91 bytes) 000873 1786831381.338664 trunc from 0x03a1f3 to 0x02aec5 (0xf32d bytes) 000875 1786831381.369593 read 0x0036cf thru 0x00e6d5 (0xb007 bytes) 000876 1786831381.392523 write 0x01cea4 thru 0x021e2f (0x4f8c bytes) 000877 1786831381.393949 trunc from 0x02aec6 to 0x0064b3 (0x24a12 bytes) 000878 1786831381.435566 MIRROR_RESYNC 6 mirrors 000879 1786831382.480664 mapwrite 0x00a858 thru 0x014c84 (0xa42d bytes) 000880 1786831382.638900 trunc from 0x014c85 to 0x028c07 (0x13f83 bytes) 000881 1786831382.682559 trunc from 0x00f9ca to 0x01b611 (0xbc48 bytes) 000882 1786831382.693479 mapwrite 0x009318 thru 0x00eb8c (0x5875 bytes) 000883 1786831382.726256 fallocate 0x0374ed thru 0x03ffff (0x8b13 bytes) 000884 1786831382.740931 trunc from 0x02474f to 0x028c07 (0x44b9 bytes) 000886 1786831382.751906 fallocate 0x03fea1 thru 0x03ffff (0x15f bytes) 000887 1786831382.760537 read 0x011a36 thru 0x0190ee (0x76b9 bytes) 000888 1786831382.777048 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x28c08): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c08): Mirror 2: 0x19e5667 (stale) Mirror 3: 0x19e5667 (stale) Mirror 4: 0x19e5667 (stale) Mirror 5: 0x19e5667 (stale) Mirror 6: 0x19e5667 (stale) CRC-32 checksum value for chunk [0, 0x28c08): Mirror 1: 0x576addc5 Mirror 7: 0x576addc5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c08: skipped 000889 1786831383.773074 mapread 0x00c271 thru 0x01c065 (0xfdf5 bytes) 000890 1786831383.838665 write 0x0158b0 thru 0x01a486 (0x4bd7 bytes) 000891 1786831383.907687 fallocate 0x01baa1 thru 0x027c1f (0xc17f bytes) 000892 1786831383.924487 mapread 0x028617 thru 0x028c07 (0x5f1 bytes) 000893 1786831383.940959 trunc from 0x0228a2 to 0x024c32 (0x2391 bytes) 000894 1786831383.960381 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x28c08): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c08): Mirror 2: 0x19e5667 (stale) Mirror 3: 0x19e5667 (stale) Mirror 4: 0x19e5667 (stale) Mirror 5: 0x19e5667 (stale) Mirror 6: 0x19e5667 (stale) Mirror 7: 0x576addc5 (stale) CRC-32 checksum value for chunk [0, 0x28c08): Mirror 1: 0x69064f03 Mirror 8: 0x69064f03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c08: skipped 000897 1786831385.142482 trunc from 0x028c08 to 0x00ac27 (0x1dfe0 bytes) 000898 1786831385.212520 read 0x000edd thru 0x00ac27 (0x9d4b bytes) 000900 1786831385.227982 mapread 0x008dfd thru 0x00ac27 (0x1e2b bytes) 000901 1786831385.229057 trunc from 0x00ac28 to 0x00014f (0xaad8 bytes) 000902 1786831385.250966 trunc from 0x000035 to 0x00014f (0x11b bytes) 000903 1786831385.257059 read 0x000046 thru 0x00014f (0x10a bytes) 000904 1786831385.263033 fallocate 0x03b122 thru 0x03f889 (0x4768 bytes) 000905 1786831385.277353 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f88a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f88a): Mirror 2: 0x4b80dadb (stale) Mirror 3: 0x4b80dadb (stale) Mirror 4: 0x4b80dadb (stale) Mirror 5: 0x4b80dadb (stale) Mirror 6: 0x4b80dadb (stale) Mirror 7: 0xe14a244d (stale) Mirror 8: 0x4fcabd64 (stale) CRC-32 checksum value for chunk [0, 0x3f88a): Mirror 1: 0xe25dc8dc Mirror 9: 0xe25dc8dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f88a: skipped 000906 1786831386.241215 trunc from 0x03f88a to 0x02db2e (0x11d5b bytes) 000907 1786831386.309275 mapread 0x001aa3 thru 0x0023e5 (0x943 bytes) 000908 1786831386.323079 fallocate 0x01c072 thru 0x02bef1 (0xfe80 bytes) 000909 1786831386.329855 trunc from 0x02db2f to 0x01dc8d (0xfea1 bytes) 000910 1786831386.357228 mapwrite 0x0363e2 thru 0x036930 (0x54f bytes) 000911 1786831386.414486 mapread 0x030b63 thru 0x031c62 (0x1100 bytes) 000912 1786831386.415609 read 0x01459e thru 0x024270 (0xfcd3 bytes) 000913 1786831386.424802 read 0x02974d thru 0x032658 (0x8f0c bytes) 000914 1786831386.450995 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x36931): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36931): Mirror 2: 0x598e2844 (stale) Mirror 3: 0x598e2844 (stale) Mirror 4: 0x598e2844 (stale) Mirror 5: 0x598e2844 (stale) Mirror 6: 0x598e2844 (stale) Mirror 7: 0x635e1c70 (stale) Mirror 8: 0xc7ffa46b (stale) Mirror 9: 0xe2ab8eaf (stale) CRC-32 checksum value for chunk [0, 0x36931): Mirror 1: 0x2cd38d31 Mirror 10: 0x2cd38d31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36931: skipped 000915 1786831387.296528 fallocate 0x01bb20 thru 0x022704 (0x6be5 bytes) 000916 1786831387.354809 punch from 0x012e9e to 0x01945e (0x65c1 bytes) 000917 1786831387.369309 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x36931): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36931): Mirror 2: 0x598e2844 (stale) Mirror 3: 0x598e2844 (stale) Mirror 4: 0x598e2844 (stale) Mirror 5: 0x598e2844 (stale) Mirror 6: 0x598e2844 (stale) Mirror 7: 0x635e1c70 (stale) Mirror 8: 0xc7ffa46b (stale) Mirror 9: 0xe2ab8eaf (stale) Mirror 10: 0x2cd38d31 (stale) CRC-32 checksum value for chunk [0, 0x36931): Mirror 1: 0x2cd38d31 Mirror 11: 0x2cd38d31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36931: skipped 000918 1786831388.500908 fallocate 0x020f1a thru 0x027323 (0x640a bytes) 000920 1786831388.560164 trunc from 0x036931 to 0x033de3 (0x2b4d bytes) 000921 1786831388.584351 mapwrite 0x01ff8c thru 0x028544 (0x85b9 bytes) 000922 1786831388.622977 mapread 0x01a29e thru 0x01e48d (0x41f0 bytes) 000923 1786831388.624248 mapwrite 0x01a9e3 thru 0x0208f4 (0x5f12 bytes) 000924 1786831388.648574 read 0x028ab6 thru 0x033de3 (0xb32e bytes) 000925 1786831388.667004 MIRROR_RESYNC 11 mirrors 000926 1786831390.776173 trunc from 0x0239b2 to 0x0318e4 (0xdf33 bytes) 000927 1786831390.832636 fallocate 0x00309b thru 0x00a665 (0x75cb bytes) 000928 1786831390.838279 fallocate 0x032354 thru 0x0362f4 (0x3fa1 bytes) 000929 1786831390.850133 mapread 0x02e73a thru 0x03526e (0x6b35 bytes) 000930 1786831390.875599 trunc from 0x0362f5 to 0x01f81a (0x16ada bytes) 000931 1786831390.899254 trunc from 0x01f81b to 0x02f3e0 (0xfbc6 bytes) 000932 1786831390.924883 MIRROR_RESYNC 11 mirrors 000933 1786831392.933632 write 0x015ff0 thru 0x02526e (0xf27f bytes) 000934 1786831392.976561 mapread 0x00d839 thru 0x010768 (0x2f30 bytes) 000935 1786831393.004779 MIRROR_RESYNC 11 mirrors 000936 1786831395.078664 trunc from 0x006442 to 0x015701 (0xf2c0 bytes) 000937 1786831395.147497 fallocate 0x00adfc thru 0x011180 (0x6385 bytes) 000939 1786831395.156295 mapread 0x0012a6 thru 0x00cfd7 (0xbd32 bytes) 000940 1786831395.175490 punch from 0x00a1e0 to 0x00ba2b (0x184c bytes) 000941 1786831395.180014 mapwrite 0x03e2ba thru 0x03ffff (0x1d46 bytes) 000942 1786831395.221327 trunc from 0x040000 to 0x035dd4 (0xa22b bytes) 000943 1786831395.238494 mapread 0x0215b1 thru 0x026d5b (0x57ab bytes) 000945 1786831395.251636 fallocate 0x00a677 thru 0x01a448 (0xfdd2 bytes) 000946 1786831395.256959 mapread 0x02c22e thru 0x035dd4 (0x9ba7 bytes) 000947 1786831395.276686 punch from 0x02765c to 0x02f28f (0x7c34 bytes) 000948 1786831395.289059 MIRROR_RESYNC 11 mirrors 000949 1786831397.249412 MIRROR_EXTEND to 12 mirrors 000950 1786831398.165376 read 0x005ab1 thru 0x013467 (0xd9b7 bytes) 000952 1786831398.180562 punch from 0x016479 to 0x018ef9 (0x2a81 bytes) 000953 1786831398.254013 fallocate 0x03aa96 thru 0x03e936 (0x3ea1 bytes) 000954 1786831398.260915 trunc from 0x035dd5 to 0x01a85b (0x1b579 bytes) 000955 1786831398.290887 trunc from 0x0013a2 to 0x00e751 (0xd3b0 bytes) 000957 1786831398.306583 MIRROR_RESYNC 12 mirrors 000958 1786831400.571608 write 0x00d3b3 thru 0x01c4ce (0xf11c bytes) 000960 1786831400.656772 mapwrite 0x026bdd thru 0x029196 (0x25ba bytes) 000961 1786831400.801381 fallocate 0x0024d8 thru 0x0093cc (0x6ef5 bytes) 000963 1786831400.808994 trunc from 0x01d6f5 to 0x029196 (0xbaa2 bytes) 000964 1786831400.814983 trunc from 0x029197 to 0x0171e1 (0x11fb5 bytes) 000965 1786831400.834421 mapread 0x009aef thru 0x013de5 (0xa2f7 bytes) 000966 1786831400.865297 mapread 0x010462 thru 0x0171e1 (0x6d80 bytes) 000967 1786831400.867675 mapread 0x00a5f3 thru 0x00efb9 (0x49c7 bytes) 000969 1786831400.869790 mapread 0x001ce3 thru 0x0030e0 (0x13fe bytes) 000971 1786831400.888985 trunc from 0x0171e2 to 0x019695 (0x24b4 bytes) 000972 1786831400.925278 MIRROR_RESYNC 12 mirrors 000973 1786831403.176034 mapwrite 0x00eb7d thru 0x01b14a (0xc5ce bytes) 000974 1786831403.273399 punch from 0x0198e1 to 0x01b14a (0x186a bytes) 000975 1786831403.280262 trunc from 0x01058f to 0x01b14a (0xabbc bytes) 000976 1786831403.286876 read 0x0013c2 thru 0x00d12d (0xbd6c bytes) 000977 1786831403.295462 trunc from 0x01b14b to 0x025d4a (0xac00 bytes) 000978 1786831403.312693 trunc from 0x025d4b to 0x03cd88 (0x1703e bytes) 000979 1786831403.338225 MIRROR_RESYNC 12 mirrors 000980 1786831405.992706 trunc from 0x03cd89 to 0x019211 (0x23b77 bytes) 000981 1786831406.041242 punch from 0x00f761 to 0x019211 (0x9ab1 bytes) 000983 1786831406.050390 write 0x009149 thru 0x01616f (0xd027 bytes) 000984 1786831406.070456 MIRROR_RESYNC 12 mirrors 000985 1786831408.394946 read 0x018080 thru 0x019211 (0x1192 bytes) 000989 1786831408.405083 write 0x01b965 thru 0x01d548 (0x1be4 bytes) 000990 1786831408.442288 mapread 0x004333 thru 0x00cb8a (0x8858 bytes) 000991 1786831408.475639 trunc from 0x000f95 to 0x004954 (0x39c0 bytes) 000993 1786831408.489357 trunc from 0x01d549 to 0x03ce42 (0x1f8fa bytes) 000994 1786831408.515842 mapwrite 0x01f1e3 thru 0x02a098 (0xaeb6 bytes) 000995 1786831408.555079 trunc from 0x02a460 to 0x02c887 (0x2428 bytes) 000996 1786831408.563287 fallocate 0x016651 thru 0x025419 (0xedc9 bytes) 000997 1786831408.569373 fallocate 0x03fee3 thru 0x03ffff (0x11d bytes) 000998 1786831408.576520 trunc from 0x025263 to 0x03356e (0xe30c bytes) 000999 1786831408.582366 punch from 0x01173c to 0x0156f6 (0x3fbb bytes) 001000 1786831408.588167 write 0x01e6c6 thru 0x01ea3f (0x37a bytes) All operations completed A-OK! PASS 70a (74s) == sanity-flr test 71: check mirror extend parent fid ==== 18:03:34 (1786831414) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1ff:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 18:03:43 (1786831423) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229451 s, 9.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118926 s, 8.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 18:03:52 (1786831432) Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre2 Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7204128 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1567296 .. failed Extending file size to 6483392 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7312992 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6951392 .. Extending file size to 7882976 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7863776 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8026720 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8316992 .. Extending file size to 8558304 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8396288 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8864256 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 61394 61395 61397 61399 61401 failed 192.168.206.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg621-client.virtnet /mnt/lustre2 (opts:) 192.168.206.121@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg621-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (75s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:05:07 (1786831507) Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre2 Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8053056 .. Extending file size to 5745920 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8269728 .. failed Extending file size to 8752928 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9010976 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8764640 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8826528 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8942944 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9273632 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 63204 63205 63207 63208 63209 192.168.206.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg621-client.virtnet /mnt/lustre2 (opts:) 192.168.206.121@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg621-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:06:24 (1786831584) Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1269032 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2212 1269092 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24924 3582096 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18912 3588108 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43836 7170204 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1269032 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2212 1269092 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24924 3582096 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18912 3588108 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43836 7170204 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112286 s, 18.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131546 s, 15.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x1c1de600 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg621-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:06:41 (1786831601) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100379 s, 20.9 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 18:06:48 (1786831608) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:06:55 (1786831615) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0617304 s, 34.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x289:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x289:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:07:02 (1786831622) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0759731 s, 27.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:07:10 (1786831630) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.064587 s, 32.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28c:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:07:41 (1786831661) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg621-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536625 s, 19.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x28b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x28b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x28c:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:08:23 (1786831703) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg621-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0918278 s, 22.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:09:08 (1786831748) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg621-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg621-server: Pool lustre.test_204f created Adding targets to pool oleg621-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg621-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0770662 s, 27.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x28d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x28d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg621-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg621-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg621-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:10:19 (1786831819) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:10:25 (1786831825) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:10:31 (1786831831) oleg621-server: Pool lustre.test_206 created Adding targets to pool oleg621-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg621-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg621-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg621-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg621-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:10:57 (1786831857) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878657 s, 11.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553972 s, 18.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x198d5968 Mirror 3: 0x198d5968 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:11:06 (1786831866) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:11:09 (1786831869) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:11:12 (1786831872) Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0462651 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x21a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1090000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1090000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1090000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x790000 Read this (4 bytes): BBBB 192.168.206.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg621-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:11:48 (1786831908) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0166222 s, 493 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:11:56 (1786831916) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599225 s, 17.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7b lmm_fid: [0x200000405:0x7b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611259 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-15 18:11:57.000000000 -0400 Modify: 2026-08-15 18:11:57.000000000 -0400 Change: 2026-08-15 18:11:58.000000000 -0400 Birth: 2026-08-15 18:11:57.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:12:03 (1786831923) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379689 s, 27.6 MB/s oleg621-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg621-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg621-server: O/280000401/d28/13c: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:12:26 (1786831946) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.167887 s, 244 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0369987 s, 111 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 15 18:12 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x80:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 15 18:12 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 18:12:33 (1786831953) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 18:12:39 (1786831959) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1672 sec =========== 18:12:48 (1786831968) === sanity-flr: start cleanup 18:12:50 (1786831970) === === sanity-flr: finish cleanup 18:12:57 (1786831977) ===