-----============= acceptance-small: sanity-flr ============----- Mon Aug 17 02:47:26 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg331-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 02:47:45 (1786949265) === oleg331-client.virtnet: executing check_config_client /mnt/lustre oleg331-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg331-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a8a58bd1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a8a58bd1800.idle_timeout=debug disable quota as required oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 02:48:23 (1786949303) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7654] == sanity-flr test 0a: lfs mirror create with -N option == 02:48:25 (1786949305) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:48:44 (1786949324) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:48:47 (1786949327) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:48:51 (1786949331) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:49:17 (1786949357) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:49:20 (1786949360) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 02:49:23 (1786949363) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8a8a58bd1800.stats=clear osc.lustre-OST0001-osc-ffff8a8a58bd1800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:49:30 (1786949370) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:49:42 (1786949382) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:49:50 (1786949390) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:49:53 (1786949393) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:50:00 (1786949400) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 02:50:09 (1786949409) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:50:15 (1786949415) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:50:18 (1786949418) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:50:24 (1786949424) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:50:33 (1786949433) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:50:40 (1786949440) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.696526 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147255 s, 7.1 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:50:49 (1786949449) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.384416 s, 51.8 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:50:57 (1786949457) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:51:05 (1786949465) Stopping /mnt/lustre-ost1 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a8a58bd1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a8a58bd1800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg331-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a8a58bd1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a8a58bd1800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg331-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (56s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:52:01 (1786949521) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.515972 s, 22.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a8a58bd1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a8a58bd1800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg331-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:52:35 (1786949555) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a8a58bd1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a8a58bd1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg331-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a8a58bd1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a8a58bd1800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg331-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (60s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:53:35 (1786949615) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.64688 s, 14.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 17 02:53 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.694s user 0m0.000s sys 0m0.181s Stopping /mnt/lustre-ost1 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a8a58bd1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a8a58bd1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.939s user 0m0.000s sys 0m0.191s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg331-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a8a58bd1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a8a58bd1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.130s user 0m0.000s sys 0m0.195s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg331-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg331-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a8a58bd1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a8a58bd1800.ost_server_uuid in FULL state after 0 sec PASS 33b (67s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:54:42 (1786949682) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:54:46 (1786949686) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (7s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:54:53 (1786949693) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 02:54:57 (1786949697) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 02:55:05 (1786949705) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.653445 s, 19.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:55:20 (1786949720) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457325 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644335 s, 16.3 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:55:29 (1786949729) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:55:38 (1786949738) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0013226 s, 774 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000813426 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0013571 s, 755 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00346841 s, 295 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 02:55:50 (1786949750) create test files with size 15731359 .. a96001c3256ccaf68e074f9caee602ad - 1ff0cc0d1b496371dea2d834f3cb0256 - 36ecdc23db9be4b182e2043a5211796b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg331-client.virtnet /mnt/lustre (opts:) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 02:56:11 (1786949771) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:56:24 (1786949784) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:56:31 (1786949791) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101235 s, 10.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329888 s, 31.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 02:56:39 (1786949799) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152418 s, 13.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.29641 s, 14.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.363721 s, 5.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 02:56:54 (1786949814) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 02:56:58 (1786949818) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:57:01 (1786949821) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339048 s, 30.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 02:57:10 (1786949830) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:57:13 (1786949833) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:57:24 (1786949844) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.351262 s, 29.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg331-client.virtnet /mnt/lustre (opts:) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:57:35 (1786949855) llite.lustre-ffff8a8a58835800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.261493 s, 40.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg331-client.virtnet /mnt/lustre (opts:) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8a8a43ae5800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:57:48 (1786949868) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (29s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:58:16 (1786949896) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:58:46 (1786949926) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 02:58:49 (1786949929) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.199375 s, 15.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.167022 s, 18.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5f2cd21b (nosync,stale) Mirror 2: 0x5f2cd21b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5589f0b0bbbc8ffe18b39d74e217c19c - mirror 1 checksum: 5b730b99d6c07d3797dc5b1f9b362ce1 - mirror 2 checksum: 5589f0b0bbbc8ffe18b39d74e217c19c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T02:58:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T02:58:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5589f0b0bbbc8ffe18b39d74e217c19c - mirror 1 checksum: 5b730b99d6c07d3797dc5b1f9b362ce1 - mirror 2 checksum: 5b730b99d6c07d3797dc5b1f9b362ce1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T02:58:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T02:58:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (8s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:58:59 (1786949939) Failing mds1 on oleg331-server Stopping /mnt/lustre-mds1 (opts:) on oleg331-server 02:59:05 (1786949945) shut down facet: mds1 facet_host: oleg331-server facet_failover_host: oleg331-server Failover mds1 to oleg331-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 02:59:37 (1786949977) targets are mounted 02:59:37 (1786949977) facet_failover done oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308585 s, 34.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (54s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:59:53 (1786949993) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:59:56 (1786949996) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:59:59 (1786949999) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0116639 s, 1.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:00:06 (1786950006) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:00:09 (1786950009) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00995157 s, 412 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0760548 s, 53.9 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:00:17 (1786950017) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:00:38 (1786950038) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:01:08 (1786950068) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:01:33 (1786950093) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:01:57 (1786950117) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 03:02:04 (1786950124) mirror create pid 54327 mirror split pid 54329 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54327 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54329 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 03:03:13 (1786950193) Seed set to 196 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1786950196.653056 write 0x00b3d8 thru 0x00cc4a (0x1873 bytes) 000005 1786950196.676425 write 0x000e03 thru 0x00776b (0x6969 bytes) 000008 1786950196.714296 write 0x03e5df thru 0x03ffff (0x1a21 bytes) 000010 1786950196.726369 mapread 0x02c3e7 thru 0x035bae (0x97c8 bytes) 000011 1786950196.810850 read 0x035892 thru 0x03aca0 (0x540f bytes) truncating to largest ever: 0x3dd7b 000012 1786950196.823518 trunc from 0x040000 to 0x03dd7a (0x2285 bytes) 000013 1786950196.911597 mapread 0x0165b3 thru 0x01e7e4 (0x8232 bytes) 000015 1786950196.950840 write 0x02a882 thru 0x0320cb (0x784a bytes) 000018 1786950196.968988 read 0x00a06c thru 0x01a04f (0xffe4 bytes) 000019 1786950197.000500 trunc from 0x03dd7b to 0x039f80 (0x3dfa bytes) 000020 1786950197.036503 trunc from 0x039f81 to 0x02e9a3 (0xb5dd bytes) 000024 1786950197.085520 read 0x002ad1 thru 0x0032a4 (0x7d4 bytes) 000027 1786950197.092595 mapwrite 0x03d459 thru 0x03ffff (0x2ba7 bytes) 000028 1786950197.310786 mapwrite 0x0070b9 thru 0x00ac99 (0x3be1 bytes) 000029 1786950197.436409 read 0x010dbc thru 0x01a3b7 (0x95fc bytes) 000031 1786950197.441574 trunc from 0x040000 to 0x005be6 (0x3a419 bytes) 000036 1786950197.567775 MIRROR_EXTEND to 2 mirrors 000037 1786950199.343305 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000038 1786950199.525356 read 0x003c98 thru 0x004f5d (0x12c6 bytes) 000040 1786950199.538948 write 0x00b024 thru 0x00c728 (0x1705 bytes) 000041 1786950199.583427 write 0x00fadd thru 0x018808 (0x8d2c bytes) 000042 1786950199.588504 trunc from 0x018809 to 0x019a7a (0x1272 bytes) 000044 1786950199.618857 MIRROR_EXTEND to 2 mirrors 000046 1786950200.359366 write 0x03577a thru 0x03fa0a (0xa291 bytes) 000047 1786950200.446406 trunc from 0x03fa0b to 0x002a8d (0x3cf7d bytes) 000049 1786950200.502303 mapwrite 0x0126bf thru 0x01caac (0xa3ee bytes) 000050 1786950200.710474 mapwrite 0x03c372 thru 0x03ffff (0x3c8e bytes) 000051 1786950200.822065 read 0x0238f2 thru 0x02d7ef (0x9efe bytes) 000053 1786950200.858312 MIRROR_RESYNC 2 mirrors 000056 1786950202.527310 mapread 0x00ba70 thru 0x013a04 (0x7f95 bytes) 000057 1786950202.564311 mapread 0x0187e6 thru 0x01aaa3 (0x22be bytes) 000059 1786950202.595067 trunc from 0x040000 to 0x0213ab (0x1ec54 bytes) 000060 1786950202.763159 trunc from 0x0213ac to 0x02c568 (0xb1bd bytes) 000063 1786950202.827999 write 0x0161c8 thru 0x023402 (0xd23b bytes) 000064 1786950202.850748 mapwrite 0x024fdf thru 0x0299c7 (0x49e9 bytes) 000065 1786950202.961808 read 0x003ca0 thru 0x0046f2 (0xa53 bytes) 000067 1786950202.987477 trunc from 0x02c569 to 0x02b1a4 (0x13c4 bytes) 000068 1786950203.045041 read 0x00409f thru 0x00d362 (0x92c4 bytes) 000073 1786950203.098085 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000074 1786950203.431828 MIRROR_EXTEND to 2 mirrors 000075 1786950204.753299 MIRROR_EXTEND to 3 mirrors truncating to largest ever: 0x3e5db 000079 1786950205.757856 trunc from 0x02b1a5 to 0x03e5da (0x13436 bytes) 000080 1786950205.866508 mapread 0x00e71f thru 0x017837 (0x9119 bytes) 000082 1786950205.894953 read 0x03cccc thru 0x03e5da (0x190f bytes) 000083 1786950205.904187 write 0x013482 thru 0x01a510 (0x708f bytes) 000084 1786950205.911425 read 0x00677d thru 0x015d48 (0xf5cc bytes) 000085 1786950205.923783 mapread 0x012574 thru 0x01b232 (0x8cbf bytes) 000086 1786950205.925280 mapwrite 0x00d7e8 thru 0x013094 (0x58ad bytes) 000089 1786950206.011171 trunc from 0x03e5db to 0x001f07 (0x3c6d3 bytes) 000090 1786950206.028077 read 0x00088e thru 0x001f07 (0x167a bytes) 000091 1786950206.029199 mapwrite 0x002343 thru 0x009700 (0x73be bytes) 000092 1786950206.137905 trunc from 0x009701 to 0x03441a (0x2ad1a bytes) 000093 1786950206.185904 trunc from 0x03441b to 0x01a213 (0x1a207 bytes) 000095 1786950206.225693 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a214): Mirror 2: 0xce67af01 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a214): Mirror 2: 0xce67af01 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a214: skipped 000096 1786950206.817725 write 0x022f70 thru 0x02e59c (0xb62d bytes) 000097 1786950206.824655 read 0x01ff6b thru 0x02cead (0xcf43 bytes) 000098 1786950206.850818 mapwrite 0x01b27b thru 0x026601 (0xb387 bytes) 000099 1786950207.006843 MIRROR_RESYNC 2 mirrors 000100 1786950208.644005 read 0x00eb10 thru 0x011dc0 (0x32b1 bytes) 000101 1786950208.651498 trunc from 0x02e59d to 0x02b2be (0x32de bytes) 000102 1786950208.799662 mapwrite 0x03b3eb thru 0x03c7c5 (0x13db bytes) 000103 1786950208.985959 mapwrite 0x02083f thru 0x02a6b5 (0x9e77 bytes) 000104 1786950209.152635 trunc from 0x03c7c6 to 0x034468 (0x835d bytes) 000105 1786950209.195234 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000106 1786950209.619793 write 0x03a01b thru 0x03febc (0x5ea2 bytes) 000107 1786950209.622671 read 0x036a7a thru 0x039e31 (0x33b8 bytes) 000108 1786950209.630441 mapwrite 0x025936 thru 0x02b869 (0x5f34 bytes) 000111 1786950209.682928 read 0x025f09 thru 0x02f53b (0x9633 bytes) 000117 1786950209.687015 mapwrite 0x03cac0 thru 0x03ffff (0x3540 bytes) 000118 1786950209.739388 mapread 0x0108a0 thru 0x01743f (0x6ba0 bytes) 000125 1786950209.780941 mapread 0x039c1b thru 0x03e6b7 (0x4a9d bytes) 000128 1786950209.786896 read 0x038d20 thru 0x03e58e (0x586f bytes) 000131 1786950209.794447 mapwrite 0x00c553 thru 0x012fd7 (0x6a85 bytes) 000135 1786950209.906731 write 0x0202ee thru 0x021b54 (0x1867 bytes) 000136 1786950209.908197 trunc from 0x040000 to 0x0290cc (0x16f33 bytes) 000138 1786950209.936694 mapwrite 0x01be53 thru 0x0225d9 (0x6787 bytes) 000139 1786950210.026417 mapread 0x01abea thru 0x0235b9 (0x89d0 bytes) 000140 1786950210.030174 mapread 0x01c2e8 thru 0x01ca00 (0x719 bytes) 000141 1786950210.034333 mapread 0x005b75 thru 0x00af25 (0x53b1 bytes) 000144 1786950210.059341 MIRROR_EXTEND to 2 mirrors 000149 1786950211.327480 mapwrite 0x028c72 thru 0x029e3f (0x11ce bytes) 000151 1786950211.515794 write 0x0036d8 thru 0x011ece (0xe7f7 bytes) 000152 1786950211.527882 write 0x03fa84 thru 0x03ffff (0x57c bytes) 000153 1786950211.529143 mapread 0x00fa5c thru 0x01e068 (0xe60d bytes) 000156 1786950211.555667 MIRROR_RESYNC 2 mirrors 000157 1786950212.732722 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000158 1786950212.912690 mapread 0x008ea0 thru 0x01724d (0xe3ae bytes) 000160 1786950212.944788 trunc from 0x040000 to 0x02a081 (0x15f7e bytes) 000161 1786950212.979574 mapread 0x0105cb thru 0x01de37 (0xd86d bytes) 000163 1786950213.000822 read 0x008641 thru 0x012934 (0xa2f4 bytes) 000164 1786950213.010396 MIRROR_EXTEND to 2 mirrors 000165 1786950213.820407 mapwrite 0x0136e3 thru 0x01d447 (0x9d65 bytes) 000168 1786950214.197164 MIRROR_RESYNC 2 mirrors 000171 1786950215.049776 read 0x02449b thru 0x028f88 (0x4aee bytes) 000174 1786950215.070044 write 0x01b6d1 thru 0x01e7ed (0x311d bytes) 000176 1786950215.155793 trunc from 0x02a082 to 0x02df07 (0x3e86 bytes) 000178 1786950215.183373 trunc from 0x02df08 to 0x026307 (0x7c00 bytes) 000179 1786950215.201955 mapread 0x004178 thru 0x00939d (0x5226 bytes) 000180 1786950215.237974 write 0x0251ff thru 0x026fdb (0x1ddd bytes) 000181 1786950215.247433 mapwrite 0x03cb84 thru 0x03ee81 (0x22fe bytes) 000187 1786950215.411016 read 0x002256 thru 0x00a2ce (0x8079 bytes) 000188 1786950215.411595 read 0x03a46c thru 0x03ee81 (0x4a16 bytes) 000189 1786950215.412184 mapwrite 0x0188ea thru 0x019c18 (0x132f bytes) 000190 1786950215.490297 mapread 0x01d674 thru 0x028841 (0xb1ce bytes) 000191 1786950215.491711 write 0x01460e thru 0x01a8e8 (0x62db bytes) 000193 1786950215.493192 read 0x036dc3 thru 0x03c9dd (0x5c1b bytes) 000194 1786950215.493813 mapread 0x018c91 thru 0x01c246 (0x35b6 bytes) 000195 1786950215.494987 mapwrite 0x0299ec thru 0x030e6d (0x7482 bytes) 000196 1786950215.604286 read 0x01e094 thru 0x02df8b (0xfef8 bytes) 000198 1786950215.606158 write 0x038839 thru 0x03cfe3 (0x47ab bytes) 000199 1786950215.614622 MIRROR_RESYNC 2 mirrors 000200 1786950216.308712 write 0x02ee15 thru 0x035194 (0x6380 bytes) 000202 1786950216.415856 mapread 0x031f69 thru 0x03ec28 (0xccc0 bytes) 000205 1786950216.484275 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ee82): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee82): Mirror 2: 0x264e1ce0 (stale) CRC-32 checksum value for chunk [0, 0x3ee82): Mirror 1: 0x6122aab4 Mirror 3: 0x6122aab4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee82: skipped 000206 1786950217.414416 MIRROR_RESYNC 3 mirrors 000208 1786950218.052003 trunc from 0x03ee82 to 0x028aa7 (0x163da bytes) 000209 1786950218.117192 read 0x006e4f thru 0x00a6e0 (0x3892 bytes) 000210 1786950218.129437 mapread 0x01387f thru 0x016e6e (0x35f0 bytes) 000211 1786950218.153469 read 0x010c4d thru 0x01903c (0x83f0 bytes) 000213 1786950218.157132 read 0x009217 thru 0x0108b7 (0x76a1 bytes) 000214 1786950218.180497 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28aa8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28aa8): Mirror 2: 0x3dae7f9 (stale) Mirror 3: 0x3dae7f9 (stale) CRC-32 checksum value for chunk [0, 0x28aa8): Mirror 1: 0x3dae7f9 Mirror 4: 0x3dae7f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28aa8: skipped 000216 1786950219.386055 read 0x008a4f thru 0x0163b4 (0xd966 bytes) 000219 1786950219.434702 read 0x006e63 thru 0x00a14b (0x32e9 bytes) 000220 1786950219.457018 mapwrite 0x02c302 thru 0x0320dd (0x5ddc bytes) 000221 1786950219.623284 trunc from 0x0320de to 0x005f3d (0x2c1a0 bytes) 000222 1786950219.661741 write 0x031c20 thru 0x03a2d6 (0x86b7 bytes) 000223 1786950219.671869 mapwrite 0x032d86 thru 0x037a2e (0x4ca9 bytes) 000225 1786950219.745266 mapread 0x022f0e thru 0x02f556 (0xc649 bytes) 000229 1786950219.915493 read 0x00d888 thru 0x00e1f3 (0x96c bytes) 000234 1786950219.956425 trunc from 0x03a2d7 to 0x021831 (0x18aa5 bytes) 000236 1786950220.011720 mapread 0x014ef8 thru 0x01e9b9 (0x9ac2 bytes) 000239 1786950220.020498 trunc from 0x021832 to 0x032c32 (0x11401 bytes) 000240 1786950220.056902 mapread 0x020807 thru 0x02998d (0x9187 bytes) 000241 1786950220.129487 trunc from 0x032c33 to 0x03b8c0 (0x8c8e bytes) 000242 1786950220.148586 write 0x0324f3 thru 0x039ae9 (0x75f7 bytes) 000244 1786950220.169755 write 0x01eedb thru 0x023362 (0x4488 bytes) 000247 1786950220.175111 mapread 0x02aec1 thru 0x0329ab (0x7aeb bytes) 000250 1786950220.247865 write 0x0074d7 thru 0x00f236 (0x7d60 bytes) 000253 1786950220.251735 trunc from 0x03b8c1 to 0x002ade (0x38de2 bytes) 000254 1786950220.281433 mapwrite 0x03d2d0 thru 0x03ffff (0x2d30 bytes) 000255 1786950220.368597 MIRROR_RESYNC 4 mirrors 000256 1786950222.303199 read 0x005541 thru 0x0133e0 (0xdea0 bytes) 000258 1786950222.317716 mapread 0x026bb3 thru 0x030edc (0xa32a bytes) 000259 1786950222.351072 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000260 1786950222.713390 mapread 0x0255b9 thru 0x028042 (0x2a8a bytes) 000262 1786950222.741327 write 0x00d079 thru 0x01758a (0xa512 bytes) 000263 1786950222.868347 read 0x02673f thru 0x03292c (0xc1ee bytes) 000265 1786950222.877875 write 0x0206e1 thru 0x02e3da (0xdcfa bytes) 000266 1786950222.884621 write 0x0325c7 thru 0x03ffff (0xda39 bytes) 000268 1786950222.893329 write 0x031b57 thru 0x03d0e1 (0xb58b bytes) 000269 1786950222.899232 mapread 0x02e856 thru 0x03831c (0x9ac7 bytes) 000271 1786950222.907299 mapread 0x00934f thru 0x014e4c (0xbafe bytes) 000272 1786950222.946694 MIRROR_RESYNC 3 mirrors 000275 1786950224.427649 mapread 0x008e4c thru 0x01406b (0xb220 bytes) 000278 1786950224.492978 mapwrite 0x01a202 thru 0x01ddf0 (0x3bef bytes) 000279 1786950224.707782 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3f77ac49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3f77ac49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000280 1786950225.036608 read 0x008b3f thru 0x0134df (0xa9a1 bytes) 000281 1786950225.048179 write 0x00c84a thru 0x013e2d (0x75e4 bytes) 000284 1786950225.051157 trunc from 0x040000 to 0x029e8c (0x16173 bytes) 000287 1786950225.069610 trunc from 0x029e8d to 0x02040c (0x9a80 bytes) 000288 1786950225.088660 mapwrite 0x039fd3 thru 0x03f39e (0x53cc bytes) 000291 1786950225.208202 MIRROR_RESYNC 2 mirrors 000293 1786950226.302247 mapwrite 0x0322e0 thru 0x03ffff (0xdd20 bytes) 000295 1786950226.530991 mapread 0x01de8c thru 0x021fd9 (0x414e bytes) 000297 1786950226.564873 read 0x031d5d thru 0x03b517 (0x97bb bytes) 000298 1786950226.574428 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7ae90a3d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x39e3979 Mirror 3: 0x39e3979 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000299 1786950227.985496 read 0x02f97c thru 0x0318a0 (0x1f25 bytes) 000300 1786950228.005230 read 0x02e097 thru 0x03d755 (0xf6bf bytes) 000303 1786950228.056752 mapwrite 0x025b05 thru 0x025fa8 (0x4a4 bytes) 000305 1786950228.291239 write 0x0102fe thru 0x0153e6 (0x50e9 bytes) 000306 1786950228.332729 read 0x0362f0 thru 0x0373e7 (0x10f8 bytes) 000307 1786950228.357785 write 0x03a909 thru 0x03ffff (0x56f7 bytes) 000308 1786950228.359005 trunc from 0x040000 to 0x00b2cf (0x34d30 bytes) 000310 1786950228.421115 MIRROR_RESYNC 3 mirrors 000311 1786950229.688177 read 0x008d28 thru 0x00b2cf (0x25a8 bytes) 000313 1786950229.707471 write 0x01d67f thru 0x02bff8 (0xe97a bytes) 000314 1786950229.878950 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bff9): Mirror 2: 0x9338f2fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bff9): Mirror 2: 0x9338f2fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bff9: skipped 000317 1786950230.426440 mapread 0x024a6a thru 0x02bff8 (0x758f bytes) 000318 1786950230.471449 trunc from 0x02bff9 to 0x02ade5 (0x1213 bytes) 000319 1786950230.497259 write 0x01972b thru 0x019914 (0x1ea bytes) 000321 1786950230.523138 mapwrite 0x03db37 thru 0x03ffff (0x24c9 bytes) 000324 1786950230.701730 write 0x00defe thru 0x018791 (0xa894 bytes) 000330 1786950230.773479 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf26d584a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2c83ea8b Mirror 3: 0x2c83ea8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000331 1786950232.370854 mapread 0x02230c thru 0x024396 (0x208b bytes) 000332 1786950232.488374 read 0x022fe2 thru 0x02cb96 (0x9bb5 bytes) 000334 1786950232.525380 read 0x03e45e thru 0x03ffff (0x1ba2 bytes) 000335 1786950232.575906 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000336 1786950233.131309 read 0x03aaef thru 0x03ffff (0x5511 bytes) 000337 1786950233.233793 MIRROR_EXTEND to 3 mirrors 000338 1786950234.227379 MIRROR_EXTEND to 4 mirrors 000339 1786950235.276054 trunc from 0x040000 to 0x03cd62 (0x329d bytes) 000340 1786950235.449702 read 0x01e9f1 thru 0x01fdfe (0x140e bytes) 000342 1786950235.475279 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 3: 0x4c1d6276 (stale) Mirror 4: 0x4c1d6276 (stale) Mirror 5: 0x4c1d6276 (stale) CRC-32 checksum value for chunk [0, 0x3cd63): Mirror 1: 0x4c1d6276 Mirror 6: 0x4c1d6276 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd63: skipped 000343 1786950237.015718 trunc from 0x03cd63 to 0x03296d (0xa3f5 bytes) 000344 1786950237.121340 MIRROR_RESYNC 5 mirrors 000345 1786950238.775385 read 0x008181 thru 0x010f2a (0x8daa bytes) 000346 1786950238.786803 mapwrite 0x018914 thru 0x01a5ee (0x1cdb bytes) 000347 1786950239.048114 trunc from 0x03296e to 0x00fa42 (0x22f2b bytes) 000348 1786950239.114164 mapread 0x00b83b thru 0x00fa42 (0x4208 bytes) 000349 1786950239.144406 MIRROR_RESYNC 5 mirrors 000350 1786950240.639693 MIRROR_RESYNC 5 mirrors 000352 1786950241.020018 mapread 0x00d689 thru 0x00ef54 (0x18cc bytes) 000353 1786950241.030494 mapwrite 0x016f69 thru 0x02569f (0xe737 bytes) 000355 1786950241.332124 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x256a0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x256a0): Mirror 3: 0x37ca507 (stale) Mirror 4: 0x37ca507 (stale) Mirror 5: 0x37ca507 (stale) Mirror 6: 0x37ca507 (stale) CRC-32 checksum value for chunk [0, 0x256a0): Mirror 1: 0x12890ffb Mirror 7: 0x12890ffb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x256a0: skipped 000356 1786950242.731900 read 0x023aec thru 0x02569f (0x1bb4 bytes) 000357 1786950242.752705 mapwrite 0x02cf58 thru 0x034bc0 (0x7c69 bytes) 000359 1786950243.024372 trunc from 0x034bc1 to 0x0260fa (0xeac6 bytes) 000361 1786950243.054011 write 0x0239d3 thru 0x03286f (0xee9d bytes) 000363 1786950243.066681 read 0x001157 thru 0x003762 (0x260c bytes) 000366 1786950243.078982 write 0x03e766 thru 0x03ffff (0x189a bytes) 000367 1786950243.080945 read 0x013564 thru 0x0141d3 (0xc70 bytes) 000368 1786950243.088405 MIRROR_RESYNC 6 mirrors 000369 1786950244.608434 mapwrite 0x00c363 thru 0x01aba7 (0xe845 bytes) 000371 1786950244.786662 write 0x031624 thru 0x03ffff (0xe9dc bytes) 000373 1786950244.794118 read 0x03f3af thru 0x03ffff (0xc51 bytes) 000374 1786950244.798488 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xed4fb9e (stale) Mirror 5: 0xed4fb9e (stale) Mirror 6: 0xed4fb9e (stale) Mirror 7: 0xed4fb9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xed4fb9e (stale) Mirror 5: 0xed4fb9e (stale) Mirror 6: 0xed4fb9e (stale) Mirror 7: 0xed4fb9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000376 1786950245.308603 write 0x03e5be thru 0x03ffff (0x1a42 bytes) 000377 1786950245.316298 mapread 0x00c70f thru 0x016e56 (0xa748 bytes) 000378 1786950245.352683 MIRROR_RESYNC 5 mirrors 000379 1786950246.646050 write 0x000a30 thru 0x0045f4 (0x3bc5 bytes) 000380 1786950246.766850 mapread 0x03fb76 thru 0x03ffff (0x48a bytes) 000381 1786950246.790635 read 0x0387ed thru 0x03ffff (0x7813 bytes) 000382 1786950246.858605 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x14bd5b3e (stale) Mirror 5: 0x14bd5b3e (stale) Mirror 6: 0x14bd5b3e (stale) Mirror 7: 0x14bd5b3e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa8d5fca8 Mirror 8: 0xa8d5fca8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000383 1786950248.126819 mapread 0x0229b0 thru 0x027fab (0x55fc bytes) 000386 1786950248.213521 read 0x00472a thru 0x009e66 (0x573d bytes) 000387 1786950248.240952 read 0x038698 thru 0x03ffff (0x7968 bytes) 000391 1786950248.303500 trunc from 0x040000 to 0x00059b (0x3fa64 bytes) 000392 1786950248.503073 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x59c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x59c): Mirror 4: 0xdd81a5ef (stale) Mirror 5: 0xdd81a5ef (stale) Mirror 6: 0xdd81a5ef (stale) Mirror 7: 0xdd81a5ef (stale) Mirror 8: 0xdd81a5ef (stale) CRC-32 checksum value for chunk [0, 0x59c): Mirror 1: 0xdd81a5ef Mirror 9: 0xdd81a5ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x59c: skipped 000394 1786950249.619700 mapread 0x00048b thru 0x00059b (0x111 bytes) 000395 1786950249.663316 read 0x00048c thru 0x00059b (0x110 bytes) 000404 1786950249.773885 trunc from 0x00059c to 0x024683 (0x240e8 bytes) 000405 1786950249.936298 mapwrite 0x01bcb3 thru 0x022c85 (0x6fd3 bytes) 000406 1786950250.017664 mapwrite 0x030211 thru 0x032800 (0x25f0 bytes) 000407 1786950250.101958 read 0x016b50 thru 0x024e8a (0xe33b bytes) 000409 1786950250.104639 write 0x02874b thru 0x02b541 (0x2df7 bytes) 000411 1786950250.116848 write 0x03f8f0 thru 0x03ffff (0x710 bytes) 000412 1786950250.117870 trunc from 0x040000 to 0x008e57 (0x371a8 bytes) 000413 1786950250.141359 trunc from 0x008e58 to 0x01fca3 (0x16e4c bytes) 000414 1786950250.161379 trunc from 0x01fca4 to 0x00f538 (0x1076b bytes) 000415 1786950250.186963 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf539): Mirror 4: 0x63092061 (stale) Mirror 5: 0x63092061 (stale) Mirror 6: 0x63092061 (stale) Mirror 8: 0xa456b03 (stale) Mirror 9: 0xb108b263 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf539): Mirror 4: 0x63092061 (stale) Mirror 5: 0x63092061 (stale) Mirror 6: 0x63092061 (stale) Mirror 8: 0xa456b03 (stale) Mirror 9: 0xb108b263 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf539: skipped 000417 1786950250.879496 write 0x004043 thru 0x0085d1 (0x458f bytes) 000418 1786950250.900745 MIRROR_RESYNC 6 mirrors 000420 1786950252.353642 write 0x027fbe thru 0x030b6a (0x8bad bytes) 000421 1786950252.475861 trunc from 0x030b6b to 0x03591c (0x4db2 bytes) 000422 1786950252.503892 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3591d): Mirror 5: 0x51748514 (stale) Mirror 6: 0x51748514 (stale) Mirror 8: 0x51748514 (stale) Mirror 9: 0x51748514 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3591d): Mirror 5: 0x51748514 (stale) Mirror 6: 0x51748514 (stale) Mirror 8: 0x51748514 (stale) Mirror 9: 0x51748514 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3591d: skipped 000423 1786950253.040811 mapread 0x0063b0 thru 0x010e6d (0xaabe bytes) 000427 1786950253.072182 write 0x003381 thru 0x00f794 (0xc414 bytes) 000428 1786950253.073998 mapwrite 0x008fc2 thru 0x016214 (0xd253 bytes) 000433 1786950253.196661 MIRROR_RESYNC 5 mirrors 000434 1786950254.896206 write 0x003ce4 thru 0x004ac6 (0xde3 bytes) 000435 1786950254.995045 trunc from 0x03591d to 0x002f7b (0x329a1 bytes) 000438 1786950255.044632 write 0x00a006 thru 0x00d95b (0x3956 bytes) 000439 1786950255.053240 MIRROR_RESYNC 5 mirrors 000440 1786950255.966287 trunc from 0x00d95c to 0x0044b6 (0x94a5 bytes) 000446 1786950256.045936 read 0x004157 thru 0x0044b6 (0x360 bytes) 000447 1786950256.052238 mapread 0x001143 thru 0x0044b6 (0x3374 bytes) 000448 1786950256.073002 mapread 0x002b86 thru 0x0044b6 (0x1931 bytes) 000449 1786950256.079659 MIRROR_RESYNC 5 mirrors 000450 1786950257.328029 mapread 0x003a0e thru 0x0044b6 (0xaa9 bytes) 000453 1786950257.338420 read 0x00230f thru 0x0039ae (0x16a0 bytes) 000454 1786950257.347467 MIRROR_RESYNC 5 mirrors 000456 1786950257.643737 mapwrite 0x012972 thru 0x0191ca (0x6859 bytes) 000458 1786950257.809430 read 0x012afd thru 0x0191ca (0x66ce bytes) 000459 1786950257.810467 mapread 0x005289 thru 0x0149d0 (0xf748 bytes) 000460 1786950257.836290 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x191cb): Mirror 6: 0xae492efa (stale) Mirror 8: 0xae492efa (stale) Mirror 9: 0xae492efa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x191cb): Mirror 6: 0xae492efa (stale) Mirror 8: 0xae492efa (stale) Mirror 9: 0xae492efa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x191cb: skipped 000462 1786950258.275610 read 0x017e81 thru 0x0191ca (0x134a bytes) 000463 1786950258.286729 mapwrite 0x014ae7 thru 0x01c30b (0x7825 bytes) 000466 1786950258.368722 write 0x01cf9a thru 0x01ffb8 (0x301f bytes) 000467 1786950258.371232 mapwrite 0x026914 thru 0x02b4e5 (0x4bd2 bytes) 000469 1786950258.528370 write 0x029dc8 thru 0x02fba6 (0x5ddf bytes) 000470 1786950258.529704 mapwrite 0x03a4d0 thru 0x03de7c (0x39ad bytes) 000471 1786950258.687096 mapread 0x00e099 thru 0x01da30 (0xf998 bytes) 000472 1786950258.709924 read 0x01c4e5 thru 0x01cb2f (0x64b bytes) 000475 1786950258.712598 write 0x015770 thru 0x01afd0 (0x5861 bytes) 000476 1786950258.714076 trunc from 0x03de7d to 0x01b804 (0x22678 bytes) 000477 1786950258.735364 trunc from 0x01b805 to 0x0217c9 (0x5fc5 bytes) 000478 1786950258.760009 write 0x03e335 thru 0x03ffff (0x1ccb bytes) 000479 1786950258.761708 mapwrite 0x03f368 thru 0x03ffff (0xc98 bytes) 000481 1786950258.881917 mapwrite 0x03920d thru 0x03ffff (0x6df3 bytes) 000482 1786950258.982353 trunc from 0x040000 to 0x02ad7c (0x15283 bytes) 000483 1786950258.997828 mapwrite 0x008a04 thru 0x00e057 (0x5654 bytes) 000485 1786950259.035866 mapread 0x0055f3 thru 0x00e8a8 (0x92b6 bytes) 000487 1786950259.039140 read 0x020256 thru 0x02ad7c (0xab27 bytes) 000488 1786950259.044492 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ad7d): Mirror 8: 0x6aeddd2f (stale) Mirror 9: 0x6aeddd2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad7d): Mirror 8: 0x6aeddd2f (stale) Mirror 9: 0x6aeddd2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad7d: skipped 000489 1786950259.326826 mapwrite 0x0190f0 thru 0x027643 (0xe554 bytes) 000490 1786950259.411644 read 0x01af56 thru 0x02586f (0xa91a bytes) 000491 1786950259.412889 write 0x01b781 thru 0x02445b (0x8cdb bytes) 000492 1786950259.422434 MIRROR_RESYNC 3 mirrors 000494 1786950260.630226 trunc from 0x02ad7d to 0x035a35 (0xacb9 bytes) 000495 1786950260.757612 write 0x03ed73 thru 0x03ffff (0x128d bytes) 000497 1786950260.759313 mapread 0x01f741 thru 0x026da5 (0x7665 bytes) 000500 1786950260.821858 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x24e67167 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x24e67167 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000501 1786950261.213661 write 0x00163d thru 0x0079c4 (0x6388 bytes) 000506 1786950261.237872 read 0x02dc45 thru 0x02dd77 (0x133 bytes) 000507 1786950261.243901 mapwrite 0x01d276 thru 0x022c1f (0x59aa bytes) 000508 1786950261.316867 mapread 0x02949b thru 0x02b3e4 (0x1f4a bytes) 000509 1786950261.330349 read 0x032fb6 thru 0x03cd68 (0x9db3 bytes) 000510 1786950261.355433 read 0x03c1e4 thru 0x03ffff (0x3e1c bytes) 000512 1786950261.358323 write 0x002844 thru 0x00cb35 (0xa2f2 bytes) 000513 1786950261.365350 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000515 1786950261.558282 mapread 0x02ec1f thru 0x0371e4 (0x85c6 bytes) 000516 1786950261.589001 read 0x01fe82 thru 0x021378 (0x14f7 bytes) 000517 1786950261.612534 MIRROR_EXTEND to 2 mirrors 000518 1786950262.766202 trunc from 0x040000 to 0x004d62 (0x3b29d bytes) 000519 1786950262.883706 read 0x00250e thru 0x004d62 (0x2855 bytes) 000520 1786950262.914958 MIRROR_RESYNC 2 mirrors 000522 1786950263.783725 mapwrite 0x00042c thru 0x008ab6 (0x868b bytes) 000526 1786950264.089262 mapread 0x005fe8 thru 0x008ab6 (0x2acf bytes) 000527 1786950264.090757 read 0x006db5 thru 0x008ab6 (0x1d02 bytes) 000529 1786950264.098265 write 0x0307d4 thru 0x039459 (0x8c86 bytes) 000530 1786950264.102243 write 0x001257 thru 0x00f591 (0xe33b bytes) 000533 1786950264.125258 write 0x028d1c thru 0x029776 (0xa5b bytes) 000534 1786950264.163175 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000537 1786950264.444661 mapwrite 0x00abfa thru 0x017ec1 (0xd2c8 bytes) 000540 1786950264.600738 mapread 0x0169e8 thru 0x019630 (0x2c49 bytes) 000545 1786950264.627966 mapwrite 0x015497 thru 0x016808 (0x1372 bytes) 000546 1786950264.761824 trunc from 0x03945a to 0x013c40 (0x25819 bytes) 000548 1786950264.799040 trunc from 0x013c41 to 0x01e47c (0xa83c bytes) 000549 1786950264.831392 write 0x01407c thru 0x018d55 (0x4cda bytes) 000552 1786950264.847987 mapwrite 0x03a150 thru 0x03ffff (0x5eb0 bytes) 000557 1786950264.965190 trunc from 0x040000 to 0x00fd6f (0x30290 bytes) 000558 1786950265.019152 write 0x00b5c2 thru 0x013e5d (0x889c bytes) 000559 1786950265.020494 write 0x03ad5b thru 0x03ffff (0x52a5 bytes) 000560 1786950265.027372 mapread 0x00e5a8 thru 0x01aae0 (0xc539 bytes) 000561 1786950265.055459 trunc from 0x040000 to 0x0243fa (0x1bc05 bytes) 000562 1786950265.082605 read 0x00e9bb thru 0x0155a2 (0x6be8 bytes) 000564 1786950265.085389 write 0x00479f thru 0x006289 (0x1aeb bytes) 000565 1786950265.091525 trunc from 0x0243fb to 0x0387d2 (0x143d8 bytes) 000567 1786950265.119050 mapwrite 0x01d206 thru 0x024717 (0x7512 bytes) 000574 1786950265.206959 write 0x008e9a thru 0x012422 (0x9589 bytes) 000582 1786950265.216887 mapwrite 0x01a1ca thru 0x02903e (0xee75 bytes) 000584 1786950265.300894 read 0x01eb63 thru 0x020b31 (0x1fcf bytes) 000585 1786950265.302454 read 0x0071f1 thru 0x01391f (0xc72f bytes) 000587 1786950265.305062 write 0x024297 thru 0x02db2a (0x9894 bytes) 000592 1786950265.318398 trunc from 0x0387d3 to 0x024d73 (0x13a5f bytes) 000593 1786950265.338442 trunc from 0x024d74 to 0x01b12f (0x9c44 bytes) 000596 1786950265.367510 write 0x016827 thru 0x01b44a (0x4c24 bytes) 000598 1786950265.374352 mapread 0x00cdff thru 0x01a918 (0xdb1a bytes) 000599 1786950265.375667 mapwrite 0x00587f thru 0x0096cc (0x3e4e bytes) 000600 1786950265.459229 MIRROR_EXTEND to 2 mirrors 000604 1786950266.227784 MIRROR_EXTEND to 3 mirrors 000606 1786950266.902399 write 0x015b88 thru 0x01bb24 (0x5f9d bytes) 000607 1786950267.009109 mapwrite 0x01b4d2 thru 0x024154 (0x8c83 bytes) 000609 1786950267.115433 read 0x0138ce thru 0x018ca8 (0x53db bytes) 000610 1786950267.124415 read 0x00c120 thru 0x01344a (0x732b bytes) 000611 1786950267.135280 trunc from 0x024155 to 0x03584e (0x116fa bytes) 000612 1786950267.157047 mapread 0x019f13 thru 0x01c043 (0x2131 bytes) 000613 1786950267.158163 read 0x003e16 thru 0x009fd2 (0x61bd bytes) 000614 1786950267.159823 write 0x03d1b5 thru 0x03db4f (0x99b bytes) 000615 1786950267.161353 mapwrite 0x02d791 thru 0x039039 (0xb8a9 bytes) 000617 1786950267.223588 write 0x00c729 thru 0x0120f7 (0x59cf bytes) 000618 1786950267.226351 read 0x006118 thru 0x009c70 (0x3b59 bytes) 000620 1786950267.230938 mapread 0x014435 thru 0x01b21b (0x6de7 bytes) 000621 1786950267.236637 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db50): Mirror 3: 0xf6a87ac1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db50): Mirror 3: 0xf6a87ac1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db50: skipped 000627 1786950267.506520 mapwrite 0x012ef7 thru 0x01ca53 (0x9b5d bytes) 000629 1786950267.569793 trunc from 0x03db50 to 0x02f5b3 (0xe59c bytes) 000630 1786950267.588278 mapread 0x025246 thru 0x029aa2 (0x485d bytes) 000634 1786950267.609241 mapwrite 0x02e7a0 thru 0x03a4c9 (0xbd2a bytes) 000636 1786950267.699911 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000638 1786950267.882848 read 0x0242df thru 0x027451 (0x3173 bytes) 000639 1786950267.896267 write 0x012beb thru 0x015150 (0x2566 bytes) 000641 1786950267.925283 read 0x008e56 thru 0x01885b (0xfa06 bytes) 000642 1786950267.953740 trunc from 0x03a4ca to 0x00dc80 (0x2c849 bytes) 000643 1786950267.998356 mapwrite 0x0296f2 thru 0x02b88b (0x219a bytes) 000644 1786950268.161150 trunc from 0x02b88c to 0x013342 (0x18549 bytes) 000648 1786950268.210914 mapwrite 0x00a002 thru 0x01399b (0x999a bytes) 000651 1786950268.414149 trunc from 0x01399c to 0x03e161 (0x2a7c6 bytes) 000652 1786950268.443188 read 0x0152fe thru 0x01e3b2 (0x90b5 bytes) 000653 1786950268.471257 trunc from 0x03e162 to 0x01faf4 (0x1e66d bytes) 000654 1786950268.501761 MIRROR_EXTEND to 2 mirrors 000655 1786950269.473612 trunc from 0x01faf5 to 0x02ef50 (0xf45c bytes) 000656 1786950269.580327 MIRROR_RESYNC 2 mirrors 000657 1786950270.587022 mapwrite 0x03cb2f thru 0x03ffff (0x34d1 bytes) 000663 1786950270.736004 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa453d6e1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x151d96f0 Mirror 3: 0x151d96f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000664 1786950272.178531 trunc from 0x040000 to 0x006e3c (0x391c3 bytes) 000665 1786950272.335292 trunc from 0x006e3d to 0x01b8b0 (0x14a74 bytes) 000666 1786950272.356331 write 0x023e64 thru 0x02f851 (0xb9ee bytes) 000667 1786950272.360143 write 0x03373f thru 0x03ffff (0xc8c1 bytes) 000668 1786950272.363207 read 0x0253af thru 0x027f70 (0x2bc2 bytes) 000670 1786950272.365815 trunc from 0x040000 to 0x000120 (0x3fedf bytes) 000672 1786950272.387127 write 0x034694 thru 0x03ffff (0xb96c bytes) 000673 1786950272.390502 mapwrite 0x020e98 thru 0x02b033 (0xa19c bytes) 000675 1786950272.523641 mapwrite 0x001a66 thru 0x00468c (0x2c27 bytes) 000677 1786950272.641185 mapwrite 0x01b859 thru 0x02017a (0x4922 bytes) 000679 1786950272.752650 write 0x007839 thru 0x00d318 (0x5ae0 bytes) 000681 1786950272.759688 trunc from 0x040000 to 0x00d1e4 (0x32e1b bytes) 000683 1786950272.789042 mapread 0x009205 thru 0x00d1e4 (0x3fe0 bytes) 000685 1786950272.806492 write 0x0070c8 thru 0x00764d (0x586 bytes) 000688 1786950272.817758 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd1e5): Mirror 3: 0x700e950d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd1e5): Mirror 3: 0x700e950d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd1e5: skipped 000689 1786950273.248017 read 0x002c25 thru 0x00d1e4 (0xa5c0 bytes) 000690 1786950273.263890 mapwrite 0x0341e0 thru 0x0346c5 (0x4e6 bytes) 000694 1786950273.361272 trunc from 0x0346c6 to 0x033a6c (0xc59 bytes) 000696 1786950273.381830 mapwrite 0x021941 thru 0x02d80f (0xbecf bytes) 000697 1786950273.466407 write 0x009fe8 thru 0x00ea89 (0x4aa2 bytes) 000699 1786950273.480984 read 0x027d82 thru 0x02a3d8 (0x2657 bytes) 000700 1786950273.483437 mapwrite 0x0369f4 thru 0x03aad8 (0x40e5 bytes) 000701 1786950273.669462 mapwrite 0x03e049 thru 0x03ffff (0x1fb7 bytes) 000702 1786950273.840371 mapwrite 0x02ceb5 thru 0x033984 (0x6ad0 bytes) 000703 1786950273.954947 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x151d96f0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x785d30a2 Mirror 4: 0x785d30a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000704 1786950275.516389 mapwrite 0x030f2b thru 0x03b3f5 (0xa4cb bytes) 000709 1786950275.908244 mapread 0x03360e thru 0x033d3a (0x72d bytes) 000710 1786950275.919097 mapwrite 0x02a18a thru 0x034662 (0xa4d9 bytes) 000712 1786950276.021468 trunc from 0x040000 to 0x03d846 (0x27b9 bytes) 000715 1786950276.084444 mapread 0x0290d0 thru 0x02c144 (0x3075 bytes) 000716 1786950276.104854 mapread 0x006142 thru 0x00f24f (0x910e bytes) 000717 1786950276.131725 mapwrite 0x01c90a thru 0x021aa9 (0x51a0 bytes) 000718 1786950276.268131 trunc from 0x03d847 to 0x02f649 (0xe1fd bytes) 000720 1786950276.336105 trunc from 0x02f64a to 0x02250b (0xd13e bytes) 000722 1786950276.382960 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2250c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2250c): Mirror 3: 0xa376d4ce (stale) Mirror 4: 0x3bec2355 (stale) CRC-32 checksum value for chunk [0, 0x2250c): Mirror 1: 0xcd17879 Mirror 5: 0xcd17879 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2250c: skipped 000724 1786950277.642117 read 0x016a06 thru 0x021400 (0xa9fb bytes) 000728 1786950277.696857 mapwrite 0x0218dc thru 0x02f31e (0xda43 bytes) 000729 1786950277.962763 write 0x02421d thru 0x02b9f0 (0x77d4 bytes) 000730 1786950277.965608 read 0x00eac1 thru 0x01ba46 (0xcf86 bytes) 000732 1786950277.979035 mapread 0x005714 thru 0x00be93 (0x6780 bytes) 000737 1786950277.984350 mapread 0x024a33 thru 0x02a28c (0x585a bytes) 000738 1786950277.987182 write 0x01b980 thru 0x01c131 (0x7b2 bytes) 000741 1786950277.991326 mapread 0x002e57 thru 0x012859 (0xfa03 bytes) 000742 1786950277.999989 mapread 0x01cf53 thru 0x028d7c (0xbe2a bytes) 000744 1786950278.007433 trunc from 0x02f31f to 0x036c76 (0x7958 bytes) 000746 1786950278.045191 mapread 0x0204ca thru 0x029a42 (0x9579 bytes) 000748 1786950278.052367 mapread 0x023b47 thru 0x02e878 (0xad32 bytes) 000751 1786950278.064044 mapread 0x00b071 thru 0x015153 (0xa0e3 bytes) 000757 1786950278.086437 mapwrite 0x01ea48 thru 0x02d3df (0xe998 bytes) 000759 1786950278.195696 MIRROR_RESYNC 4 mirrors 000761 1786950280.046120 trunc from 0x036c77 to 0x0111c1 (0x25ab5 bytes) 000762 1786950280.191137 read 0x009977 thru 0x0111c1 (0x784b bytes) 000763 1786950280.217430 mapwrite 0x030853 thru 0x03eee6 (0xe694 bytes) 000765 1786950280.516733 trunc from 0x03eee7 to 0x00b457 (0x33a8f bytes) 000771 1786950280.540221 write 0x023ea5 thru 0x032c46 (0xeda2 bytes) 000774 1786950280.545276 read 0x01c15d thru 0x020908 (0x47ac bytes) 000781 1786950280.565501 trunc from 0x032c47 to 0x024d00 (0xdf46 bytes) 000782 1786950280.595117 trunc from 0x024d01 to 0x00729d (0x1da63 bytes) 000783 1786950280.615414 read 0x0035af thru 0x00729d (0x3cef bytes) 000784 1786950280.618055 write 0x0096c3 thru 0x00ef53 (0x5891 bytes) 000785 1786950280.627697 read 0x0076b1 thru 0x00a3a5 (0x2cf5 bytes) 000787 1786950280.644808 mapwrite 0x007aa1 thru 0x00daa7 (0x6007 bytes) 000789 1786950280.734653 trunc from 0x00ef54 to 0x013c45 (0x4cf2 bytes) 000790 1786950280.760112 mapwrite 0x031620 thru 0x03867d (0x705e bytes) 000791 1786950280.887066 write 0x00e618 thru 0x01b4b8 (0xcea1 bytes) 000792 1786950280.889913 trunc from 0x03867e to 0x008d4e (0x2f92f bytes) 000793 1786950280.918144 trunc from 0x008d4f to 0x015a75 (0xcd27 bytes) 000795 1786950280.953936 mapwrite 0x03e6c5 thru 0x03ffff (0x193b bytes) 000796 1786950281.146031 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x84866078 (stale) Mirror 5: 0x84866078 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x84866078 (stale) Mirror 5: 0x84866078 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000797 1786950281.743023 mapread 0x02beff thru 0x0366a9 (0xa7ab bytes) 000798 1786950281.781339 read 0x00defc thru 0x01095c (0x2a61 bytes) 000799 1786950281.788253 read 0x038b97 thru 0x03fb5b (0x6fc5 bytes) 000800 1786950281.811647 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x84866078 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x84866078 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000803 1786950282.363826 mapwrite 0x03eded thru 0x03ffff (0x1213 bytes) 000804 1786950282.481141 mapwrite 0x00eccb thru 0x01b58d (0xc8c3 bytes) 000806 1786950282.599171 write 0x017d45 thru 0x01cf0e (0x51ca bytes) 000808 1786950282.612032 read 0x00bd38 thru 0x0185bd (0xc886 bytes) 000810 1786950282.613474 read 0x03be6f thru 0x03ffff (0x4191 bytes) 000811 1786950282.648658 trunc from 0x040000 to 0x039d8b (0x6274 bytes) 000812 1786950282.674274 mapread 0x009f9c thru 0x01529b (0xb300 bytes) 000816 1786950282.708327 MIRROR_RESYNC 2 mirrors 000817 1786950283.845308 trunc from 0x039d8c to 0x02248f (0x178fc bytes) 000820 1786950283.914362 write 0x023605 thru 0x02ee02 (0xb7fe bytes) 000821 1786950283.919050 mapwrite 0x000ee6 thru 0x002348 (0x1463 bytes) 000823 1786950284.004744 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ee03): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee03): Mirror 5: 0xaba91e56 (stale) CRC-32 checksum value for chunk [0, 0x2ee03): Mirror 1: 0xad4a35a1 Mirror 6: 0xad4a35a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee03: skipped 000824 1786950285.283658 read 0x00f8e2 thru 0x01f6bd (0xfddc bytes) 000826 1786950285.356411 write 0x02efbc thru 0x039117 (0xa15c bytes) 000830 1786950285.516584 write 0x01f732 thru 0x02115e (0x1a2d bytes) 000831 1786950285.530178 read 0x02c875 thru 0x0308c0 (0x404c bytes) 000833 1786950285.548907 write 0x035ec9 thru 0x036330 (0x468 bytes) 000834 1786950285.556683 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39118): Mirror 6: 0xdc57066b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39118): Mirror 6: 0xdc57066b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39118: skipped 000835 1786950286.182119 write 0x021cac thru 0x030399 (0xe6ee bytes) 000837 1786950286.225895 mapread 0x0028d9 thru 0x00e37d (0xbaa5 bytes) 000838 1786950286.286409 mapwrite 0x00f005 thru 0x01922d (0xa229 bytes) 000839 1786950286.442518 read 0x03154f thru 0x03622f (0x4ce1 bytes) truncating to largest ever: 0x3ebf5 000843 1786950286.464133 trunc from 0x039118 to 0x03ebf4 (0x5add bytes) 000844 1786950286.490621 write 0x0120e1 thru 0x013423 (0x1343 bytes) 000845 1786950286.492135 write 0x0269be thru 0x0274fd (0xb40 bytes) 000846 1786950286.494700 trunc from 0x03ebf5 to 0x02ea97 (0x1015d bytes) 000847 1786950286.518439 read 0x025e10 thru 0x02ea97 (0x8c88 bytes) 000849 1786950286.522447 trunc from 0x02ea98 to 0x015345 (0x19752 bytes) 000850 1786950286.554315 read 0x0051ec thru 0x006ffd (0x1e12 bytes) 000851 1786950286.555106 trunc from 0x015346 to 0x0214b2 (0xc16d bytes) 000852 1786950286.603205 MIRROR_RESYNC 2 mirrors 000853 1786950287.421778 MIRROR_EXTEND to 3 mirrors 000855 1786950288.285111 mapread 0x00b0b9 thru 0x00bf72 (0xeba bytes) 000858 1786950288.337477 MIRROR_RESYNC 3 mirrors 000859 1786950289.037715 mapread 0x006c76 thru 0x00e560 (0x78eb bytes) 000861 1786950289.129223 mapread 0x01b307 thru 0x01e1c1 (0x2ebb bytes) 000863 1786950289.244538 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000864 1786950289.662383 write 0x004f24 thru 0x0094f7 (0x45d4 bytes) 000865 1786950289.799309 write 0x026643 thru 0x02ad52 (0x4710 bytes) 000866 1786950289.803259 trunc from 0x02ad53 to 0x015e81 (0x14ed1 bytes) 000867 1786950289.830130 mapwrite 0x01f1cc thru 0x02266c (0x34a1 bytes) 000868 1786950289.941268 trunc from 0x02266d to 0x02328c (0xc20 bytes) 000869 1786950289.963267 trunc from 0x02328d to 0x0076ba (0x1bbd2 bytes) 000870 1786950289.997015 read 0x0048f7 thru 0x0076ba (0x2dc4 bytes) 000871 1786950290.018819 MIRROR_RESYNC 2 mirrors 000874 1786950291.205014 mapwrite 0x01154f thru 0x0185b2 (0x7064 bytes) 000876 1786950291.425723 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000877 1786950291.607255 mapread 0x0124fa thru 0x0185b2 (0x60b9 bytes) 000878 1786950291.636074 read 0x004fee thru 0x00aed4 (0x5ee7 bytes) 000882 1786950291.650588 read 0x002dd2 thru 0x00b90e (0x8b3d bytes) 000883 1786950291.659742 mapread 0x0110bb thru 0x0185b2 (0x74f8 bytes) 000886 1786950291.663724 write 0x002c28 thru 0x007e65 (0x523e bytes) 000887 1786950291.665050 trunc from 0x0185b3 to 0x031a75 (0x194c3 bytes) 000890 1786950291.681092 write 0x006032 thru 0x0107b0 (0xa77f bytes) 000892 1786950291.684244 trunc from 0x031a76 to 0x02c5cd (0x54a8 bytes) 000893 1786950291.705922 mapwrite 0x00c80b thru 0x012eb4 (0x66aa bytes) 000895 1786950291.758968 mapwrite 0x025209 thru 0x02803a (0x2e32 bytes) 000896 1786950291.821009 write 0x0041a5 thru 0x01176f (0xd5cb bytes) 000897 1786950291.824632 write 0x00b894 thru 0x016132 (0xa89f bytes) 000898 1786950291.826864 mapwrite 0x003b8e thru 0x011239 (0xd6ac bytes) 000899 1786950291.892203 mapwrite 0x01bbe0 thru 0x029443 (0xd864 bytes) 000902 1786950291.949825 trunc from 0x02c5ce to 0x03cf89 (0x109bc bytes) 000904 1786950291.981367 mapread 0x01f914 thru 0x023640 (0x3d2d bytes) 000906 1786950291.985900 read 0x002d12 thru 0x006ad1 (0x3dc0 bytes) 000909 1786950291.988875 read 0x004e13 thru 0x00b01c (0x620a bytes) 000910 1786950291.989856 mapwrite 0x029ed5 thru 0x02fe30 (0x5f5c bytes) 000911 1786950292.062901 write 0x03c6f9 thru 0x03ffff (0x3907 bytes) 000916 1786950292.068359 read 0x0392b4 thru 0x03ffff (0x6d4c bytes) 000917 1786950292.069338 write 0x0002b9 thru 0x00e875 (0xe5bd bytes) 000918 1786950292.072286 write 0x020758 thru 0x02db15 (0xd3be bytes) 000922 1786950292.078775 read 0x020a8d thru 0x02d2fa (0xc86e bytes) 000923 1786950292.079720 trunc from 0x040000 to 0x02d48d (0x12b72 bytes) 000924 1786950292.100987 trunc from 0x02d48e to 0x039e4a (0xc9bd bytes) 000925 1786950292.140875 read 0x001838 thru 0x001e2e (0x5f7 bytes) 000926 1786950292.141641 read 0x026f24 thru 0x02c308 (0x53e5 bytes) 000927 1786950292.142289 trunc from 0x039e4b to 0x01d213 (0x1cc37 bytes) 000928 1786950292.198613 mapwrite 0x029425 thru 0x02adec (0x19c8 bytes) 000933 1786950292.370706 write 0x03888c thru 0x03ffff (0x7774 bytes) 000934 1786950292.374064 mapread 0x00dab8 thru 0x011ab3 (0x3ffc bytes) 000935 1786950292.377077 write 0x009a05 thru 0x015c8c (0xc288 bytes) 000937 1786950292.407098 MIRROR_EXTEND to 2 mirrors 000938 1786950293.825078 mapwrite 0x03e8ec thru 0x03ffff (0x1714 bytes) 000939 1786950294.063741 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc5cbcec6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2464986f Mirror 3: 0x2464986f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000940 1786950295.154139 mapwrite 0x0074ea thru 0x007957 (0x46e bytes) 000943 1786950295.383540 write 0x005edf thru 0x00ec44 (0x8d66 bytes) 000944 1786950295.403183 MIRROR_RESYNC 3 mirrors 000945 1786950297.288620 mapwrite 0x0333a7 thru 0x034ec5 (0x1b1f bytes) 000946 1786950297.560299 write 0x03e469 thru 0x03ffff (0x1b97 bytes) 000947 1786950297.577216 trunc from 0x040000 to 0x03204a (0xdfb5 bytes) 000949 1786950297.611756 write 0x018527 thru 0x01f3d7 (0x6eb1 bytes) 000950 1786950297.639506 write 0x00a894 thru 0x0190ce (0xe83b bytes) 000951 1786950297.652847 write 0x00679e thru 0x00d66e (0x6ed1 bytes) 000952 1786950297.668909 mapread 0x02c255 thru 0x03204a (0x5df6 bytes) 000953 1786950297.692047 read 0x014365 thru 0x015d46 (0x19e2 bytes) 000954 1786950297.694042 read 0x00f6c3 thru 0x017d62 (0x86a0 bytes) 000955 1786950297.695099 mapwrite 0x0122b5 thru 0x017518 (0x5264 bytes) 000957 1786950297.782016 mapread 0x007446 thru 0x0102c1 (0x8e7c bytes) 000958 1786950297.790966 mapwrite 0x0399e0 thru 0x03dffb (0x461c bytes) 000959 1786950297.955182 write 0x036f25 thru 0x03a069 (0x3145 bytes) 000962 1786950297.967668 write 0x01ef3b thru 0x028c62 (0x9d28 bytes) 000965 1786950297.985919 read 0x011a15 thru 0x01cfcc (0xb5b8 bytes) 000966 1786950297.987196 write 0x02ad60 thru 0x037232 (0xc4d3 bytes) 000967 1786950298.012169 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dffc): Mirror 2: 0xeb7ce763 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dffc): Mirror 2: 0xeb7ce763 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dffc: skipped 000969 1786950298.455078 read 0x00fc0b thru 0x0146cc (0x4ac2 bytes) 000970 1786950298.470702 read 0x00228b thru 0x00dd94 (0xbb0a bytes) 000973 1786950298.495815 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000975 1786950298.786146 mapwrite 0x0251b0 thru 0x025687 (0x4d8 bytes) 000977 1786950298.894092 read 0x02cb2c thru 0x03835d (0xb832 bytes) 000978 1786950298.920833 mapread 0x038f1c thru 0x03c68d (0x3772 bytes) 000982 1786950298.943475 mapread 0x00cc6b thru 0x00df43 (0x12d9 bytes) 000984 1786950298.956982 read 0x024d9d thru 0x025028 (0x28c bytes) 000985 1786950298.994800 mapwrite 0x026b20 thru 0x0314c8 (0xa9a9 bytes) 000986 1786950299.089542 trunc from 0x03dffc to 0x00eaa8 (0x2f553 bytes) 000988 1786950299.155024 mapread 0x0009d6 thru 0x009e4f (0x947a bytes) 000991 1786950299.171317 write 0x014b1b thru 0x019092 (0x4578 bytes) 000993 1786950299.173662 trunc from 0x019093 to 0x02863b (0xf5a9 bytes) 000994 1786950299.226150 mapread 0x01ba34 thru 0x02863b (0xcc08 bytes) 000996 1786950299.356059 write 0x02e696 thru 0x032f18 (0x4883 bytes) 000997 1786950299.359405 mapread 0x00a8dd thru 0x015239 (0xa95d bytes) 000998 1786950299.384777 mapwrite 0x0211e5 thru 0x02b889 (0xa6a5 bytes) 000999 1786950299.502719 mapread 0x0262ee thru 0x02c798 (0x64ab bytes) All operations completed A-OK! PASS 70a (112s) == sanity-flr test 71: check mirror extend parent fid ==== 03:05:06 (1786950306) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 03:05:10 (1786950310) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.34237 s, 6.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143173 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 03:05:20 (1786950320) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre2 Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6406656 .. Extending file size to 7647808 .. Extending file size to 8860512 .. Extending file size to 6437792 .. Extending file size to 6668608 .. Extending file size to 8527392 .. Extending file size to 9050112 .. Extending file size to 8968512 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57345 57346 57348 57351 57354 done 192.168.203.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre2 (opts:) 192.168.203.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:06:42 (1786950402) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre2 Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7968736 .. Extending file size to 4667616 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4918208 .. failed Extending file size to 8404160 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8881472 .. Extending file size to 8751808 .. Extending file size to 8819328 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8961280 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58818 58819 58820 58821 58824 done 192.168.203.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre2 (opts:) 192.168.203.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:08:04 (1786950484) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139308 s, 15.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.328064 s, 6.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x51749cdd (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:08:26 (1786950506) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0815999 s, 25.7 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 03:08:33 (1786950513) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ce:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:08:42 (1786950522) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202521 s, 10.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:08:52 (1786950532) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.341677 s, 6.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:09:01 (1786950541) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.275328 s, 7.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:09:36 (1786950576) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg331-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188526 s, 5.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:10:34 (1786950634) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg331-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.196459 s, 10.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:11:35 (1786950695) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg331-server mds-ost sync done. Creating new pool test_204f oleg331-server: Pool lustre.test_204f created Adding targets to pool oleg331-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg331-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.273489 s, 7.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg331-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg331-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg331-server: Pool lustre.test_204f destroyed PASS 204f (86s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:13:01 (1786950781) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:13:09 (1786950789) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:13:17 (1786950797) oleg331-server: Pool lustre.test_206 created Adding targets to pool oleg331-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg331-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg331-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg331-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg331-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:13:43 (1786950823) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17001 s, 6.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625695 s, 16.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x119328bb Mirror 3: 0x119328bb PASS 207 (18s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:14:01 (1786950841) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:14:05 (1786950845) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:14:09 (1786950849) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.110842 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1810000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d30000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d30000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d30000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2500000 Read this (4 bytes): BBBB 192.168.203.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:14:47 (1786950887) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0195429 s, 419 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:14:54 (1786950894) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806497 s, 13.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f3 lmm_fid: [0x200000404:0x1f3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f3 lmm_fid: [0x200000404:0x1f3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834419 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-17 03:14:56.000000000 -0400 Modify: 2026-08-17 03:14:56.000000000 -0400 Change: 2026-08-17 03:14:57.993000000 -0400 Birth: 2026-08-17 03:14:56.999000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:15:03 (1786950903) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:15:06 (1786950906) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.686789 s, 59.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0876551 s, 46.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 17 03:15 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 17 03:15 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:15:15 (1786950915) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:15:23 (1786950923) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1686 sec =========== 03:15:34 (1786950934) === sanity-flr: start cleanup 03:15:36 (1786950936) === === sanity-flr: finish cleanup 03:15:43 (1786950943) ===