-----============= acceptance-small: sanity-flr ============----- Wed Aug 19 00:49:37 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg353-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 00:49:55 (1787114995) === oleg353-client.virtnet: executing check_config_client /mnt/lustre oleg353-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg353-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9daa02c3e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9daa02c3e000.idle_timeout=debug disable quota as required oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 00:50:36 (1787115036) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7622] == sanity-flr test 0a: lfs mirror create with -N option == 00:50:41 (1787115041) PASS 0a (28s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 00:51:09 (1787115069) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 00:51:12 (1787115072) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 00:51:16 (1787115076) PASS 0d (31s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 00:51:47 (1787115107) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 00:51:51 (1787115111) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 00:51:55 (1787115115) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9daa02c3e000.stats=clear osc.lustre-OST0001-osc-ffff9daa02c3e000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 00:52:04 (1787115124) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 00:52:17 (1787115137) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 00:52:28 (1787115148) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 00:52:32 (1787115152) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 00:52:43 (1787115163) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (14s) == sanity-flr test 2: create components from existing files ========================================================== 00:52:58 (1787115178) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 00:53:05 (1787115185) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 00:53:10 (1787115190) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 00:53:17 (1787115197) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 00:53:29 (1787115209) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 00:53:37 (1787115217) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.99098 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23448 s, 4.5 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 00:53:50 (1787115230) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.450974 s, 34.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 00:53:59 (1787115239) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 00:54:09 (1787115249) Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9daa02c3e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9daa02c3e000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9daa02c3e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9daa02c3e000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg353-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 00:55:28 (1787115328) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128093 s, 16.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9daa02c3e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9daa02c3e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 00:56:15 (1787115375) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9daa02c3e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9daa02c3e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9daa02c3e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9daa02c3e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg353-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (95s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:57:50 (1787115470) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.86227 s, 7.6 MB/s -rw-r--r-- 1 root root 52428800 Aug 19 00:58 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.502s user 0m0.000s sys 0m0.158s Stopping /mnt/lustre-ost1 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9daa02c3e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9daa02c3e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.133s user 0m0.001s sys 0m0.179s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg353-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg353-server oleg353-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9daa02c3e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9daa02c3e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.998s user 0m0.003s sys 0m0.176s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg353-server: oleg353-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg353-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9daa02c3e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9daa02c3e000.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:59:21 (1787115561) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:59:24 (1787115564) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:59:28 (1787115568) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 00:59:33 (1787115573) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 00:59:43 (1787115583) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.518416 s, 24.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:59:55 (1787115595) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636006 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260333 s, 40.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:00:01 (1787115601) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:00:09 (1787115609) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000915106 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00563741 s, 182 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109954 s, 931 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00152692 s, 671 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 01:00:20 (1787115620) create test files with size 15750528 .. 9186425d0293fe99bb0521528062f0ae - 1c416bb348aa8f8e4a003eee040c90bf - b701d916c35cfe863552652477ff2106 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 01:00:46 (1787115646) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:01:05 (1787115665) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:01:12 (1787115672) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169178 s, 6.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295963 s, 35.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 01:01:21 (1787115681) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.314558 s, 6.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.489781 s, 8.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.299915 s, 7.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 01:01:39 (1787115699) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 01:01:43 (1787115703) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:01:46 (1787115706) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301343 s, 34.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 01:01:54 (1787115714) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:01:57 (1787115717) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:02:07 (1787115727) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.277849 s, 37.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:02:18 (1787115738) llite.lustre-ffff9daa0906a000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.209611 s, 50.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg353-client.virtnet /mnt/lustre (opts:) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9daa055e9000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:02:31 (1787115751) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:03:10 (1787115790) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:03:45 (1787115825) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 01:03:48 (1787115828) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.226047 s, 13.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.204012 s, 15.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x81870e91 (nosync,stale) Mirror 2: 0x81870e91 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 1d81e63e317bd590bbb16119f42e16eb - mirror 1 checksum: 94a98d79c1764ef365c2bc9287e290f4 - mirror 2 checksum: 1d81e63e317bd590bbb16119f42e16eb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T01:03:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T01:03:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1d81e63e317bd590bbb16119f42e16eb - mirror 1 checksum: 94a98d79c1764ef365c2bc9287e290f4 - mirror 2 checksum: 94a98d79c1764ef365c2bc9287e290f4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T01:03:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T01:03:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:04:00 (1787115840) Failing mds1 on oleg353-server Stopping /mnt/lustre-mds1 (opts:) on oleg353-server 01:04:06 (1787115846) shut down facet: mds1 facet_host: oleg353-server facet_failover_host: oleg353-server Failover mds1 to oleg353-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 01:04:38 (1787115878) targets are mounted 01:04:38 (1787115878) facet_failover done oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668256 s, 15.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:05:04 (1787115904) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:05:07 (1787115907) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:05:11 (1787115911) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00746031 s, 2.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:05:19 (1787115919) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:05:22 (1787115922) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.011582 s, 354 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0861868 s, 47.5 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:05:30 (1787115930) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:05:52 (1787115952) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:06:19 (1787115979) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:06:42 (1787116002) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:07:06 (1787116026) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 01:07:13 (1787116033) mirror create pid 54267 mirror split pid 54268 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 54267 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 54268 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 01:08:19 (1787116099) Seed set to 6101 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002 1787116102.174611 write 0x039cd2 thru 0x03ffff (0x632e bytes) truncating to largest ever: 0x306c2 000004 1787116102.183674 trunc from 0x040000 to 0x0306c1 (0xf93e bytes) 000006 1787116102.216419 write 0x013f72 thru 0x014d36 (0xdc5 bytes) 000010 1787116102.257635 write 0x02cce5 thru 0x03352d (0x6849 bytes) 000012 1787116102.294172 MIRROR_EXTEND to 2 mirrors 000018 1787116103.336291 write 0x03e7b3 thru 0x03ffff (0x184d bytes) 000021 1787116103.425359 mapwrite 0x01b782 thru 0x0203fa (0x4c79 bytes) 000022 1787116103.530400 read 0x031a30 thru 0x03b70f (0x9ce0 bytes) 000023 1787116103.547438 write 0x036bb1 thru 0x03ffff (0x944f bytes) 000024 1787116103.549516 read 0x00f2a7 thru 0x015dca (0x6b24 bytes) 000026 1787116103.566294 mapread 0x02b9b7 thru 0x03a1dc (0xe826 bytes) 000028 1787116103.576849 mapwrite 0x03da72 thru 0x03ffff (0x258e bytes) truncating to largest ever: 0x349cf 000030 1787116103.640444 trunc from 0x040000 to 0x0349ce (0xb631 bytes) 000031 1787116103.665599 write 0x0361b1 thru 0x0396ab (0x34fb bytes) 000032 1787116103.667730 read 0x024bd1 thru 0x028998 (0x3dc8 bytes) 000034 1787116103.671707 mapwrite 0x01c7a8 thru 0x0257e5 (0x903e bytes) 000035 1787116103.724607 mapwrite 0x039679 thru 0x03ffff (0x6987 bytes) 000036 1787116103.861305 read 0x00cf58 thru 0x01cdf6 (0xfe9f bytes) 000037 1787116103.865460 write 0x01e766 thru 0x01e9b3 (0x24e bytes) 000038 1787116103.869197 read 0x03a5fb thru 0x03ffff (0x5a05 bytes) 000040 1787116103.873213 read 0x032b70 thru 0x037657 (0x4ae8 bytes) 000042 1787116103.877213 mapread 0x033ddc thru 0x0379e3 (0x3c08 bytes) 000046 1787116103.892674 MIRROR_RESYNC 2 mirrors 000047 1787116105.263248 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000049 1787116105.496383 mapwrite 0x006493 thru 0x012f03 (0xca71 bytes) 000050 1787116105.591603 write 0x016180 thru 0x020ca6 (0xab27 bytes) 000052 1787116105.598120 write 0x033e17 thru 0x039f6a (0x6154 bytes) 000053 1787116105.609424 read 0x00041b thru 0x0036f9 (0x32df bytes) 000054 1787116105.615238 trunc from 0x040000 to 0x00ed25 (0x312da bytes) 000055 1787116105.644145 mapwrite 0x03138b thru 0x034392 (0x3008 bytes) 000057 1787116105.730563 mapwrite 0x00c7ba thru 0x016a8e (0xa2d5 bytes) 000058 1787116105.845104 mapwrite 0x028240 thru 0x02b212 (0x2fd3 bytes) 000059 1787116105.901151 mapread 0x0179c0 thru 0x026c15 (0xf256 bytes) 000061 1787116105.911402 mapwrite 0x02ac01 thru 0x02cf37 (0x2337 bytes) 000064 1787116105.960343 mapwrite 0x01aed9 thru 0x01bef4 (0x101c bytes) 000065 1787116106.031262 write 0x026148 thru 0x02bc97 (0x5b50 bytes) 000067 1787116106.033065 write 0x005dc9 thru 0x009b92 (0x3dca bytes) 000068 1787116106.033665 trunc from 0x034393 to 0x01e0b5 (0x162dd bytes) 000069 1787116106.065075 trunc from 0x01e0b6 to 0x00e50c (0xfba9 bytes) 000071 1787116106.101557 read 0x000889 thru 0x004ceb (0x4463 bytes) 000072 1787116106.102363 mapread 0x007690 thru 0x00e50c (0x6e7d bytes) 000075 1787116106.105389 mapread 0x008b08 thru 0x00a84a (0x1d43 bytes) 000076 1787116106.107493 mapread 0x002207 thru 0x00e50c (0xc306 bytes) 000077 1787116106.110447 write 0x01c8c9 thru 0x02c864 (0xff9c bytes) 000079 1787116106.115472 mapwrite 0x03a98b thru 0x03ab48 (0x1be bytes) 000080 1787116106.264644 mapread 0x00b677 thru 0x00e562 (0x2eec bytes) 000081 1787116106.270401 mapread 0x00650f thru 0x00ca17 (0x6509 bytes) 000082 1787116106.273742 trunc from 0x03ab49 to 0x014783 (0x263c5 bytes) 000084 1787116106.317198 mapread 0x011a53 thru 0x014783 (0x2d31 bytes) 000088 1787116106.339715 MIRROR_EXTEND to 2 mirrors 000089 1787116107.478973 read 0x000148 thru 0x009809 (0x96c2 bytes) 000092 1787116107.525848 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000093 1787116107.675423 trunc from 0x014784 to 0x02c000 (0x1787d bytes) 000095 1787116107.705126 mapwrite 0x03a3b1 thru 0x03ffff (0x5c4f bytes) 000097 1787116107.786465 read 0x01e3ce thru 0x02da46 (0xf679 bytes) 000099 1787116107.796539 read 0x00c4a2 thru 0x013f83 (0x7ae2 bytes) 000100 1787116107.810266 trunc from 0x040000 to 0x012e8b (0x2d174 bytes) 000101 1787116107.826762 trunc from 0x012e8c to 0x003f6d (0xef1e bytes) 000102 1787116107.842602 trunc from 0x003f6e to 0x01ba60 (0x17af3 bytes) 000103 1787116107.861444 mapread 0x00aa6d thru 0x00c8df (0x1e73 bytes) truncating to largest ever: 0x3ed4f 000106 1787116107.879182 trunc from 0x01ba61 to 0x03ed4e (0x232ee bytes) 000109 1787116107.901390 mapwrite 0x00446d thru 0x011ed2 (0xda66 bytes) 000110 1787116107.999476 mapread 0x008183 thru 0x017f51 (0xfdcf bytes) 000111 1787116108.028518 trunc from 0x03ed4f to 0x007fcc (0x36d82 bytes) 000112 1787116108.045009 write 0x02a9fb thru 0x037ce9 (0xd2ef bytes) 000114 1787116108.047815 mapread 0x02cf1f thru 0x037630 (0xa712 bytes) 000119 1787116108.057110 write 0x018bb1 thru 0x023135 (0xa585 bytes) 000120 1787116108.070072 mapwrite 0x039ce5 thru 0x03e7ad (0x4ac9 bytes) 000122 1787116108.217820 trunc from 0x03e7ae to 0x039001 (0x57ac bytes) 000123 1787116108.244483 trunc from 0x039002 to 0x02ae26 (0xe1db bytes) 000125 1787116108.270198 read 0x013043 thru 0x01d4f1 (0xa4af bytes) 000127 1787116108.288162 MIRROR_EXTEND to 2 mirrors 000129 1787116109.197318 mapread 0x01790c thru 0x019a23 (0x2118 bytes) 000132 1787116109.285308 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000134 1787116109.521635 mapwrite 0x016518 thru 0x019382 (0x2e6b bytes) 000135 1787116109.615373 trunc from 0x02ae27 to 0x0176ea (0x1373c bytes) 000138 1787116109.708224 read 0x00f831 thru 0x0166a8 (0x6e78 bytes) 000139 1787116109.715189 trunc from 0x0176eb to 0x004c84 (0x12a66 bytes) 000140 1787116109.735193 mapwrite 0x0371c1 thru 0x0377a2 (0x5e2 bytes) 000142 1787116109.854912 trunc from 0x0377a3 to 0x03811f (0x97d bytes) 000143 1787116109.907710 mapwrite 0x017f9f thru 0x01c538 (0x459a bytes) 000144 1787116110.048039 trunc from 0x038120 to 0x02b103 (0xd01c bytes) 000145 1787116110.097981 write 0x03e043 thru 0x03ffff (0x1fbd bytes) 000148 1787116110.103748 trunc from 0x040000 to 0x0089a6 (0x37659 bytes) 000150 1787116110.176137 MIRROR_EXTEND to 2 mirrors 000152 1787116111.185034 trunc from 0x0089a7 to 0x00d7ec (0x4e46 bytes) 000153 1787116111.257057 mapwrite 0x03f74b thru 0x03ffff (0x8b5 bytes) 000154 1787116111.319454 trunc from 0x040000 to 0x030acb (0xf534 bytes) 000155 1787116111.336267 mapread 0x02c760 thru 0x02f823 (0x30c4 bytes) 000156 1787116111.355316 read 0x02403a thru 0x0306ee (0xc6b5 bytes) 000158 1787116111.366100 read 0x026ebb thru 0x02a18a (0x32d0 bytes) 000159 1787116111.369732 mapwrite 0x016d7e thru 0x0201c4 (0x9447 bytes) 000161 1787116111.492090 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000162 1787116111.707126 write 0x03773c thru 0x03ffff (0x88c4 bytes) 000163 1787116111.708715 read 0x02c478 thru 0x031d3a (0x58c3 bytes) 000165 1787116111.718756 mapwrite 0x011678 thru 0x020f62 (0xf8eb bytes) 000169 1787116111.885667 trunc from 0x040000 to 0x02d1ac (0x12e53 bytes) 000170 1787116111.919396 write 0x036370 thru 0x036ec7 (0xb58 bytes) 000174 1787116111.925031 read 0x024c52 thru 0x02d602 (0x89b1 bytes) 000177 1787116111.952580 read 0x00d2fd thru 0x00e822 (0x1526 bytes) 000178 1787116111.953298 trunc from 0x036ec8 to 0x02d67a (0x984d bytes) 000180 1787116111.986025 trunc from 0x02d67b to 0x0257e7 (0x7e93 bytes) 000183 1787116112.004988 mapwrite 0x03a6c1 thru 0x03ffff (0x593f bytes) 000184 1787116112.141056 mapwrite 0x035f07 thru 0x03d984 (0x7a7e bytes) 000185 1787116112.210332 write 0x02bd02 thru 0x031b15 (0x5e14 bytes) 000187 1787116112.214562 mapwrite 0x015182 thru 0x01ace0 (0x5b5f bytes) 000188 1787116112.299607 write 0x01fdac thru 0x026bb0 (0x6e05 bytes) 000189 1787116112.300941 write 0x0352e0 thru 0x03ffff (0xad20 bytes) 000190 1787116112.311169 MIRROR_EXTEND to 2 mirrors 000191 1787116113.348066 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000193 1787116113.549433 read 0x037f11 thru 0x03ae12 (0x2f02 bytes) 000195 1787116113.572136 mapwrite 0x0077de thru 0x0168ad (0xf0d0 bytes) 000197 1787116113.730321 write 0x02c1bf thru 0x03b648 (0xf48a bytes) 000199 1787116113.758543 mapwrite 0x024557 thru 0x025a6f (0x1519 bytes) 000201 1787116113.843164 mapread 0x02b06b thru 0x02f0f5 (0x408b bytes) 000202 1787116113.845297 mapwrite 0x017bfc thru 0x01c1b9 (0x45be bytes) 000206 1787116113.936352 trunc from 0x040000 to 0x027eb8 (0x18147 bytes) 000207 1787116113.962008 read 0x0040c4 thru 0x007ba5 (0x3ae2 bytes) 000209 1787116113.965374 read 0x01fd90 thru 0x023ec9 (0x413a bytes) 000211 1787116113.969641 mapread 0x0225ca thru 0x027eb8 (0x58ef bytes) 000213 1787116113.972963 mapread 0x00d175 thru 0x01b418 (0xe2a4 bytes) 000214 1787116113.978602 trunc from 0x027eb9 to 0x01a3c5 (0xdaf3 bytes) 000216 1787116114.003393 mapread 0x019b6e thru 0x019dbb (0x24e bytes) 000217 1787116114.004413 mapwrite 0x00ef2c thru 0x012f7c (0x4051 bytes) 000218 1787116114.083326 mapwrite 0x02dce1 thru 0x02e9ff (0xd1f bytes) 000220 1787116114.186034 MIRROR_EXTEND to 2 mirrors 000221 1787116114.968699 read 0x01c9cf thru 0x02840a (0xba3c bytes) 000222 1787116114.984336 read 0x02d2e0 thru 0x02e9ff (0x1720 bytes) 000223 1787116115.005388 write 0x0088c6 thru 0x00fe1b (0x7556 bytes) 000228 1787116115.110719 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000229 1787116115.276396 write 0x015631 thru 0x016fb5 (0x1985 bytes) 000230 1787116115.291966 trunc from 0x02ea00 to 0x00c87e (0x22181 bytes) 000233 1787116115.331458 read 0x0076e9 thru 0x00951d (0x1e35 bytes) 000234 1787116115.337950 write 0x037ebb thru 0x03ffff (0x8145 bytes) 000235 1787116115.339556 trunc from 0x040000 to 0x00308b (0x3cf74 bytes) 000238 1787116115.360548 trunc from 0x00308c to 0x03e0c9 (0x3b03e bytes) 000239 1787116115.379287 write 0x012772 thru 0x019360 (0x6bef bytes) 000240 1787116115.399280 mapread 0x0005a6 thru 0x00a20a (0x9c65 bytes) 000242 1787116115.433052 write 0x010806 thru 0x01540c (0x4c07 bytes) 000244 1787116115.435356 read 0x01e129 thru 0x02c7ef (0xe6c7 bytes) 000246 1787116115.451155 mapread 0x017105 thru 0x01a7f0 (0x36ec bytes) 000248 1787116115.455436 write 0x01922a thru 0x01cfbd (0x3d94 bytes) 000249 1787116115.457027 write 0x01b2e2 thru 0x02ad2f (0xfa4e bytes) 000250 1787116115.460021 mapwrite 0x039f8c thru 0x03ffff (0x6074 bytes) 000251 1787116115.582713 write 0x015155 thru 0x01fa07 (0xa8b3 bytes) 000252 1787116115.585450 write 0x02e389 thru 0x03785b (0x94d3 bytes) 000253 1787116115.587508 trunc from 0x040000 to 0x039629 (0x69d6 bytes) 000258 1787116115.631273 mapwrite 0x039ab8 thru 0x03ffff (0x6548 bytes) 000261 1787116115.817326 read 0x023028 thru 0x032d4a (0xfd23 bytes) 000262 1787116115.820412 mapwrite 0x02c4ae thru 0x03290a (0x645d bytes) 000263 1787116115.902202 write 0x008c79 thru 0x014100 (0xb488 bytes) 000264 1787116115.904615 read 0x002170 thru 0x00faa0 (0xd931 bytes) 000265 1787116115.907299 trunc from 0x040000 to 0x00d937 (0x326c8 bytes) 000267 1787116115.932660 trunc from 0x00d938 to 0x00d52f (0x408 bytes) 000268 1787116115.959974 read 0x001318 thru 0x0044d8 (0x31c1 bytes) 000269 1787116115.961845 mapread 0x0097c0 thru 0x00d52f (0x3d70 bytes) 000271 1787116115.979281 MIRROR_EXTEND to 2 mirrors 000272 1787116116.824761 read 0x0074c3 thru 0x00d52f (0x606d bytes) 000274 1787116116.863033 trunc from 0x00d530 to 0x005af5 (0x7a3a bytes) 000277 1787116116.970718 mapread 0x001f47 thru 0x005af5 (0x3baf bytes) 000278 1787116116.991214 write 0x014163 thru 0x020d8f (0xcc2d bytes) 000279 1787116117.004701 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20d90): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20d90): Mirror 2: 0x786660a6 (stale) CRC-32 checksum value for chunk [0, 0x20d90): Mirror 1: 0x7311f9c9 Mirror 3: 0x7311f9c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20d90: skipped 000280 1787116117.974304 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x20d90): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20d90): Mirror 2: 0x786660a6 (stale) CRC-32 checksum value for chunk [0, 0x20d90): Mirror 1: 0x7311f9c9 Mirror 3: 0x7311f9c9 Mirror 4: 0x7311f9c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20d90: skipped 000281 1787116119.175206 read 0x00c8ce thru 0x00cbcc (0x2ff bytes) 000282 1787116119.181996 read 0x01d89d thru 0x020d8f (0x34f3 bytes) 000283 1787116119.191999 mapread 0x00b59e thru 0x00df01 (0x2964 bytes) 000285 1787116119.193151 write 0x031027 thru 0x034e25 (0x3dff bytes) 000286 1787116119.276306 mapread 0x03227c thru 0x034e25 (0x2baa bytes) 000287 1787116119.277474 write 0x00e659 thru 0x01d271 (0xec19 bytes) 000289 1787116119.300855 read 0x0179f8 thru 0x026630 (0xec39 bytes) 000292 1787116119.331474 read 0x026269 thru 0x026430 (0x1c8 bytes) 000293 1787116119.333013 mapwrite 0x0180e1 thru 0x025d7a (0xdc9a bytes) 000295 1787116119.430116 MIRROR_RESYNC 4 mirrors 000296 1787116120.755609 trunc from 0x034e26 to 0x00b127 (0x29cfe bytes) 000297 1787116120.867537 mapwrite 0x0392e0 thru 0x03ffff (0x6d20 bytes) 000302 1787116121.010644 trunc from 0x040000 to 0x0321fc (0xde03 bytes) 000304 1787116121.044011 read 0x02a5e8 thru 0x031feb (0x7a04 bytes) 000305 1787116121.056354 mapread 0x026654 thru 0x0321fc (0xbba9 bytes) 000306 1787116121.073224 write 0x025f06 thru 0x030d02 (0xadfd bytes) 000308 1787116121.075484 trunc from 0x0321fd to 0x03cfd4 (0xadd8 bytes) 000310 1787116121.107343 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cfd5): Mirror 2: 0x2a6ac00c (stale) Mirror 3: 0x2a6ac00c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cfd5): Mirror 2: 0x2a6ac00c (stale) Mirror 3: 0x2a6ac00c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cfd5: skipped 000311 1787116121.516343 mapwrite 0x012df3 thru 0x0229dd (0xfbeb bytes) 000312 1787116121.657177 mapwrite 0x027dbc thru 0x0371fc (0xf441 bytes) 000314 1787116121.786472 mapwrite 0x01077a thru 0x01d6bf (0xcf46 bytes) 000315 1787116121.940193 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cfd5): Mirror 3: 0x2a6ac00c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cfd5): Mirror 3: 0x2a6ac00c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cfd5: skipped 000316 1787116122.298435 read 0x02761c thru 0x033f61 (0xc946 bytes) 000317 1787116122.321432 trunc from 0x03cfd5 to 0x017509 (0x25acb bytes) 000318 1787116122.356435 mapwrite 0x022fd6 thru 0x032839 (0xf864 bytes) 000319 1787116122.484149 read 0x02dd30 thru 0x032839 (0x4b0a bytes) 000321 1787116122.488158 trunc from 0x03283a to 0x036330 (0x3af7 bytes) 000323 1787116122.510404 trunc from 0x036331 to 0x01b565 (0x1adcb bytes) 000324 1787116122.554470 mapwrite 0x02a2b0 thru 0x03418d (0x9ede bytes) 000326 1787116122.730651 write 0x037323 thru 0x03e784 (0x7462 bytes) 000327 1787116122.732744 write 0x03b59a thru 0x03ffff (0x4a66 bytes) 000331 1787116122.742412 trunc from 0x040000 to 0x03cdbb (0x3244 bytes) 000334 1787116122.811526 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cdbc): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cdbc): Mirror 3: 0x1ac8ee60 (stale) CRC-32 checksum value for chunk [0, 0x3cdbc): Mirror 1: 0x3a4903c9 Mirror 4: 0x3a4903c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cdbc: skipped 000337 1787116124.028925 mapread 0x037d92 thru 0x03b5c4 (0x3833 bytes) 000338 1787116124.063339 mapwrite 0x02ea89 thru 0x03a38f (0xb907 bytes) 000340 1787116124.398303 write 0x01a9cb thru 0x01d542 (0x2b78 bytes) 000341 1787116124.416970 read 0x02343c thru 0x02d423 (0x9fe8 bytes) 000342 1787116124.428681 mapwrite 0x02d1e5 thru 0x02e4e7 (0x1303 bytes) 000344 1787116124.551689 mapread 0x021509 thru 0x023d28 (0x2820 bytes) 000345 1787116124.568128 mapread 0x02f78f thru 0x03cdbb (0xd62d bytes) 000347 1787116124.571154 mapwrite 0x033773 thru 0x03d35d (0x9beb bytes) 000351 1787116124.732660 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d35e): Mirror 3: 0x2187fc8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d35e): Mirror 3: 0x2187fc8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d35e: skipped 000355 1787116125.117859 trunc from 0x03d35e to 0x0399f2 (0x396b bytes) 000358 1787116125.163954 write 0x009008 thru 0x018842 (0xf83b bytes) 000360 1787116125.182726 mapread 0x009613 thru 0x015d22 (0xc710 bytes) 000364 1787116125.191173 mapwrite 0x034fe6 thru 0x03f434 (0xa44f bytes) 000366 1787116125.395198 read 0x00d90c thru 0x01af6a (0xd65f bytes) 000369 1787116125.423339 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000370 1787116125.612286 read 0x018fe8 thru 0x026f52 (0xdf6b bytes) 000371 1787116125.641410 write 0x0186e4 thru 0x01b869 (0x3186 bytes) 000373 1787116125.650031 MIRROR_EXTEND to 2 mirrors 000378 1787116126.710397 write 0x020fbd thru 0x0258a9 (0x48ed bytes) 000380 1787116126.787261 write 0x0140e5 thru 0x0207c8 (0xc6e4 bytes) 000381 1787116126.793411 read 0x001b1a thru 0x00a3ea (0x88d1 bytes) 000382 1787116126.800993 read 0x023106 thru 0x0305b5 (0xd4b0 bytes) 000385 1787116126.812664 read 0x00e160 thru 0x019367 (0xb208 bytes) 000386 1787116126.815210 write 0x010c33 thru 0x01b173 (0xa541 bytes) 000389 1787116126.819061 trunc from 0x03f435 to 0x02d608 (0x11e2c bytes) 000392 1787116126.845361 trunc from 0x02d609 to 0x031cd7 (0x46cf bytes) 000393 1787116126.862386 mapwrite 0x013d08 thru 0x022daa (0xf0a3 bytes) 000394 1787116126.919063 write 0x0108eb thru 0x01eefe (0xe614 bytes) 000395 1787116126.928257 mapread 0x0011a0 thru 0x004941 (0x37a2 bytes) 000398 1787116126.930886 trunc from 0x031cd8 to 0x0213e3 (0x108f4 bytes) 000399 1787116126.960215 write 0x021ec4 thru 0x023dbe (0x1efb bytes) 000400 1787116126.964514 write 0x033f8a thru 0x038977 (0x49ee bytes) 000401 1787116126.969444 write 0x00f25e thru 0x01600b (0x6dae bytes) 000404 1787116126.977564 trunc from 0x038978 to 0x00b83e (0x2d139 bytes) 000405 1787116127.031213 mapread 0x001987 thru 0x00b83e (0x9eb8 bytes) 000408 1787116127.036162 mapread 0x00321c thru 0x00b83e (0x8623 bytes) 000409 1787116127.037477 mapread 0x0074b3 thru 0x007658 (0x1a6 bytes) 000411 1787116127.041982 write 0x02ef23 thru 0x031aad (0x2b8b bytes) 000412 1787116127.044351 read 0x019754 thru 0x01af2d (0x17da bytes) 000414 1787116127.056301 mapread 0x014505 thru 0x020798 (0xc294 bytes) 000415 1787116127.058628 read 0x02c99e thru 0x031aad (0x5110 bytes) 000416 1787116127.059683 mapread 0x01569c thru 0x024b1b (0xf480 bytes) 000417 1787116127.062549 trunc from 0x031aae to 0x01f0f0 (0x129bd bytes) 000419 1787116127.102139 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000420 1787116127.320482 mapwrite 0x0069db thru 0x00b5ff (0x4c25 bytes) 000421 1787116127.385333 read 0x00ea68 thru 0x012ecb (0x4464 bytes) 000422 1787116127.392159 trunc from 0x01f0f1 to 0x03c662 (0x1d572 bytes) 000426 1787116127.431400 mapread 0x01b371 thru 0x02a3fa (0xf08a bytes) 000427 1787116127.454923 mapwrite 0x00d340 thru 0x014271 (0x6f32 bytes) 000428 1787116127.500642 read 0x002874 thru 0x008096 (0x5823 bytes) 000433 1787116127.506268 read 0x01e01d thru 0x02a171 (0xc155 bytes) 000435 1787116127.509175 mapwrite 0x012c41 thru 0x01b7c0 (0x8b80 bytes) 000438 1787116127.560428 write 0x01b76c thru 0x028b32 (0xd3c7 bytes) 000440 1787116127.563376 mapwrite 0x0198d4 thru 0x024f94 (0xb6c1 bytes) 000441 1787116127.642499 mapwrite 0x02cdd6 thru 0x038e9b (0xc0c6 bytes) 000442 1787116127.727173 read 0x01af30 thru 0x02872b (0xd7fc bytes) 000446 1787116127.749282 read 0x0079b6 thru 0x00c5a9 (0x4bf4 bytes) 000449 1787116127.751457 read 0x015178 thru 0x0161fa (0x1083 bytes) 000450 1787116127.757200 trunc from 0x03c663 to 0x03d07e (0xa1c bytes) 000451 1787116127.783186 write 0x028386 thru 0x032136 (0x9db1 bytes) 000452 1787116127.785004 mapwrite 0x0373ea thru 0x03d2b7 (0x5ece bytes) 000453 1787116127.898159 read 0x0051d5 thru 0x01226b (0xd097 bytes) 000455 1787116127.899607 read 0x000893 thru 0x002d29 (0x2497 bytes) 000456 1787116127.901495 read 0x02d885 thru 0x02e5a3 (0xd1f bytes) 000457 1787116127.903058 trunc from 0x03d2b8 to 0x00b267 (0x32050 bytes) 000460 1787116127.937144 mapwrite 0x00a9e2 thru 0x00d32e (0x294d bytes) 000461 1787116128.006146 mapwrite 0x01e997 thru 0x028c70 (0xa2da bytes) 000462 1787116128.091273 mapwrite 0x001ea8 thru 0x0059d2 (0x3b2b bytes) 000463 1787116128.129445 read 0x003f9b thru 0x0119f3 (0xda59 bytes) 000464 1787116128.130308 mapwrite 0x02c6c9 thru 0x02d015 (0x94d bytes) 000468 1787116128.198713 write 0x021489 thru 0x024ceb (0x3863 bytes) 000471 1787116128.202242 trunc from 0x02d016 to 0x0326ab (0x5696 bytes) 000472 1787116128.218092 mapwrite 0x00c06d thru 0x0138ca (0x785e bytes) 000474 1787116128.263899 mapread 0x0158b6 thru 0x01d9ca (0x8115 bytes) 000477 1787116128.266954 trunc from 0x0326ac to 0x036ab4 (0x4409 bytes) 000478 1787116128.283553 mapwrite 0x0159e9 thru 0x018259 (0x2871 bytes) 000480 1787116128.319760 write 0x039762 thru 0x03b87d (0x211c bytes) 000481 1787116128.321468 trunc from 0x03b87e to 0x0045bf (0x372be bytes) 000483 1787116128.341556 write 0x0074af thru 0x00de4e (0x69a0 bytes) 000485 1787116128.344337 read 0x009b90 thru 0x00de4e (0x42bf bytes) 000486 1787116128.345526 mapread 0x002b47 thru 0x007349 (0x4803 bytes) 000487 1787116128.355556 MIRROR_EXTEND to 2 mirrors 000488 1787116129.135049 mapwrite 0x027e8b thru 0x02f717 (0x788d bytes) 000489 1787116129.412017 MIRROR_RESYNC 2 mirrors 000490 1787116130.471226 mapwrite 0x0287db thru 0x02fec6 (0x76ec bytes) 000491 1787116130.708417 mapread 0x024970 thru 0x0275b6 (0x2c47 bytes) 000495 1787116130.718968 read 0x007694 thru 0x010c55 (0x95c2 bytes) 000496 1787116130.734181 mapwrite 0x02e431 thru 0x031184 (0x2d54 bytes) 000497 1787116130.830602 write 0x029da9 thru 0x037f0d (0xe165 bytes) 000500 1787116130.838263 mapwrite 0x036e5f thru 0x03ffff (0x91a1 bytes) 000503 1787116130.910395 read 0x01e00c thru 0x02cf29 (0xef1e bytes) 000505 1787116130.926225 read 0x03370e thru 0x036cc5 (0x35b8 bytes) 000506 1787116130.927158 read 0x01186f thru 0x01ce9f (0xb631 bytes) 000507 1787116130.928506 read 0x03a6d9 thru 0x03f9ef (0x5317 bytes) 000511 1787116130.935620 mapread 0x031d45 thru 0x03c35e (0xa61a bytes) 000512 1787116130.937514 read 0x0392a7 thru 0x03ffc9 (0x6d23 bytes) 000513 1787116130.938993 trunc from 0x040000 to 0x02a522 (0x15add bytes) 000514 1787116130.958694 mapwrite 0x037180 thru 0x03ffff (0x8e80 bytes) 000518 1787116131.032332 MIRROR_RESYNC 2 mirrors 000519 1787116132.095939 trunc from 0x040000 to 0x01eef1 (0x2110e bytes) 000520 1787116132.208122 mapwrite 0x01344e thru 0x01f816 (0xc3c9 bytes) 000522 1787116132.336047 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1f817): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f817): Mirror 2: 0x55de8d8b (stale) CRC-32 checksum value for chunk [0, 0x1f817): Mirror 1: 0x4b3f75a Mirror 3: 0x4b3f75a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f817: skipped 000523 1787116133.378431 read 0x014682 thru 0x0181f2 (0x3b71 bytes) 000525 1787116133.441940 mapwrite 0x002da9 thru 0x00d392 (0xa5ea bytes) 000526 1787116133.664882 trunc from 0x01f817 to 0x007225 (0x185f1 bytes) 000528 1787116133.715327 read 0x00647a thru 0x007225 (0xdac bytes) 000529 1787116133.716880 write 0x026bd3 thru 0x031082 (0xa4b0 bytes) 000530 1787116133.724365 MIRROR_RESYNC 3 mirrors 000533 1787116135.108163 MIRROR_EXTEND to 4 mirrors 000534 1787116135.984307 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000538 1787116136.492514 write 0x03ad80 thru 0x03ffff (0x5280 bytes) 000540 1787116136.578177 write 0x015427 thru 0x01c47c (0x7056 bytes) 000542 1787116136.589124 write 0x004f13 thru 0x012ca2 (0xdd90 bytes) 000545 1787116136.600502 read 0x0105b4 thru 0x01b1b7 (0xac04 bytes) 000546 1787116136.612159 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc4d0c0be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc4d0c0be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000547 1787116137.021724 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc4d0c0be (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbb1e5140 Mirror 3: 0xbb1e5140 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000549 1787116138.437482 mapread 0x034472 thru 0x036556 (0x20e5 bytes) 000551 1787116138.490631 mapread 0x013a1b thru 0x0201f9 (0xc7df bytes) 000552 1787116138.544528 read 0x022658 thru 0x02b636 (0x8fdf bytes) 000555 1787116138.601025 read 0x02e1a4 thru 0x03c496 (0xe2f3 bytes) 000556 1787116138.622584 read 0x033104 thru 0x03b4d1 (0x83ce bytes) 000557 1787116138.635829 trunc from 0x040000 to 0x028f82 (0x1707d bytes) 000558 1787116138.752403 write 0x03d976 thru 0x03fbae (0x2239 bytes) 000560 1787116138.756358 trunc from 0x03fbaf to 0x00dcff (0x31eaf bytes) 000563 1787116138.788953 trunc from 0x00dd00 to 0x031f3b (0x2423c bytes) 000565 1787116138.820120 trunc from 0x031f3c to 0x01f923 (0x12618 bytes) 000567 1787116138.848222 trunc from 0x01f924 to 0x0367a0 (0x16e7d bytes) 000568 1787116138.873356 trunc from 0x0367a1 to 0x005450 (0x31350 bytes) 000569 1787116138.900271 mapwrite 0x00fabf thru 0x013f57 (0x4499 bytes) 000573 1787116139.051939 write 0x039f90 thru 0x03ffff (0x6070 bytes) 000574 1787116139.063354 mapread 0x019882 thru 0x01bc18 (0x2397 bytes) 000576 1787116139.103280 MIRROR_RESYNC 3 mirrors 000579 1787116140.660407 mapread 0x01a2ee thru 0x01d367 (0x307a bytes) 000581 1787116140.681338 mapwrite 0x00d5ea thru 0x01a6d4 (0xd0eb bytes) 000582 1787116140.896573 mapwrite 0x01979d thru 0x020aed (0x7351 bytes) 000587 1787116140.984230 mapwrite 0x0275bd thru 0x034328 (0xcd6c bytes) 000590 1787116141.119227 mapread 0x03edaf thru 0x03ffff (0x1251 bytes) 000591 1787116141.138410 read 0x006209 thru 0x00702c (0xe24 bytes) 000593 1787116141.139755 read 0x02d1ee thru 0x031d9b (0x4bae bytes) 000594 1787116141.140417 read 0x00e412 thru 0x01629f (0x7e8e bytes) 000598 1787116141.146257 read 0x035275 thru 0x03ffff (0xad8b bytes) 000601 1787116141.152390 mapwrite 0x01d8e7 thru 0x023895 (0x5faf bytes) 000602 1787116141.207285 mapread 0x02b996 thru 0x034a67 (0x90d2 bytes) 000606 1787116141.215385 mapread 0x018c95 thru 0x02811a (0xf486 bytes) 000608 1787116141.218104 mapwrite 0x00bbe8 thru 0x01b312 (0xf72b bytes) 000611 1787116141.292857 trunc from 0x040000 to 0x022008 (0x1dff7 bytes) 000612 1787116141.329116 trunc from 0x022009 to 0x035e3a (0x13e32 bytes) 000613 1787116141.354715 read 0x032e06 thru 0x035e3a (0x3035 bytes) 000614 1787116141.367201 read 0x00977c thru 0x010a05 (0x728a bytes) 000615 1787116141.367994 write 0x01aa67 thru 0x01bd79 (0x1313 bytes) 000616 1787116141.369077 trunc from 0x035e3b to 0x03d3c3 (0x7589 bytes) 000617 1787116141.400415 mapread 0x033fbf thru 0x037f67 (0x3fa9 bytes) 000618 1787116141.427460 read 0x014cb5 thru 0x0189a3 (0x3cef bytes) 000621 1787116141.431695 trunc from 0x03d3c4 to 0x013483 (0x29f40 bytes) 000622 1787116141.467394 trunc from 0x013484 to 0x0181c7 (0x4d44 bytes) 000625 1787116141.505658 mapread 0x006b4a thru 0x00fe6c (0x9323 bytes) 000627 1787116141.510192 mapwrite 0x00a900 thru 0x00b2f6 (0x9f7 bytes) 000628 1787116141.558863 write 0x032b64 thru 0x03ffff (0xd49c bytes) 000630 1787116141.562804 mapwrite 0x036581 thru 0x03ffff (0x9a7f bytes) 000631 1787116141.635637 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbb611be4 (stale) Mirror 3: 0xbb611be4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf9402775 Mirror 4: 0xf9402775 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000632 1787116142.977090 trunc from 0x040000 to 0x006c65 (0x3939a bytes) 000634 1787116143.081537 write 0x0385d3 thru 0x03ffff (0x7a2d bytes) 000635 1787116143.084349 read 0x0292c4 thru 0x036df8 (0xdb35 bytes) 000636 1787116143.092504 mapread 0x01add8 thru 0x0217a4 (0x69cd bytes) 000638 1787116143.111944 mapread 0x01eb64 thru 0x02b0e0 (0xc57d bytes) 000643 1787116143.121019 trunc from 0x040000 to 0x021236 (0x1edc9 bytes) 000644 1787116143.147915 trunc from 0x021237 to 0x000e51 (0x203e5 bytes) 000645 1787116143.167643 write 0x00a55c thru 0x0131db (0x8c80 bytes) 000647 1787116143.170649 read 0x0019a0 thru 0x003e69 (0x24ca bytes) 000649 1787116143.182207 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x131dc): Mirror 3: 0x2a67085 (stale) Mirror 4: 0xbef201fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x131dc): Mirror 3: 0x2a67085 (stale) Mirror 4: 0xbef201fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x131dc: skipped 000650 1787116143.654751 mapread 0x00094c thru 0x00b433 (0xaae8 bytes) 000651 1787116143.730130 MIRROR_RESYNC 3 mirrors 000653 1787116144.912120 read 0x00c136 thru 0x0131db (0x70a6 bytes) 000654 1787116144.931733 mapwrite 0x00bc1d thru 0x00ccf2 (0x10d6 bytes) 000655 1787116145.137365 read 0x009c6c thru 0x0131db (0x9570 bytes) 000656 1787116145.146211 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x131dc): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x131dc): Mirror 3: 0x582d9a1b (stale) Mirror 4: 0x582d9a1b (stale) CRC-32 checksum value for chunk [0, 0x131dc): Mirror 1: 0x3c4afb96 Mirror 5: 0x3c4afb96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x131dc: skipped 000657 1787116146.200266 MIRROR_RESYNC 4 mirrors 000658 1787116147.243895 mapwrite 0x036ee0 thru 0x03ffff (0x9120 bytes) 000660 1787116147.573736 mapread 0x027131 thru 0x0297d7 (0x26a7 bytes) 000661 1787116147.604862 mapwrite 0x03cb52 thru 0x03da65 (0xf14 bytes) 000662 1787116147.707750 read 0x02b39b thru 0x035de8 (0xaa4e bytes) 000663 1787116147.714877 mapread 0x036290 thru 0x03ffff (0x9d70 bytes) 000664 1787116147.716181 write 0x017d6e thru 0x0230a6 (0xb339 bytes) 000665 1787116147.723216 mapread 0x03b8de thru 0x03c684 (0xda7 bytes) 000667 1787116147.727032 read 0x00a6b4 thru 0x00cebf (0x280c bytes) 000669 1787116147.738191 write 0x037fc1 thru 0x03e989 (0x69c9 bytes) 000670 1787116147.740272 mapwrite 0x0347e8 thru 0x03ffff (0xb818 bytes) 000672 1787116147.875125 read 0x02755f thru 0x02bb66 (0x4608 bytes) 000673 1787116147.876016 mapwrite 0x02dfa8 thru 0x02fb0d (0x1b66 bytes) 000677 1787116147.917382 mapwrite 0x0266a6 thru 0x0339fd (0xd358 bytes) 000678 1787116147.970476 write 0x029370 thru 0x034710 (0xb3a1 bytes) 000681 1787116147.974244 read 0x007f31 thru 0x011c38 (0x9d08 bytes) 000682 1787116147.975390 trunc from 0x040000 to 0x03ae14 (0x51eb bytes) 000683 1787116147.998207 trunc from 0x03ae15 to 0x010e77 (0x29f9d bytes) 000685 1787116148.021475 mapread 0x0011de thru 0x002baa (0x19cd bytes) 000687 1787116148.024250 trunc from 0x010e78 to 0x03aa9b (0x29c24 bytes) 000689 1787116148.043914 mapwrite 0x029a60 thru 0x02d1fe (0x379f bytes) 000690 1787116148.094291 write 0x026c9d thru 0x02bf8b (0x52ef bytes) 000691 1787116148.103290 MIRROR_RESYNC 4 mirrors 000694 1787116149.650493 trunc from 0x03aa9c to 0x01735b (0x23740 bytes) 000698 1787116149.782953 trunc from 0x01735c to 0x022ab7 (0xb75c bytes) 000699 1787116149.809644 mapwrite 0x028283 thru 0x03377d (0xb4fb bytes) 000700 1787116149.910802 write 0x01a096 thru 0x01e87e (0x47e9 bytes) 000703 1787116149.925512 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3377e): Mirror 4: 0xce5dc256 (stale) Mirror 5: 0xce5dc256 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3377e): Mirror 4: 0xce5dc256 (stale) Mirror 5: 0xce5dc256 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3377e: skipped 000704 1787116150.533618 mapwrite 0x014fbf thru 0x01655a (0x159c bytes) 000705 1787116150.673315 mapwrite 0x01c7e0 thru 0x02c2f1 (0xfb12 bytes) 000706 1787116150.792870 trunc from 0x03377e to 0x00dd81 (0x259fc bytes) 000707 1787116150.815047 mapread 0x003ce0 thru 0x0056cf (0x19f0 bytes) 000708 1787116150.826425 read 0x00d751 thru 0x00dd81 (0x631 bytes) 000710 1787116150.833301 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdd82): Mirror 5: 0x364169a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdd82): Mirror 5: 0x364169a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdd82: skipped 000711 1787116151.213295 mapwrite 0x00f34e thru 0x011ee8 (0x2b9b bytes) 000712 1787116151.344087 read 0x0087eb thru 0x00d485 (0x4c9b bytes) 000713 1787116151.352986 write 0x01cff7 thru 0x02a817 (0xd821 bytes) 000714 1787116151.354886 mapwrite 0x018fee thru 0x01e97a (0x598d bytes) 000715 1787116151.446924 read 0x005838 thru 0x00f4fa (0x9cc3 bytes) 000716 1787116151.460226 read 0x0169bd thru 0x01d928 (0x6f6c bytes) 000718 1787116151.461800 read 0x012521 thru 0x0199ab (0x748b bytes) 000719 1787116151.462493 mapread 0x014218 thru 0x014afd (0x8e6 bytes) 000722 1787116151.471707 write 0x039afc thru 0x03ffff (0x6504 bytes) 000723 1787116151.474698 trunc from 0x040000 to 0x03a0ea (0x5f15 bytes) 000724 1787116151.550201 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a0eb): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0eb): Mirror 5: 0xd39df53b (stale) CRC-32 checksum value for chunk [0, 0x3a0eb): Mirror 1: 0xac51133e Mirror 6: 0xac51133e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0eb: skipped 000726 1787116153.025847 write 0x036aa5 thru 0x03cacf (0x602b bytes) 000727 1787116153.175285 read 0x013d01 thru 0x01ce35 (0x9135 bytes) 000729 1787116153.186290 read 0x02cf0c thru 0x02d94b (0xa40 bytes) 000730 1787116153.197317 mapread 0x006819 thru 0x00d306 (0x6aee bytes) 000731 1787116153.204294 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cad0): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cad0): Mirror 5: 0x1d7f7c6f (stale) Mirror 6: 0xa6215aed (stale) CRC-32 checksum value for chunk [0, 0x3cad0): Mirror 1: 0x65a29f8 Mirror 7: 0x65a29f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cad0: skipped 000733 1787116154.528471 mapread 0x0219f7 thru 0x02c70b (0xad15 bytes) 000734 1787116154.601733 trunc from 0x03cad0 to 0x033790 (0x933f bytes) 000735 1787116154.830580 mapwrite 0x032d49 thru 0x032f98 (0x250 bytes) 000738 1787116154.943035 read 0x0037bb thru 0x0079d2 (0x4218 bytes) 000739 1787116154.961600 mapwrite 0x02a19a thru 0x039b6c (0xf9d3 bytes) 000740 1787116155.116979 write 0x031ace thru 0x03ea34 (0xcf67 bytes) 000741 1787116155.125989 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ea35): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea35): Mirror 5: 0xce300c13 (stale) Mirror 6: 0x9d327b9b (stale) Mirror 7: 0x3dc6e116 (stale) CRC-32 checksum value for chunk [0, 0x3ea35): Mirror 1: 0x17e3f91b Mirror 8: 0x17e3f91b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea35: skipped 000743 1787116156.676255 mapread 0x036c23 thru 0x03ea34 (0x7e12 bytes) 000744 1787116156.797845 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ea35): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea35): Mirror 5: 0xce300c13 (stale) Mirror 6: 0x9d327b9b (stale) Mirror 7: 0x3dc6e116 (stale) CRC-32 checksum value for chunk [0, 0x3ea35): Mirror 1: 0x17e3f91b Mirror 8: 0x17e3f91b Mirror 9: 0x17e3f91b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea35: skipped 000745 1787116158.388666 read 0x02e3ce thru 0x030050 (0x1c83 bytes) 000746 1787116158.421045 mapwrite 0x01ca9d thru 0x021c09 (0x516d bytes) 000747 1787116158.755184 trunc from 0x03ea35 to 0x018068 (0x269cc bytes) 000749 1787116158.818947 trunc from 0x018069 to 0x023a9e (0xba36 bytes) 000751 1787116158.853608 read 0x007d6d thru 0x012a0c (0xaca0 bytes) 000752 1787116158.881335 mapread 0x01a560 thru 0x021d84 (0x7825 bytes) 000753 1787116158.884235 mapwrite 0x030033 thru 0x03e8e7 (0xe8b5 bytes) 000754 1787116159.016327 read 0x03c151 thru 0x03e8e7 (0x2797 bytes) 000755 1787116159.029101 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e8e8): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8e8): Mirror 5: 0x6667a262 (stale) Mirror 6: 0xf33e28d3 (stale) Mirror 7: 0xdc2f22b4 (stale) Mirror 8: 0xf46a1bed (stale) Mirror 9: 0xf46a1bed (stale) CRC-32 checksum value for chunk [0, 0x3e8e8): Mirror 1: 0xb62cf1f4 Mirror 10: 0xb62cf1f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8e8: skipped 000756 1787116160.326045 mapwrite 0x029250 thru 0x030230 (0x6fe1 bytes) 000757 1787116160.609435 mapwrite 0x008529 thru 0x00a964 (0x243c bytes) 000758 1787116160.743081 mapwrite 0x004cd9 thru 0x00fc4d (0xaf75 bytes) 000759 1787116160.855170 write 0x01c919 thru 0x028957 (0xc03f bytes) 000761 1787116160.870108 mapread 0x02314c thru 0x02ee9d (0xbd52 bytes) 000763 1787116160.904930 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e8e8): Mirror 5: 0x6667a262 (stale) Mirror 6: 0xf33e28d3 (stale) Mirror 7: 0xdc2f22b4 (stale) Mirror 9: 0xf46a1bed (stale) Mirror 10: 0xb62cf1f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8e8): Mirror 5: 0x6667a262 (stale) Mirror 6: 0xf33e28d3 (stale) Mirror 7: 0xdc2f22b4 (stale) Mirror 9: 0xf46a1bed (stale) Mirror 10: 0xb62cf1f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8e8: skipped 000764 1787116161.684429 read 0x03144c thru 0x032053 (0xc08 bytes) 000765 1787116161.700959 trunc from 0x03e8e8 to 0x035bbd (0x8d2a bytes) 000766 1787116161.745265 trunc from 0x035bbe to 0x02202c (0x13b91 bytes) 000767 1787116161.770273 read 0x020b9b thru 0x021a69 (0xecf bytes) 000768 1787116161.792714 read 0x015a2b thru 0x0214db (0xbab1 bytes) 000770 1787116161.794176 trunc from 0x02202d to 0x00229e (0x1fd8e bytes) 000771 1787116161.859083 MIRROR_RESYNC 6 mirrors 000773 1787116163.924230 MIRROR_RESYNC 6 mirrors 000774 1787116164.502321 MIRROR_RESYNC 6 mirrors 000775 1787116165.233106 write 0x03aa5a thru 0x03aa88 (0x2f bytes) 000779 1787116165.327260 write 0x001161 thru 0x00ed38 (0xdbd8 bytes) 000782 1787116165.372331 write 0x00e370 thru 0x00f6fa (0x138b bytes) 000784 1787116165.387901 write 0x00709f thru 0x0079bb (0x91d bytes) 000786 1787116165.394026 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aa89): Mirror 5: 0xc1fd11bc (stale) Mirror 6: 0xc1fd11bc (stale) Mirror 7: 0xc1fd11bc (stale) Mirror 10: 0xc1fd11bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa89): Mirror 5: 0xc1fd11bc (stale) Mirror 6: 0xc1fd11bc (stale) Mirror 7: 0xc1fd11bc (stale) Mirror 10: 0xc1fd11bc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa89: skipped 000787 1787116165.860715 read 0x02d303 thru 0x03aa88 (0xd786 bytes) 000788 1787116165.871556 mapread 0x00e0bc thru 0x017fec (0x9f31 bytes) 000789 1787116165.894693 mapread 0x025a92 thru 0x034b23 (0xf092 bytes) 000790 1787116165.907676 mapwrite 0x025722 thru 0x031602 (0xbee1 bytes) 000791 1787116165.972990 mapwrite 0x02845c thru 0x02945b (0x1000 bytes) 000792 1787116166.015579 trunc from 0x03aa89 to 0x002904 (0x38184 bytes) 000793 1787116166.046813 mapread 0x001f72 thru 0x002904 (0x993 bytes) 000794 1787116166.060044 read 0x0028aa thru 0x002904 (0x5b bytes) 000798 1787116166.067708 write 0x007458 thru 0x0114a3 (0xa04c bytes) 000800 1787116166.073230 read 0x003c5f thru 0x006535 (0x28d7 bytes) 000801 1787116166.092742 mapread 0x006e7f thru 0x011178 (0xa2fa bytes) 000802 1787116166.100734 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x114a4): Mirror 6: 0xad1151b1 (stale) Mirror 7: 0xad1151b1 (stale) Mirror 10: 0xad1151b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x114a4): Mirror 6: 0xad1151b1 (stale) Mirror 7: 0xad1151b1 (stale) Mirror 10: 0xad1151b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x114a4: skipped 000809 1787116166.556761 write 0x013336 thru 0x01734f (0x401a bytes) 000810 1787116166.561917 mapwrite 0x01f243 thru 0x02d388 (0xe146 bytes) 000813 1787116166.803647 mapwrite 0x00bc6d thru 0x01726c (0xb600 bytes) 000818 1787116166.880976 mapwrite 0x0385ac thru 0x03ffff (0x7a54 bytes) 000821 1787116167.008061 read 0x00f176 thru 0x01b698 (0xc523 bytes) 000822 1787116167.039301 MIRROR_RESYNC 4 mirrors 000823 1787116169.604188 mapread 0x0007e0 thru 0x00bbe7 (0xb408 bytes) 000824 1787116169.644096 read 0x0391c9 thru 0x03ffff (0x6e37 bytes) 000826 1787116169.661320 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000828 1787116170.207665 write 0x030758 thru 0x03e61a (0xdec3 bytes) 000832 1787116170.314715 write 0x01e70e thru 0x023707 (0x4ffa bytes) 000833 1787116170.329193 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x2b147950 (stale) Mirror 10: 0x2b147950 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x156b6d52 Mirror 11: 0x156b6d52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000834 1787116172.352539 read 0x023036 thru 0x02efd9 (0xbfa4 bytes) 000835 1787116172.385303 trunc from 0x040000 to 0x023796 (0x1c869 bytes) 000836 1787116172.572060 MIRROR_RESYNC 4 mirrors 000838 1787116175.431755 mapwrite 0x0019be thru 0x002b35 (0x1178 bytes) 000839 1787116175.649611 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23797): Mirror 7: 0x94acb5da (stale) Mirror 11: 0x94acb5da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23797): Mirror 7: 0x94acb5da (stale) Mirror 11: 0x94acb5da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23797: skipped 000840 1787116176.034166 write 0x03b364 thru 0x03ffff (0x4c9c bytes) 000843 1787116176.044527 trunc from 0x040000 to 0x01c649 (0x239b6 bytes) 000844 1787116176.071456 write 0x00c184 thru 0x014c4b (0x8ac8 bytes) 000846 1787116176.093326 write 0x00905a thru 0x00909e (0x45 bytes) 000847 1787116176.104751 mapread 0x001bc2 thru 0x008bf2 (0x7031 bytes) 000848 1787116176.146506 trunc from 0x01c64a to 0x01f28d (0x2c44 bytes) 000850 1787116176.167350 mapread 0x01a732 thru 0x01f28d (0x4b5c bytes) 000851 1787116176.197343 write 0x036c95 thru 0x03ffff (0x936b bytes) 000853 1787116176.207724 read 0x01fcd0 thru 0x02a8bd (0xabee bytes) 000855 1787116176.230130 mapread 0x0093f0 thru 0x0167af (0xd3c0 bytes) 000856 1787116176.248939 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xc44c6d89 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xc44c6d89 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000857 1787116176.719080 mapwrite 0x02c953 thru 0x038973 (0xc021 bytes) 000860 1787116176.871695 trunc from 0x040000 to 0x01d1b1 (0x22e4e bytes) 000862 1787116176.905832 trunc from 0x01d1b2 to 0x00384d (0x19964 bytes) 000863 1787116176.928488 write 0x019ace thru 0x01b1dd (0x1710 bytes) 000864 1787116176.937150 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b1de): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b1de): Mirror 11: 0xf2614177 (stale) CRC-32 checksum value for chunk [0, 0x1b1de): Mirror 1: 0xa124143d Mirror 12: 0xa124143d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b1de: skipped 000866 1787116178.770534 trunc from 0x01b1de to 0x03cc63 (0x21a86 bytes) 000867 1787116178.908032 mapread 0x0128b5 thru 0x01b9e9 (0x9135 bytes) 000868 1787116178.933046 mapwrite 0x022816 thru 0x0278c7 (0x50b2 bytes) 000870 1787116178.986921 read 0x00a981 thru 0x00e004 (0x3684 bytes) 000871 1787116178.993608 read 0x011a70 thru 0x01ec29 (0xd1ba bytes) 000872 1787116179.001193 MIRROR_RESYNC 3 mirrors 000873 1787116180.853315 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000874 1787116181.342274 mapread 0x0160de thru 0x017ddc (0x1cff bytes) 000875 1787116181.359019 mapread 0x038ddf thru 0x03cc63 (0x3e85 bytes) 000876 1787116181.369986 mapwrite 0x026e89 thru 0x02dcef (0x6e67 bytes) 000879 1787116181.554147 mapread 0x01e5f9 thru 0x01fa8e (0x1496 bytes) 000881 1787116181.570309 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cc64): Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc64): Mirror 12: 0xb790cb12 (stale) CRC-32 checksum value for chunk [0, 0x3cc64): Mirror 1: 0x35a4efe9 Mirror 13: 0x35a4efe9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc64: skipped 000882 1787116183.032825 write 0x013cf3 thru 0x016575 (0x2883 bytes) 000883 1787116183.177891 write 0x0016e4 thru 0x00852a (0x6e47 bytes) 000886 1787116183.206734 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cc64): Mirror 13: 0x35a4efe9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc64): Mirror 13: 0x35a4efe9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc64: skipped 000887 1787116183.801586 trunc from 0x03cc64 to 0x02559e (0x176c5 bytes) 000889 1787116183.859027 MIRROR_RESYNC 2 mirrors 000890 1787116184.952475 mapwrite 0x007b6a thru 0x012bbe (0xb055 bytes) 000891 1787116185.287386 mapread 0x023e88 thru 0x02559e (0x1717 bytes) 000895 1787116185.312892 write 0x023573 thru 0x02d57e (0xa00c bytes) 000897 1787116185.321215 write 0x033356 thru 0x03ffff (0xccaa bytes) 000898 1787116185.323507 read 0x008e39 thru 0x016e93 (0xe05b bytes) 000899 1787116185.324268 trunc from 0x040000 to 0x01b1a7 (0x24e58 bytes) 000900 1787116185.364641 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000902 1787116185.592945 mapread 0x015645 thru 0x01b1a7 (0x5b63 bytes) 000903 1787116185.615648 trunc from 0x01b1a8 to 0x020c53 (0x5aac bytes) 000904 1787116185.667977 write 0x004d9a thru 0x010459 (0xb6c0 bytes) 000907 1787116185.690656 write 0x0320b0 thru 0x03b867 (0x97b8 bytes) 000910 1787116185.698828 trunc from 0x03b868 to 0x03c028 (0x7c1 bytes) 000912 1787116185.735885 write 0x03a755 thru 0x03aa0c (0x2b8 bytes) 000913 1787116185.737270 mapread 0x0319fa thru 0x03702d (0x5634 bytes) 000914 1787116185.749184 trunc from 0x03c029 to 0x03056e (0xbaba bytes) 000915 1787116185.790705 read 0x025637 thru 0x03056e (0xaf38 bytes) 000918 1787116185.825189 MIRROR_EXTEND to 2 mirrors 000919 1787116187.114548 read 0x0167be thru 0x020117 (0x995a bytes) 000920 1787116187.186365 write 0x01524e thru 0x018112 (0x2ec5 bytes) 000922 1787116187.361408 write 0x002e25 thru 0x0049d4 (0x1bb0 bytes) 000923 1787116187.387441 write 0x01381f thru 0x0156ba (0x1e9c bytes) 000924 1787116187.399675 MIRROR_RESYNC 2 mirrors 000925 1787116188.475179 mapread 0x003be1 thru 0x00854e (0x496e bytes) 000926 1787116188.515963 write 0x001b0f thru 0x00402a (0x251c bytes) 000929 1787116188.612470 mapread 0x018b76 thru 0x01b4eb (0x2976 bytes) 000932 1787116188.627166 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000936 1787116188.854684 write 0x026147 thru 0x02de61 (0x7d1b bytes) 000938 1787116188.881892 MIRROR_EXTEND to 2 mirrors 000943 1787116190.590101 read 0x0172cc thru 0x01d3de (0x6113 bytes) 000944 1787116190.609590 read 0x004794 thru 0x009759 (0x4fc6 bytes) 000945 1787116190.630611 mapwrite 0x01cbb4 thru 0x01f153 (0x25a0 bytes) 000947 1787116190.868662 trunc from 0x03056f to 0x00939c (0x271d2 bytes) 000950 1787116190.908739 read 0x003999 thru 0x00939c (0x5a04 bytes) 000951 1787116190.925015 MIRROR_RESYNC 2 mirrors 000953 1787116192.028193 trunc from 0x00939d to 0x02bf52 (0x22bb6 bytes) 000955 1787116192.214695 read 0x0238fd thru 0x02bf52 (0x8656 bytes) 000956 1787116192.226088 mapwrite 0x00d504 thru 0x01133b (0x3e38 bytes) 000957 1787116192.386136 mapread 0x012bb0 thru 0x01f602 (0xca53 bytes) 000958 1787116192.400949 read 0x012047 thru 0x01bdbe (0x9d78 bytes) 000959 1787116192.401838 write 0x017bc0 thru 0x017cbe (0xff bytes) 000960 1787116192.417754 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b2:0x0]: Invalid argument (22) 000961 1787116192.666540 trunc from 0x02bf53 to 0x008414 (0x23b3e bytes) 000962 1787116192.691694 write 0x01303d thru 0x01e453 (0xb417 bytes) 000964 1787116192.699647 mapread 0x012254 thru 0x01e453 (0xc200 bytes) 000965 1787116192.717080 trunc from 0x01e454 to 0x02c926 (0xe4d3 bytes) 000967 1787116192.752548 trunc from 0x02c927 to 0x005fce (0x26958 bytes) 000969 1787116192.794098 mapwrite 0x010f85 thru 0x01330c (0x2388 bytes) 000970 1787116192.883116 mapread 0x00af2c thru 0x01330c (0x83e1 bytes) 000972 1787116192.888852 read 0x012654 thru 0x01330c (0xcb9 bytes) 000973 1787116192.896496 read 0x006c7e thru 0x01330c (0xc68f bytes) 000975 1787116192.913091 mapwrite 0x007f00 thru 0x008796 (0x897 bytes) 000978 1787116193.001229 write 0x03edab thru 0x03ffff (0x1255 bytes) 000981 1787116193.007669 trunc from 0x040000 to 0x0035ff (0x3ca00 bytes) 000983 1787116193.051365 trunc from 0x003600 to 0x008134 (0x4b35 bytes) 000984 1787116193.091800 mapwrite 0x00d31e thru 0x010f79 (0x3c5c bytes) 000987 1787116193.234071 write 0x03314d thru 0x037849 (0x46fd bytes) 000989 1787116193.236615 mapwrite 0x01e162 thru 0x02dd2a (0xfbc9 bytes) 000993 1787116193.366547 mapwrite 0x036946 thru 0x03ffff (0x96ba bytes) 000998 1787116193.492542 read 0x02784f thru 0x031d7e (0xa530 bytes) All operations completed A-OK! PASS 70a (100s) == sanity-flr test 71: check mirror extend parent fid ==== 01:09:59 (1787116199) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 01:10:02 (1787116202) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.256965 s, 8.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180565 s, 5.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 01:10:12 (1787116212) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3266976 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8693856 .. failed Extending file size to 7976192 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8784896 .. Extending file size to 8960032 .. Extending file size to 9040160 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 57311 57312 57315 57316 57318 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) 192.168.203.153@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:11:34 (1787116294) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2272672 .. Extending file size to 8391424 .. Extending file size to 3060576 .. Extending file size to 8636192 .. Extending file size to 4202912 .. Extending file size to 4974464 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5905664 .. Extending file size to 7238624 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8164192 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8895520 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8220352 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8557664 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58844 58845 58846 58849 58851 done 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) 192.168.203.153@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:12:59 (1787116379) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111925 s, 18.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.323077 s, 6.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe22aa31c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:13:17 (1787116397) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0832282 s, 25.2 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 01:13:25 (1787116405) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f9:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:13:34 (1787116414) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.275402 s, 7.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:13:43 (1787116423) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.443801 s, 4.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:13:52 (1787116432) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.284604 s, 7.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:14:25 (1787116465) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg353-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764089 s, 13.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:15:23 (1787116523) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg353-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.317745 s, 6.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:16:27 (1787116587) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg353-server mds-ost sync done. Creating new pool test_204f oleg353-server: Pool lustre.test_204f created Adding targets to pool oleg353-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg353-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.400044 s, 5.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg353-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg353-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg353-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:17:48 (1787116668) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:17:57 (1787116677) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:18:05 (1787116685) oleg353-server: Pool lustre.test_206 created Adding targets to pool oleg353-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg353-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg353-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg353-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg353-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:18:30 (1787116710) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082747 s, 12.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762766 s, 13.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1b3b4831 Mirror 3: 0x1b3b4831 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:18:40 (1787116720) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:18:44 (1787116724) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:18:47 (1787116727) Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0794188 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1fe0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ad0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ad0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ad0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1f10000 Read this (4 bytes): BBBB 192.168.203.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg353-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:19:26 (1787116766) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.042976 s, 191 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:19:35 (1787116775) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671254 s, 15.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x21e lmm_fid: [0x200000404:0x21e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x21e lmm_fid: [0x200000404:0x21e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834462 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-19 01:19:37.000000000 -0400 Modify: 2026-08-19 01:19:37.000000000 -0400 Change: 2026-08-19 01:19:39.000000000 -0400 Birth: 2026-08-19 01:19:38.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:19:45 (1787116785) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:19:49 (1787116789) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.905798 s, 45.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.20747 s, 19.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 19 01:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x220:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 19 01:19 /mnt/lustre/f211.sanity-flr PASS 211 (12s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:20:01 (1787116801) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:20:11 (1787116811) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1842 sec =========== 01:20:21 (1787116821) === sanity-flr: start cleanup 01:20:22 (1787116822) === === sanity-flr: finish cleanup 01:20:29 (1787116829) ===