-----============= acceptance-small: sanity-flr ============----- Sat Jul 4 00:06:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg360-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 00:06:23 (1783137983) === oleg360-client.virtnet: executing check_config_client /mnt/lustre oleg360-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg360-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff894a0701e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff894a0701e800.idle_timeout=debug disable quota as required oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 00:06:57 (1783138017) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7613] == sanity-flr test 0a: lfs mirror create with -N option == 00:06:59 (1783138019) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 00:07:16 (1783138036) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 00:07:19 (1783138039) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 00:07:22 (1783138042) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 00:07:44 (1783138064) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 00:07:48 (1783138068) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 00:07:51 (1783138071) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff894a0701e800.stats=clear osc.lustre-OST0001-osc-ffff894a0701e800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 00:07:59 (1783138079) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 00:08:09 (1783138089) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 00:08:17 (1783138097) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 00:08:20 (1783138100) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 00:08:27 (1783138107) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 00:08:37 (1783138117) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 00:08:44 (1783138124) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 00:08:47 (1783138127) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 00:08:54 (1783138134) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 00:09:04 (1783138144) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 00:09:10 (1783138150) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.603481 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124544 s, 8.4 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 00:09:20 (1783138160) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.525524 s, 39.9 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 00:09:28 (1783138168) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 00:09:37 (1783138177) Stopping /mnt/lustre-ost1 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff894a0701e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff894a0701e800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg360-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff894a0701e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff894a0701e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg360-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (65s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 00:10:42 (1783138242) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.592103 s, 10.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff894a0701e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff894a0701e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg360-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 00:11:22 (1783138282) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff894a0701e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff894a0701e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg360-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff894a0701e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff894a0701e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg360-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (71s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:12:33 (1783138353) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.52486 s, 11.6 MB/s -rw-r--r-- 1 root root 52428800 Jul 4 00:12 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.008s user 0m0.001s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff894a0701e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff894a0701e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.042s user 0m0.000s sys 0m0.170s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg360-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff894a0701e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff894a0701e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.970s user 0m0.001s sys 0m0.177s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg360-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff894a0701e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff894a0701e800.ost_server_uuid in FULL state after 0 sec PASS 33b (85s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:13:58 (1783138438) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:14:01 (1783138441) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:14:04 (1783138444) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 00:14:07 (1783138447) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 00:14:14 (1783138454) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.50252 s, 25.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:14:26 (1783138466) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212962 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023175 s, 45.2 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 00:14:33 (1783138473) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 00:14:40 (1783138480) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00123783 s, 827 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0023148 s, 442 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00508573 s, 201 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0105248 s, 97.3 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 00:14:49 (1783138489) create test files with size 15743141 .. e685866f53928f40af5c7d2a109f1d7b - 370f8778cdc04b32be57e09c28ecc158 - e8dcbf9c01348c9c47f9d411029729a3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg360-client.virtnet /mnt/lustre (opts:) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 00:15:10 (1783138510) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 00:15:23 (1783138523) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 00:15:29 (1783138529) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110231 s, 9.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304028 s, 34.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 00:15:38 (1783138538) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181913 s, 11.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.340486 s, 12.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228681 s, 9.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 00:15:55 (1783138555) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 00:15:58 (1783138558) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 00:16:01 (1783138561) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302629 s, 34.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 00:16:08 (1783138568) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 00:16:11 (1783138571) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 00:16:19 (1783138579) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.302198 s, 34.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg360-client.virtnet /mnt/lustre (opts:) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 00:16:29 (1783138589) llite.lustre-ffff894a1295a000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.272802 s, 38.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg360-client.virtnet /mnt/lustre (opts:) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff894a07ff3000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 00:16:42 (1783138602) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 00:17:18 (1783138638) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 00:17:52 (1783138672) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 00:17:55 (1783138675) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.196752 s, 16.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.136743 s, 23.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x45ec5162 (nosync,stale) Mirror 2: 0x45ec5162 (nosync,stale) ** verify mirror 2 doesn't change original checksum: cfe40684e43944a2414cc85c217dc5c6 - mirror 1 checksum: ffd63b6f00caf0222e01c6a7d105591a - mirror 2 checksum: cfe40684e43944a2414cc85c217dc5c6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-04T00:17:59-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-04T00:17:59-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: cfe40684e43944a2414cc85c217dc5c6 - mirror 1 checksum: ffd63b6f00caf0222e01c6a7d105591a - mirror 2 checksum: ffd63b6f00caf0222e01c6a7d105591a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-04T00:17:59-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-04T00:17:59-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 00:18:06 (1783138686) Failing mds1 on oleg360-server Stopping /mnt/lustre-mds1 (opts:) on oleg360-server 00:18:12 (1783138692) shut down facet: mds1 facet_host: oleg360-server facet_failover_host: oleg360-server Failover mds1 to oleg360-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 00:18:35 (1783138715) targets are mounted 00:18:35 (1783138715) facet_failover done oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368455 s, 28.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (46s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 00:18:52 (1783138732) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 00:18:55 (1783138735) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 00:18:59 (1783138739) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0107515 s, 1.5 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 00:19:06 (1783138746) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 00:19:09 (1783138749) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 00:19:12 (1783138752) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 00:19:32 (1783138772) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:20:00 (1783138800) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 00:20:23 (1783138823) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 00:20:47 (1783138847) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 00:20:55 (1783138855) mirror create pid 53762 mirror split pid 53763 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53762 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53763 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 00:22:01 (1783138921) Seed set to 8922 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x19f7e 000001 1783138922.979418 trunc from 00000000 to 0x019f7d (0x19f7e bytes) 000002 1783138923.002700 read 0x012298 thru 0x019f7d (0x7ce6 bytes) truncating to largest ever: 0x1b2b5 000003 1783138923.015839 trunc from 0x019f7e to 0x01b2b4 (0x1337 bytes) 000008 1783138923.056423 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3d5f6 000009 1783138923.783996 trunc from 0x01b2b5 to 0x03d5f5 (0x22341 bytes) 000010 1783138923.879837 read 0x01ef0d thru 0x02158a (0x267e bytes) 000012 1783138923.890313 mapread 0x0061f3 thru 0x00a423 (0x4231 bytes) 000015 1783138923.918913 mapwrite 0x023d6c thru 0x033af1 (0xfd86 bytes) 000016 1783138924.044960 read 0x038109 thru 0x03d5f5 (0x54ed bytes) 000018 1783138924.061917 trunc from 0x03d5f6 to 0x00a3c2 (0x33233 bytes) 000019 1783138924.091262 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000021 1783138924.289433 trunc from 0x00a3c3 to 0x03884f (0x2e48d bytes) 000024 1783138924.322983 trunc from 0x038850 to 0x0025ae (0x362a1 bytes) 000025 1783138924.347101 mapread 0x0003db thru 0x0025ae (0x21d4 bytes) 000027 1783138924.362470 read 0x0017a3 thru 0x0025ae (0xe0c bytes) 000028 1783138924.363518 trunc from 0x0025af to 0x02de08 (0x2b85a bytes) 000029 1783138924.397536 read 0x01938f thru 0x0236c9 (0xa33b bytes) 000032 1783138924.419778 write 0x003183 thru 0x00e84c (0xb6ca bytes) 000033 1783138924.422048 mapread 0x0238d9 thru 0x02cc36 (0x935e bytes) 000035 1783138924.429047 mapwrite 0x024081 thru 0x029b35 (0x5ab5 bytes) 000037 1783138924.514260 write 0x006ef8 thru 0x00e87f (0x7988 bytes) 000038 1783138924.515512 write 0x0161c5 thru 0x01a32c (0x4168 bytes) truncating to largest ever: 0x3ea36 000039 1783138924.517716 trunc from 0x02de09 to 0x03ea35 (0x10c2d bytes) 000042 1783138924.553108 trunc from 0x03ea36 to 0x0253a4 (0x19691 bytes) 000045 1783138924.586822 mapwrite 0x02fa7e thru 0x033f02 (0x4485 bytes) 000046 1783138924.705468 write 0x03d0af thru 0x03ffff (0x2f51 bytes) 000047 1783138924.706641 write 0x03d5a3 thru 0x03ffff (0x2a5d bytes) 000048 1783138924.707606 mapwrite 0x009c4f thru 0x016a51 (0xce03 bytes) 000049 1783138924.751122 read 0x032415 thru 0x032f4c (0xb38 bytes) 000050 1783138924.751842 mapwrite 0x017b37 thru 0x01f800 (0x7cca bytes) 000056 1783138924.803148 write 0x03813c thru 0x03ffff (0x7ec4 bytes) 000057 1783138924.820181 write 0x03f3b9 thru 0x03ffff (0xc47 bytes) 000060 1783138924.826001 mapwrite 0x03bced thru 0x03d69e (0x19b2 bytes) 000061 1783138924.875638 write 0x011125 thru 0x015d6f (0x4c4b bytes) 000062 1783138924.878697 read 0x015fa7 thru 0x01f31b (0x9375 bytes) 000064 1783138924.881318 write 0x003116 thru 0x005cfa (0x2be5 bytes) 000065 1783138924.883489 trunc from 0x040000 to 0x0195c1 (0x26a3e bytes) 000067 1783138924.908746 read 0x00b2b0 thru 0x0195c1 (0xe312 bytes) 000068 1783138924.909812 trunc from 0x0195c2 to 0x01e9a5 (0x53e4 bytes) 000069 1783138924.932398 write 0x02c84e thru 0x03017b (0x392e bytes) 000070 1783138924.939460 MIRROR_EXTEND to 2 mirrors 000071 1783138925.589302 MIRROR_RESYNC 2 mirrors 000073 1783138925.946008 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000074 1783138926.106591 trunc from 0x03017c to 0x017db5 (0x183c6 bytes) 000079 1783138926.148831 trunc from 0x017db6 to 0x01be3f (0x408a bytes) 000080 1783138926.164963 read 0x00ee69 thru 0x01a884 (0xba1c bytes) 000082 1783138926.181584 mapread 0x00a6d0 thru 0x0179fb (0xd32c bytes) 000085 1783138926.240899 mapread 0x00917c thru 0x011e76 (0x8cfb bytes) 000086 1783138926.246925 write 0x010563 thru 0x0182a3 (0x7d41 bytes) 000087 1783138926.251139 read 0x0012dd thru 0x00b945 (0xa669 bytes) 000089 1783138926.277919 read 0x005e3b thru 0x0118f0 (0xbab6 bytes) 000090 1783138926.293138 MIRROR_EXTEND to 2 mirrors 000091 1783138926.810555 trunc from 0x01be40 to 0x025179 (0x933a bytes) 000092 1783138926.914629 write 0x01fa0c thru 0x02644d (0x6a42 bytes) 000093 1783138926.920103 write 0x022adf thru 0x03077c (0xdc9e bytes) 000095 1783138926.922182 trunc from 0x03077d to 0x033e57 (0x36db bytes) 000097 1783138926.943311 write 0x0269ae thru 0x02f300 (0x8953 bytes) 000098 1783138926.945442 write 0x01978c thru 0x0270c4 (0xd939 bytes) 000099 1783138926.951724 mapread 0x01afdd thru 0x028406 (0xd42a bytes) 000100 1783138926.956710 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000103 1783138927.100064 trunc from 0x033e58 to 0x038189 (0x4332 bytes) 000106 1783138927.119023 write 0x01656b thru 0x023d28 (0xd7be bytes) 000109 1783138927.131911 read 0x018fe5 thru 0x01ddf0 (0x4e0c bytes) 000110 1783138927.132553 trunc from 0x03818a to 0x02d551 (0xac38 bytes) 000111 1783138927.153407 read 0x00864c thru 0x0116e1 (0x9096 bytes) 000113 1783138927.169436 write 0x03d4f3 thru 0x03ffff (0x2b0d bytes) 000115 1783138927.171755 write 0x0127bd thru 0x019b01 (0x7345 bytes) truncating to largest ever: 0x3ed27 000116 1783138927.173843 trunc from 0x040000 to 0x03ed26 (0x12d9 bytes) 000120 1783138927.201794 trunc from 0x03ed27 to 0x018071 (0x26cb5 bytes) 000121 1783138927.255944 mapread 0x015009 thru 0x018071 (0x3069 bytes) 000123 1783138927.258538 write 0x0205c8 thru 0x023f2c (0x3965 bytes) 000124 1783138927.260097 mapread 0x005f3c thru 0x012f59 (0xd01e bytes) 000125 1783138927.261700 mapwrite 0x014cef thru 0x01dabe (0x8dd0 bytes) 000126 1783138927.338527 write 0x00457f thru 0x0113b5 (0xce37 bytes) 000127 1783138927.340632 mapwrite 0x001d3f thru 0x009799 (0x7a5b bytes) 000132 1783138927.391844 write 0x034df2 thru 0x0364d9 (0x16e8 bytes) 000133 1783138927.393104 trunc from 0x0364da to 0x0108fc (0x25bdd bytes) 000135 1783138927.427050 trunc from 0x0108fd to 0x00c8a1 (0x405b bytes) 000136 1783138927.451705 mapread 0x00ac8d thru 0x00c8a1 (0x1c15 bytes) 000137 1783138927.452837 read 0x005304 thru 0x00c8a1 (0x759e bytes) 000138 1783138927.455580 trunc from 0x00c8a2 to 0x009a10 (0x2e91 bytes) 000141 1783138927.485417 write 0x03b5e6 thru 0x03ffff (0x4a1a bytes) 000143 1783138927.488290 write 0x02da6c thru 0x035fa2 (0x8537 bytes) 000144 1783138927.513342 trunc from 0x040000 to 0x025c12 (0x1a3ed bytes) 000148 1783138927.549271 mapread 0x003c61 thru 0x00b3ee (0x778e bytes) 000150 1783138927.554985 MIRROR_EXTEND to 2 mirrors 000152 1783138928.209391 trunc from 0x025c13 to 0x033212 (0xd600 bytes) 000159 1783138928.326032 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000161 1783138928.531140 write 0x0338dc thru 0x039e21 (0x6546 bytes) 000162 1783138928.539289 write 0x001f91 thru 0x00d433 (0xb4a3 bytes) 000168 1783138928.581624 mapwrite 0x034b09 thru 0x03ffff (0xb4f7 bytes) 000171 1783138928.792464 trunc from 0x040000 to 0x01b1d5 (0x24e2a bytes) 000172 1783138928.811679 mapwrite 0x022992 thru 0x0288ee (0x5f5d bytes) 000173 1783138928.887982 trunc from 0x0288ef to 0x00de1f (0x1aacf bytes) 000178 1783138928.942258 read 0x001837 thru 0x009972 (0x813c bytes) 000179 1783138928.943047 mapwrite 0x026925 thru 0x035c19 (0xf2f5 bytes) 000182 1783138929.082017 mapwrite 0x01add3 thru 0x026e2a (0xc058 bytes) 000183 1783138929.179938 read 0x031971 thru 0x035c19 (0x42a9 bytes) 000184 1783138929.180701 read 0x01e461 thru 0x027271 (0x8e11 bytes) 000185 1783138929.183654 mapwrite 0x005f24 thru 0x00f954 (0x9a31 bytes) 000187 1783138929.266387 mapread 0x020eaf thru 0x024de7 (0x3f39 bytes) 000190 1783138929.270989 write 0x029812 thru 0x02ab3e (0x132d bytes) 000191 1783138929.272421 write 0x0365c7 thru 0x03f110 (0x8b4a bytes) 000195 1783138929.278921 trunc from 0x03f111 to 0x00cabd (0x32653 bytes) 000197 1783138929.299330 write 0x02f9bf thru 0x0398f4 (0x9f36 bytes) 000199 1783138929.303408 read 0x00efc9 thru 0x017887 (0x88bf bytes) 000202 1783138929.325482 MIRROR_EXTEND to 2 mirrors 000203 1783138929.819032 read 0x005c49 thru 0x013491 (0xd849 bytes) 000204 1783138929.830771 write 0x030984 thru 0x033130 (0x27ad bytes) 000206 1783138929.910948 MIRROR_RESYNC 2 mirrors 000207 1783138930.510955 read 0x0009b2 thru 0x004f5f (0x45ae bytes) 000208 1783138930.520946 read 0x004b36 thru 0x010d6e (0xc239 bytes) 000213 1783138930.549392 mapread 0x023445 thru 0x0283d9 (0x4f95 bytes) 000214 1783138930.554250 MIRROR_EXTEND to 3 mirrors 000218 1783138931.128719 MIRROR_RESYNC 3 mirrors 000220 1783138931.506824 write 0x017bb0 thru 0x01c1e4 (0x4635 bytes) 000223 1783138931.652273 mapwrite 0x03f10f thru 0x03ffff (0xef1 bytes) 000224 1783138931.757880 read 0x005f3a thru 0x00e79c (0x8863 bytes) 000227 1783138931.778696 mapread 0x0326fc thru 0x03ec92 (0xc597 bytes) 000228 1783138931.802457 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5941494c (stale) Mirror 3: 0x5941494c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2aa74b68 Mirror 4: 0x2aa74b68 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000230 1783138932.735564 read 0x0228ab thru 0x027bbe (0x5314 bytes) 000231 1783138932.766683 write 0x00e787 thru 0x00f732 (0xfac bytes) 000233 1783138933.002990 trunc from 0x040000 to 0x00995a (0x366a5 bytes) 000234 1783138933.032137 trunc from 0x00995b to 0x024b1b (0x1b1c1 bytes) 000235 1783138933.065596 mapread 0x00a549 thru 0x00c05e (0x1b16 bytes) 000236 1783138933.118956 write 0x0111c8 thru 0x01ad49 (0x9b82 bytes) 000237 1783138933.143685 write 0x02a0c2 thru 0x02a9ad (0x8ec bytes) 000239 1783138933.150919 trunc from 0x02a9ae to 0x01bbf8 (0xedb5 bytes) 000242 1783138933.205416 mapwrite 0x009d14 thru 0x00d8b6 (0x3ba3 bytes) 000245 1783138933.304737 read 0x01199d thru 0x01a063 (0x86c7 bytes) 000246 1783138933.308408 read 0x006221 thru 0x00e266 (0x8046 bytes) 000247 1783138933.316807 read 0x00205a thru 0x00cec3 (0xae6a bytes) 000250 1783138933.331923 read 0x00730e thru 0x00bdfb (0x4aee bytes) 000252 1783138933.335796 mapwrite 0x00c324 thru 0x01173c (0x5419 bytes) 000255 1783138933.430554 read 0x0036cc thru 0x0042db (0xc10 bytes) 000257 1783138933.435251 trunc from 0x01bbf9 to 0x037abc (0x1bec4 bytes) 000258 1783138933.465012 mapread 0x005cc5 thru 0x00b044 (0x5380 bytes) 000259 1783138933.466785 mapwrite 0x0295d0 thru 0x039234 (0xfc65 bytes) 000260 1783138933.551988 MIRROR_RESYNC 4 mirrors 000261 1783138934.353563 mapread 0x022075 thru 0x029a73 (0x79ff bytes) 000262 1783138934.372438 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000263 1783138934.693178 write 0x01d25b thru 0x02c750 (0xf4f6 bytes) 000266 1783138934.842846 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39235): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39235): Mirror 2: 0xd6a020e8 (stale) Mirror 4: 0xd6a020e8 (stale) CRC-32 checksum value for chunk [0, 0x39235): Mirror 1: 0xcdb229c8 Mirror 5: 0xcdb229c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39235: skipped 000271 1783138935.956819 mapread 0x0311a5 thru 0x033e45 (0x2ca1 bytes) 000273 1783138935.994841 mapwrite 0x005f30 thru 0x00f730 (0x9801 bytes) 000274 1783138936.263825 mapread 0x00f06c thru 0x0111dc (0x2171 bytes) 000276 1783138936.284709 write 0x01b611 thru 0x02ac96 (0xf686 bytes) 000277 1783138936.323296 read 0x01e6ee thru 0x02af44 (0xc857 bytes) 000278 1783138936.324402 write 0x018435 thru 0x01d1fd (0x4dc9 bytes) 000279 1783138936.338056 mapwrite 0x00649e thru 0x00d869 (0x73cc bytes) 000280 1783138936.411756 mapwrite 0x00bed1 thru 0x011adf (0x5c0f bytes) 000282 1783138936.456002 write 0x01cc6f thru 0x0264dd (0x986f bytes) 000283 1783138936.458282 mapwrite 0x01028b thru 0x01b638 (0xb3ae bytes) 000284 1783138936.521087 mapread 0x021e0c thru 0x0238fb (0x1af0 bytes) 000287 1783138936.525941 read 0x0351cc thru 0x039234 (0x4069 bytes) 000289 1783138936.536311 mapwrite 0x013600 thru 0x0179ec (0x43ed bytes) 000290 1783138936.592033 write 0x00f8ac thru 0x0171dc (0x7931 bytes) 000293 1783138936.602287 read 0x0045b0 thru 0x0078ba (0x330b bytes) 000296 1783138936.606934 trunc from 0x039235 to 0x01abd4 (0x1e660 bytes) 000297 1783138936.640076 mapread 0x007de6 thru 0x009b88 (0x1da3 bytes) 000299 1783138936.649467 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1abd5): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1abd5): Mirror 2: 0x7d6c0ff0 (stale) Mirror 4: 0x7d6c0ff0 (stale) Mirror 5: 0x7d6c0ff0 (stale) CRC-32 checksum value for chunk [0, 0x1abd5): Mirror 1: 0x6c772331 Mirror 6: 0x6c772331 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1abd5: skipped 000300 1783138937.693134 write 0x01bca5 thru 0x01e4b8 (0x2814 bytes) 000304 1783138937.789588 mapwrite 0x02b573 thru 0x032b7b (0x7609 bytes) 000305 1783138937.887081 write 0x006e13 thru 0x00ca8f (0x5c7d bytes) 000306 1783138937.894800 mapwrite 0x015e5b thru 0x01d3fb (0x75a1 bytes) 000308 1783138937.944531 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32b7c): Mirror 2: 0x2285978 (stale) Mirror 4: 0x2285978 (stale) Mirror 5: 0x33b4c88d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b7c): Mirror 2: 0x2285978 (stale) Mirror 4: 0x2285978 (stale) Mirror 5: 0x33b4c88d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b7c: skipped 000309 1783138938.326313 mapwrite 0x02374d thru 0x023983 (0x237 bytes) 000310 1783138938.366605 mapread 0x00a5d8 thru 0x00de8d (0x38b6 bytes) 000311 1783138938.382109 MIRROR_RESYNC 4 mirrors 000312 1783138939.122442 mapread 0x016192 thru 0x026022 (0xfe91 bytes) 000314 1783138939.177538 mapwrite 0x02475e thru 0x0299e6 (0x5289 bytes) 000318 1783138939.417030 trunc from 0x032b7c to 0x01aab1 (0x180ca bytes) 000319 1783138939.474556 read 0x00a462 thru 0x00fa9b (0x563a bytes) 000320 1783138939.486546 read 0x000e54 thru 0x00abc6 (0x9d73 bytes) 000322 1783138939.498686 mapread 0x0109ba thru 0x0189c1 (0x8008 bytes) 000324 1783138939.501304 read 0x005214 thru 0x00f207 (0x9ff4 bytes) 000326 1783138939.503640 write 0x01bcf6 thru 0x02a961 (0xec6c bytes) 000327 1783138939.507040 write 0x020b35 thru 0x02ceb5 (0xc381 bytes) 000328 1783138939.509494 mapread 0x004be0 thru 0x00f581 (0xa9a2 bytes) 000329 1783138939.512482 mapread 0x016150 thru 0x0226e9 (0xc59a bytes) 000331 1783138939.514980 mapread 0x003c01 thru 0x00a9a7 (0x6da7 bytes) 000334 1783138939.519667 write 0x00c6df thru 0x01685a (0xa17c bytes) 000335 1783138939.521906 write 0x01f1d5 thru 0x02ac59 (0xba85 bytes) 000336 1783138939.525069 read 0x00095c thru 0x004af0 (0x4195 bytes) 000337 1783138939.531381 MIRROR_RESYNC 4 mirrors 000338 1783138940.343994 mapwrite 0x03d4c2 thru 0x03fa48 (0x2587 bytes) 000340 1783138940.452602 mapwrite 0x032600 thru 0x03453a (0x1f3b bytes) 000341 1783138940.496432 mapwrite 0x011553 thru 0x01cf2c (0xb9da bytes) 000342 1783138940.643685 read 0x035906 thru 0x03ce23 (0x751e bytes) 000343 1783138940.647673 write 0x02d4ab thru 0x036e39 (0x998f bytes) 000344 1783138940.650832 read 0x01af5d thru 0x022123 (0x71c7 bytes) 000346 1783138940.660524 mapread 0x027ee1 thru 0x0319f3 (0x9b13 bytes) 000347 1783138940.680502 mapread 0x01677b thru 0x01807d (0x1903 bytes) 000348 1783138940.685905 mapwrite 0x03ee07 thru 0x03ffff (0x11f9 bytes) 000350 1783138940.821304 mapread 0x038cbf thru 0x03ffff (0x7341 bytes) 000353 1783138940.828195 write 0x030bf5 thru 0x039f90 (0x939c bytes) 000354 1783138940.832252 read 0x03bab1 thru 0x03ffff (0x454f bytes) 000357 1783138940.837469 read 0x0227fa thru 0x0313f9 (0xec00 bytes) 000359 1783138940.840682 mapread 0x02c392 thru 0x02cea0 (0xb0f bytes) 000361 1783138940.842462 mapread 0x033cb7 thru 0x039585 (0x58cf bytes) 000363 1783138940.848669 mapread 0x03c669 thru 0x03f15f (0x2af7 bytes) 000365 1783138940.851936 mapwrite 0x02639e thru 0x026653 (0x2b6 bytes) 000366 1783138940.929967 mapread 0x00b650 thru 0x015ffd (0xa9ae bytes) 000368 1783138940.931742 mapread 0x006df0 thru 0x00e277 (0x7488 bytes) 000369 1783138940.934436 mapwrite 0x02a420 thru 0x02b7bf (0x13a0 bytes) 000370 1783138940.975750 write 0x02441a thru 0x02c9b6 (0x859d bytes) 000372 1783138940.979196 write 0x02a3d8 thru 0x03009a (0x5cc3 bytes) 000377 1783138940.984044 mapread 0x002cd4 thru 0x0089a9 (0x5cd6 bytes) 000378 1783138940.985545 write 0x02b0bc thru 0x035dec (0xad31 bytes) 000380 1783138940.988307 mapread 0x01ddbf thru 0x026e5f (0x90a1 bytes) 000381 1783138940.992635 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf43bb43a (stale) Mirror 4: 0xf43bb43a (stale) Mirror 5: 0xf43bb43a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x752c5c17 Mirror 6: 0x752c5c17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000382 1783138941.970624 MIRROR_RESYNC 5 mirrors 000383 1783138942.962741 mapread 0x01e25a thru 0x027e0b (0x9bb2 bytes) 000385 1783138942.984941 mapwrite 0x034d4c thru 0x03e064 (0x9319 bytes) 000386 1783138943.183752 mapwrite 0x01c43b thru 0x0201ab (0x3d71 bytes) 000387 1783138943.233068 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x752c5c17 (stale) Mirror 4: 0x752c5c17 (stale) Mirror 5: 0x752c5c17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x752c5c17 (stale) Mirror 4: 0x752c5c17 (stale) Mirror 5: 0x752c5c17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000388 1783138943.653915 trunc from 0x040000 to 0x023179 (0x1ce86 bytes) 000389 1783138943.673301 trunc from 0x02317a to 0x03c697 (0x1951e bytes) 000393 1783138943.695719 mapwrite 0x01ad44 thru 0x02473e (0x99fb bytes) 000394 1783138943.787702 mapread 0x02643d thru 0x036276 (0xfe3a bytes) 000396 1783138943.807508 mapread 0x02ee0a thru 0x03c697 (0xd88e bytes) 000397 1783138943.820227 write 0x003c75 thru 0x01037a (0xc706 bytes) 000401 1783138943.843289 mapread 0x034df3 thru 0x039ec7 (0x50d5 bytes) 000402 1783138943.844457 mapread 0x00e02d thru 0x0107dc (0x27b0 bytes) 000403 1783138943.851230 trunc from 0x03c698 to 0x016923 (0x25d74 bytes) 000404 1783138943.874668 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x16924): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16924): Mirror 2: 0x48628dcd (stale) Mirror 4: 0x48628dcd (stale) Mirror 5: 0x48628dcd (stale) CRC-32 checksum value for chunk [0, 0x16924): Mirror 1: 0x7391f49a Mirror 6: 0x7391f49a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16924: skipped 000406 1783138944.816701 MIRROR_RESYNC 5 mirrors 000408 1783138945.644430 mapread 0x013e0e thru 0x016923 (0x2b16 bytes) 000411 1783138945.663510 mapwrite 0x013236 thru 0x015d3e (0x2b09 bytes) 000413 1783138945.797440 read 0x010540 thru 0x016923 (0x63e4 bytes) 000414 1783138945.798174 read 0x01007b thru 0x016923 (0x68a9 bytes) 000418 1783138945.800774 trunc from 0x016924 to 0x012e30 (0x3af3 bytes) 000420 1783138945.827039 trunc from 0x012e31 to 0x001132 (0x11cfe bytes) 000421 1783138945.849918 write 0x00990a thru 0x015d99 (0xc490 bytes) 000422 1783138945.862398 MIRROR_RESYNC 5 mirrors 000423 1783138946.806420 trunc from 0x015d9a to 0x028b89 (0x12df0 bytes) 000425 1783138946.938374 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x28b8a): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28b8a): Mirror 2: 0x280b3425 (stale) Mirror 4: 0x280b3425 (stale) Mirror 5: 0x280b3425 (stale) Mirror 6: 0x280b3425 (stale) CRC-32 checksum value for chunk [0, 0x28b8a): Mirror 1: 0x280b3425 Mirror 7: 0x280b3425 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28b8a: skipped 000426 1783138947.809596 mapwrite 0x01762f thru 0x01e112 (0x6ae4 bytes) 000427 1783138948.017710 read 0x018f13 thru 0x02666b (0xd759 bytes) 000428 1783138948.038746 read 0x012e25 thru 0x020ba3 (0xdd7f bytes) 000431 1783138948.044958 trunc from 0x028b8a to 0x001852 (0x27337 bytes) 000432 1783138948.078053 MIRROR_RESYNC 6 mirrors 000433 1783138948.975948 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000435 1783138949.340133 write 0x014c88 thru 0x016aa5 (0x1e1e bytes) 000436 1783138949.423097 mapwrite 0x038c13 thru 0x038c84 (0x72 bytes) 000437 1783138949.542096 trunc from 0x038c85 to 0x03bebb (0x3237 bytes) 000438 1783138949.564585 trunc from 0x03bebc to 0x03984c (0x266f bytes) 000439 1783138949.593151 read 0x02a1b2 thru 0x032800 (0x864f bytes) 000440 1783138949.612167 MIRROR_RESYNC 5 mirrors 000441 1783138950.489402 mapwrite 0x02473a thru 0x0345ba (0xfe81 bytes) 000442 1783138950.641404 mapread 0x01bbd6 thru 0x020bc7 (0x4ff2 bytes) 000446 1783138950.655702 trunc from 0x03984d to 0x01d19c (0x1c6b0 bytes) 000448 1783138950.679128 mapread 0x00a084 thru 0x014773 (0xa6f0 bytes) 000452 1783138950.695513 read 0x0025de thru 0x00936d (0x6d90 bytes) 000455 1783138950.704653 mapwrite 0x03251d thru 0x0335ef (0x10d3 bytes) 000458 1783138950.779422 mapwrite 0x01777a thru 0x0265a4 (0xee2b bytes) 000462 1783138950.835280 write 0x03e3ff thru 0x03ffff (0x1c01 bytes) 000463 1783138950.837468 mapread 0x01cda8 thru 0x02b84b (0xeaa4 bytes) 000464 1783138950.849395 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd7468bfd (stale) Mirror 6: 0xd7468bfd (stale) Mirror 7: 0xd7468bfd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd7468bfd (stale) Mirror 6: 0xd7468bfd (stale) Mirror 7: 0xd7468bfd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000466 1783138951.146829 read 0x03e446 thru 0x03ffff (0x1bba bytes) 000469 1783138951.153194 mapwrite 0x03fa98 thru 0x03ffff (0x568 bytes) 000470 1783138951.186835 mapread 0x027fb8 thru 0x02cbfc (0x4c45 bytes) 000474 1783138951.209971 write 0x01d1f6 thru 0x02a11a (0xcf25 bytes) 000475 1783138951.215559 trunc from 0x040000 to 0x002f4e (0x3d0b1 bytes) 000478 1783138951.236755 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f4f): Mirror 6: 0xff85fb10 (stale) Mirror 7: 0xff85fb10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f4f): Mirror 6: 0xff85fb10 (stale) Mirror 7: 0xff85fb10 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f4f: skipped 000479 1783138951.496001 write 0x00bba7 thru 0x00c82c (0xc86 bytes) 000482 1783138951.509025 trunc from 0x00c82d to 0x03b662 (0x2ee36 bytes) 000483 1783138951.545577 write 0x01ec04 thru 0x02b15e (0xc55b bytes) 000484 1783138951.584611 write 0x023d29 thru 0x02ce5e (0x9136 bytes) 000486 1783138951.593118 read 0x000cfb thru 0x00cea9 (0xc1af bytes) 000488 1783138951.609916 trunc from 0x03b663 to 0x00527f (0x363e3 bytes) 000489 1783138951.640778 trunc from 0x005280 to 0x00c59c (0x731d bytes) 000491 1783138951.667363 mapread 0x00a734 thru 0x00c59c (0x1e69 bytes) 000492 1783138951.702832 mapwrite 0x026ba9 thru 0x033fe1 (0xd439 bytes) 000495 1783138951.819419 mapread 0x02dec9 thru 0x02ef1e (0x1056 bytes) 000496 1783138951.821257 read 0x032617 thru 0x033fe1 (0x19cb bytes) 000499 1783138951.825618 mapwrite 0x022270 thru 0x0315c4 (0xf355 bytes) 000502 1783138951.874512 write 0x012503 thru 0x018e72 (0x6970 bytes) 000503 1783138951.885243 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 6: 0x290efced (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 6: 0x290efced (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fe2: skipped 000504 1783138952.137435 mapwrite 0x01af13 thru 0x01d03d (0x212b bytes) 000505 1783138952.198826 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 6: 0x290efced (stale) CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 1: 0x11e49f1b Mirror 7: 0x11e49f1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fe2: skipped 000507 1783138952.830687 read 0x01faf2 thru 0x025a72 (0x5f81 bytes) 000511 1783138952.842280 trunc from 0x033fe2 to 0x010753 (0x2388e bytes) 000517 1783138952.931627 mapread 0x00e16e thru 0x010753 (0x25e6 bytes) 000518 1783138952.953789 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x10754): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10754): Mirror 6: 0x8cdba282 (stale) Mirror 7: 0x8cdba282 (stale) CRC-32 checksum value for chunk [0, 0x10754): Mirror 1: 0x8cdba282 Mirror 8: 0x8cdba282 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10754: skipped 000520 1783138953.657574 trunc from 0x010754 to 0x0245db (0x13e88 bytes) 000523 1783138953.753917 mapwrite 0x036c25 thru 0x03ffff (0x93db bytes) 000527 1783138953.876720 trunc from 0x040000 to 0x027974 (0x1868b bytes) 000529 1783138953.901179 trunc from 0x027975 to 0x028ec5 (0x1551 bytes) 000532 1783138953.935506 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x28ec6): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ec6): Mirror 6: 0x194b97bb (stale) Mirror 7: 0x6beabba4 (stale) Mirror 8: 0x136d2686 (stale) CRC-32 checksum value for chunk [0, 0x28ec6): Mirror 1: 0x136d2686 Mirror 9: 0x136d2686 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ec6: skipped 000534 1783138954.838741 trunc from 0x028ec6 to 0x02f093 (0x61ce bytes) 000535 1783138954.957461 MIRROR_RESYNC 5 mirrors 000536 1783138955.857308 mapread 0x003fe0 thru 0x00c3e0 (0x8401 bytes) 000538 1783138955.879146 mapwrite 0x013df4 thru 0x014954 (0xb61 bytes) 000540 1783138956.061548 write 0x039441 thru 0x03ffff (0x6bbf bytes) 000541 1783138956.066772 write 0x01d251 thru 0x023916 (0x66c6 bytes) 000542 1783138956.084435 trunc from 0x040000 to 0x0244a8 (0x1bb57 bytes) 000543 1783138956.121648 mapread 0x01c598 thru 0x0244a8 (0x7f11 bytes) 000544 1783138956.133679 mapread 0x01e24c thru 0x0244a8 (0x625d bytes) 000545 1783138956.135086 mapwrite 0x01900e thru 0x0194bf (0x4b2 bytes) 000546 1783138956.188829 trunc from 0x0244a9 to 0x0291bb (0x4d13 bytes) 000547 1783138956.230033 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x291bc): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x291bc): Mirror 6: 0xa968082b (stale) Mirror 7: 0xa968082b (stale) Mirror 8: 0xa968082b (stale) Mirror 9: 0xa968082b (stale) CRC-32 checksum value for chunk [0, 0x291bc): Mirror 1: 0xa3eb6c00 Mirror 10: 0xa3eb6c00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x291bc: skipped 000548 1783138957.055421 write 0x02edc6 thru 0x0346b8 (0x58f3 bytes) 000549 1783138957.120168 read 0x00eee1 thru 0x014cdc (0x5dfc bytes) 000550 1783138957.127963 read 0x02232a thru 0x027d07 (0x59de bytes) 000551 1783138957.137564 mapwrite 0x008d7e thru 0x012002 (0x9285 bytes) 000552 1783138957.189421 read 0x0024d2 thru 0x00f2c9 (0xcdf8 bytes) 000556 1783138957.194190 mapwrite 0x00b837 thru 0x010756 (0x4f20 bytes) 000558 1783138957.239309 mapwrite 0x007131 thru 0x00ae1a (0x3cea bytes) 000560 1783138957.306957 MIRROR_RESYNC 6 mirrors 000562 1783138958.203705 read 0x00cdd4 thru 0x00dfed (0x121a bytes) 000563 1783138958.208605 trunc from 0x0346b9 to 0x0352a1 (0xbe9 bytes) 000564 1783138958.302266 mapread 0x00c1ac thru 0x014f7b (0x8dd0 bytes) 000567 1783138958.320649 mapread 0x0251c0 thru 0x033025 (0xde66 bytes) 000570 1783138958.342069 read 0x01fdff thru 0x02d5ce (0xd7d0 bytes) 000572 1783138958.347715 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x352a2): Mirror 6: 0xdddc8672 (stale) Mirror 7: 0xdddc8672 (stale) Mirror 8: 0xdddc8672 (stale) Mirror 9: 0xdddc8672 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x352a2): Mirror 6: 0xdddc8672 (stale) Mirror 7: 0xdddc8672 (stale) Mirror 8: 0xdddc8672 (stale) Mirror 9: 0xdddc8672 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x352a2: skipped 000573 1783138958.723522 write 0x02401d thru 0x027115 (0x30f9 bytes) 000574 1783138958.744485 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x352a2): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x352a2): Mirror 6: 0xdddc8672 (stale) Mirror 7: 0xdddc8672 (stale) Mirror 8: 0xdddc8672 (stale) Mirror 9: 0xdddc8672 (stale) CRC-32 checksum value for chunk [0, 0x352a2): Mirror 1: 0xb88f1cf9 Mirror 10: 0xb88f1cf9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x352a2: skipped 000576 1783138959.722544 trunc from 0x0352a2 to 0x0364ea (0x1249 bytes) 000577 1783138959.835556 read 0x003a08 thru 0x008eb8 (0x54b1 bytes) 000578 1783138959.849396 read 0x01dfcf thru 0x0213bb (0x33ed bytes) 000579 1783138959.861359 mapread 0x01506f thru 0x019f18 (0x4eaa bytes) 000580 1783138959.862905 write 0x01263c thru 0x017578 (0x4f3d bytes) 000583 1783138959.869295 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x364eb): Mirror 6: 0x1913045c (stale) Mirror 7: 0x1913045c (stale) Mirror 9: 0x1913045c (stale) Mirror 10: 0xc002948e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364eb): Mirror 6: 0x1913045c (stale) Mirror 7: 0x1913045c (stale) Mirror 9: 0x1913045c (stale) Mirror 10: 0xc002948e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364eb: skipped 000584 1783138960.281154 write 0x007f58 thru 0x00bdb5 (0x3e5e bytes) 000585 1783138960.290522 mapread 0x00ce18 thru 0x017d49 (0xaf32 bytes) 000586 1783138960.323448 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x364eb): Mirror 7: 0x1913045c (stale) Mirror 9: 0x1913045c (stale) Mirror 10: 0xc002948e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364eb): Mirror 7: 0x1913045c (stale) Mirror 9: 0x1913045c (stale) Mirror 10: 0xc002948e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364eb: skipped 000587 1783138960.791789 mapwrite 0x00aac6 thru 0x00c199 (0x16d4 bytes) 000588 1783138960.845562 read 0x0293ef thru 0x0325e8 (0x91fa bytes) 000590 1783138960.855923 mapread 0x02793a thru 0x02b52a (0x3bf1 bytes) 000593 1783138960.869534 mapread 0x0030c8 thru 0x011a23 (0xe95c bytes) 000594 1783138960.876671 mapread 0x0363f3 thru 0x0364ea (0xf8 bytes) 000595 1783138960.882453 mapwrite 0x026269 thru 0x035425 (0xf1bd bytes) 000597 1783138960.940679 write 0x00e291 thru 0x013ed5 (0x5c45 bytes) 000598 1783138960.942097 mapread 0x02bb4a thru 0x02cc63 (0x111a bytes) 000599 1783138960.943501 mapwrite 0x0045d6 thru 0x00cd36 (0x8761 bytes) 000600 1783138960.997271 MIRROR_RESYNC 4 mirrors 000601 1783138961.829607 read 0x0066c6 thru 0x00cbae (0x64e9 bytes) 000603 1783138961.840537 read 0x0155ef thru 0x0243e2 (0xedf4 bytes) 000604 1783138961.862960 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000605 1783138962.216487 mapwrite 0x0266c9 thru 0x02f8f1 (0x9229 bytes) 000606 1783138962.398498 mapread 0x029e1d thru 0x03373e (0x9922 bytes) 000607 1783138962.411080 mapread 0x02e20c thru 0x030a22 (0x2817 bytes) 000609 1783138962.412459 read 0x03333c thru 0x0364ea (0x31af bytes) 000610 1783138962.414571 trunc from 0x0364eb to 0x01d31b (0x191cf bytes) 000612 1783138962.442205 read 0x00e527 thru 0x01cff2 (0xeacc bytes) 000613 1783138962.461880 mapwrite 0x026491 thru 0x02ebe1 (0x8751 bytes) 000615 1783138962.540382 read 0x01ecdb thru 0x022b61 (0x3e87 bytes) 000616 1783138962.541224 trunc from 0x02ebe2 to 0x037c89 (0x90a8 bytes) 000617 1783138962.560677 write 0x024922 thru 0x02ae86 (0x6565 bytes) 000618 1783138962.561935 write 0x01e354 thru 0x02909d (0xad4a bytes) 000620 1783138962.578485 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37c8a): Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c8a): Mirror 7: 0x2867bad4 (stale) Mirror 10: 0x2867bad4 (stale) CRC-32 checksum value for chunk [0, 0x37c8a): Mirror 1: 0xe309215 Mirror 11: 0xe309215 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c8a: skipped 000622 1783138963.316709 read 0x0296fb thru 0x036e92 (0xd798 bytes) 000623 1783138963.334584 mapwrite 0x01161b thru 0x0142d3 (0x2cb9 bytes) 000625 1783138963.505327 write 0x0309f6 thru 0x038312 (0x791d bytes) 000628 1783138963.518557 mapwrite 0x031d47 thru 0x03b597 (0x9851 bytes) 000631 1783138963.657162 MIRROR_RESYNC 4 mirrors 000632 1783138964.470794 trunc from 0x03b598 to 0x02d574 (0xe023 bytes) 000634 1783138964.559208 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d575): Mirror 10: 0xfd675fad (stale) Mirror 11: 0xfd675fad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d575): Mirror 10: 0xfd675fad (stale) Mirror 11: 0xfd675fad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d575: skipped 000635 1783138964.932079 mapread 0x001373 thru 0x0102dd (0xef6b bytes) 000636 1783138964.953948 mapread 0x029859 thru 0x02add7 (0x157f bytes) 000637 1783138964.970021 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d575): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d575): Mirror 10: 0xfd675fad (stale) Mirror 11: 0xfd675fad (stale) CRC-32 checksum value for chunk [0, 0x2d575): Mirror 1: 0xfd675fad Mirror 12: 0xfd675fad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d575: skipped 000639 1783138965.837923 read 0x027078 thru 0x02d574 (0x64fd bytes) 000640 1783138965.851839 write 0x00c8ab thru 0x016da7 (0xa4fd bytes) 000641 1783138965.942448 trunc from 0x02d575 to 0x00967e (0x23ef6 bytes) 000642 1783138965.962359 write 0x02e77a thru 0x0327cd (0x4054 bytes) 000646 1783138965.969599 trunc from 0x0327ce to 0x02f773 (0x305a bytes) 000647 1783138966.001217 write 0x00e8ff thru 0x012b62 (0x4264 bytes) 000649 1783138966.022676 read 0x004231 thru 0x00447f (0x24f bytes) 000650 1783138966.031148 read 0x02b5c6 thru 0x02d6f8 (0x2133 bytes) 000654 1783138966.033564 mapwrite 0x00615b thru 0x00cfb9 (0x6e5f bytes) 000655 1783138966.088051 trunc from 0x02f774 to 0x02765b (0x8118 bytes) 000656 1783138966.117092 write 0x010305 thru 0x01d9bd (0xd6b9 bytes) 000657 1783138966.118667 mapread 0x01cc61 thru 0x01f9a4 (0x2d44 bytes) 000658 1783138966.121353 trunc from 0x02765c to 0x016835 (0x10e26 bytes) 000659 1783138966.137061 write 0x010a19 thru 0x01d610 (0xcbf8 bytes) 000660 1783138966.140743 write 0x02e440 thru 0x0358b3 (0x7474 bytes) 000663 1783138966.146116 read 0x00e5a7 thru 0x00f589 (0xfe3 bytes) 000664 1783138966.147708 mapread 0x0067dd thru 0x00b38b (0x4baf bytes) 000665 1783138966.148913 read 0x01de49 thru 0x027c3a (0x9df2 bytes) 000670 1783138966.161691 write 0x034883 thru 0x03ffff (0xb77d bytes) 000671 1783138966.163792 trunc from 0x040000 to 0x025ea3 (0x1a15c bytes) 000674 1783138966.189098 read 0x008b31 thru 0x013abc (0xaf8c bytes) 000675 1783138966.190106 read 0x001022 thru 0x009a7f (0x8a5e bytes) 000676 1783138966.191450 trunc from 0x025ea4 to 0x01238f (0x13b14 bytes) 000679 1783138966.215810 mapwrite 0x038ee8 thru 0x03f2c0 (0x63d9 bytes) 000680 1783138966.299141 write 0x03ab18 thru 0x03ffff (0x54e8 bytes) 000682 1783138966.300892 write 0x02eddd thru 0x031fd9 (0x31fd bytes) 000684 1783138966.304306 mapread 0x038734 thru 0x03ffff (0x78cc bytes) 000687 1783138966.311752 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xe86949d2 (stale) Mirror 11: 0xe86949d2 (stale) Mirror 12: 0x8ea27e80 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdc520ea0 Mirror 13: 0xdc520ea0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000689 1783138967.175234 mapread 0x01e456 thru 0x02e1d6 (0xfd81 bytes) 000690 1783138967.238591 mapwrite 0x01b224 thru 0x023700 (0x84dd bytes) 000693 1783138967.391379 write 0x02fb54 thru 0x036cd7 (0x7184 bytes) 000695 1783138967.412255 MIRROR_RESYNC 5 mirrors 000697 1783138968.657486 mapwrite 0x0339ee thru 0x03ffff (0xc612 bytes) 000698 1783138968.875650 write 0x03ec3d thru 0x03ffff (0x13c3 bytes) 000699 1783138968.876956 trunc from 0x040000 to 0x0026ba (0x3d945 bytes) 000700 1783138968.918798 trunc from 0x0026bb to 0x014553 (0x11e99 bytes) 000701 1783138968.936785 write 0x01c123 thru 0x02b6cf (0xf5ad bytes) 000702 1783138968.947840 MIRROR_RESYNC 5 mirrors 000703 1783138970.048690 mapread 0x0114f0 thru 0x01e51c (0xd02d bytes) 000704 1783138970.082476 MIRROR_EXTEND to 6 mirrors 000706 1783138970.740140 read 0x00443b thru 0x0112f2 (0xceb8 bytes) 000707 1783138970.749591 mapwrite 0x0199d1 thru 0x023a99 (0xa0c9 bytes) 000708 1783138970.940916 mapread 0x02020f thru 0x02200d (0x1dff bytes) 000709 1783138970.943055 write 0x007172 thru 0x0151d2 (0xe061 bytes) 000710 1783138970.963618 write 0x0125af thru 0x013926 (0x1378 bytes) 000711 1783138970.964900 mapread 0x00069c thru 0x002194 (0x1af9 bytes) 000712 1783138970.988634 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2b6d0): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b6d0): Mirror 10: 0x8772b0af (stale) Mirror 11: 0x8772b0af (stale) Mirror 12: 0x8772b0af (stale) Mirror 13: 0x8772b0af (stale) Mirror 14: 0x8772b0af (stale) CRC-32 checksum value for chunk [0, 0x2b6d0): Mirror 1: 0x9cd1dd82 Mirror 15: 0x9cd1dd82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b6d0: skipped 000714 1783138972.095723 mapread 0x0253b3 thru 0x02b222 (0x5e70 bytes) 000717 1783138972.154712 write 0x0155d6 thru 0x017cdf (0x270a bytes) 000718 1783138972.270596 read 0x023726 thru 0x02997b (0x6256 bytes) 000719 1783138972.313097 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b6d0): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b6d0): Mirror 10: 0x8772b0af (stale) Mirror 11: 0x8772b0af (stale) Mirror 12: 0x8772b0af (stale) Mirror 13: 0x8772b0af (stale) Mirror 14: 0x8772b0af (stale) Mirror 15: 0x9cd1dd82 (stale) CRC-32 checksum value for chunk [0, 0x2b6d0): Mirror 1: 0x8e7f2a16 Mirror 16: 0x8e7f2a16 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b6d0: skipped 000720 1783138973.200678 mapwrite 0x0061c1 thru 0x00e762 (0x85a2 bytes) 000722 1783138973.381105 trunc from 0x02b6d0 to 0x018dc2 (0x1290d bytes) 000725 1783138973.417455 read 0x00ba71 thru 0x014f57 (0x94e7 bytes) 000726 1783138973.423233 mapread 0x0123db thru 0x018dc2 (0x69e8 bytes) 000727 1783138973.431740 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x18dc3): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18dc3): Mirror 10: 0xf4608d74 (stale) Mirror 11: 0xf4608d74 (stale) Mirror 12: 0xf4608d74 (stale) Mirror 13: 0xf4608d74 (stale) Mirror 14: 0xf4608d74 (stale) Mirror 15: 0x8c3a6e32 (stale) Mirror 16: 0x4a9947c8 (stale) CRC-32 checksum value for chunk [0, 0x18dc3): Mirror 1: 0x6ce71edb Mirror 17: 0x6ce71edb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18dc3: skipped 000728 1783138974.157986 trunc from 0x018dc3 to 0x03202d (0x1926b bytes) 000729 1783138974.262999 mapwrite 0x012ec6 thru 0x014210 (0x134b bytes) 000730 1783138974.358417 write 0x0269a9 thru 0x02f4f3 (0x8b4b bytes) 000731 1783138974.368674 mapwrite 0x03f2fe thru 0x03ffff (0xd02 bytes) 000732 1783138974.443094 read 0x02bc81 thru 0x02f04c (0x33cc bytes) 000733 1783138974.444479 trunc from 0x040000 to 0x029081 (0x16f7e bytes) 000734 1783138974.466193 read 0x013992 thru 0x0176a5 (0x3d14 bytes) 000736 1783138974.468526 write 0x00ea88 thru 0x01caf0 (0xe069 bytes) 000740 1783138974.478365 mapwrite 0x023807 thru 0x02e043 (0xa83d bytes) 000742 1783138974.566057 mapwrite 0x00da58 thru 0x0135a8 (0x5b51 bytes) 000743 1783138974.615828 read 0x009685 thru 0x014ee1 (0xb85d bytes) 000744 1783138974.617570 mapwrite 0x002727 thru 0x00726d (0x4b47 bytes) 000745 1783138974.690223 read 0x01ece5 thru 0x01faa9 (0xdc5 bytes) 000746 1783138974.691558 write 0x039a15 thru 0x03ffff (0x65eb bytes) 000747 1783138974.696281 read 0x0311cd thru 0x031bdb (0xa0f bytes) 000752 1783138974.711358 trunc from 0x040000 to 0x017979 (0x28686 bytes) 000753 1783138974.739807 write 0x01217b thru 0x01d3b3 (0xb239 bytes) 000754 1783138974.741605 trunc from 0x01d3b4 to 0x022f7f (0x5bcc bytes) 000755 1783138974.761125 write 0x02cb50 thru 0x038d9b (0xc24c bytes) 000756 1783138974.762816 write 0x02c31c thru 0x03126c (0x4f51 bytes) 000757 1783138974.764515 read 0x018a8f thru 0x01a9de (0x1f50 bytes) 000759 1783138974.766251 mapwrite 0x005c6c thru 0x015248 (0xf5dd bytes) 000760 1783138974.817494 mapread 0x02aff5 thru 0x02c797 (0x17a3 bytes) 000762 1783138974.827012 read 0x035fc6 thru 0x036c71 (0xcac bytes) 000763 1783138974.827962 read 0x0271f0 thru 0x030d3d (0x9b4e bytes) 000765 1783138974.831395 mapwrite 0x020a2e thru 0x023074 (0x2647 bytes) 000766 1783138974.887447 read 0x001dd9 thru 0x003aad (0x1cd5 bytes) 000771 1783138974.893282 read 0x032ab3 thru 0x03865d (0x5bab bytes) 000772 1783138974.894427 mapwrite 0x00065a thru 0x0057ed (0x5194 bytes) 000773 1783138974.954365 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x38d9c): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d9c): Mirror 10: 0xc50801e0 (stale) Mirror 11: 0xc50801e0 (stale) Mirror 12: 0xc50801e0 (stale) Mirror 13: 0xc50801e0 (stale) Mirror 14: 0xc50801e0 (stale) Mirror 15: 0x742c9474 (stale) Mirror 16: 0xc1007091 (stale) Mirror 17: 0xef1a3c41 (stale) CRC-32 checksum value for chunk [0, 0x38d9c): Mirror 1: 0x2bea14c9 Mirror 18: 0x2bea14c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d9c: skipped 000774 1783138975.985141 MIRROR_RESYNC 10 mirrors 000775 1783138977.181784 mapread 0x00f2f1 thru 0x01c8c8 (0xd5d8 bytes) 000780 1783138977.244233 mapwrite 0x034657 thru 0x036378 (0x1d22 bytes) 000781 1783138977.429011 mapread 0x007e04 thru 0x013fb4 (0xc1b1 bytes) 000782 1783138977.465365 mapwrite 0x012edb thru 0x01c449 (0x956f bytes) 000786 1783138977.547083 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38d9c): Mirror 10: 0x2bea14c9 (stale) Mirror 11: 0x2bea14c9 (stale) Mirror 12: 0x2bea14c9 (stale) Mirror 13: 0x2bea14c9 (stale) Mirror 14: 0x2bea14c9 (stale) Mirror 15: 0x2bea14c9 (stale) Mirror 16: 0x2bea14c9 (stale) Mirror 18: 0x2bea14c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d9c): Mirror 10: 0x2bea14c9 (stale) Mirror 11: 0x2bea14c9 (stale) Mirror 12: 0x2bea14c9 (stale) Mirror 13: 0x2bea14c9 (stale) Mirror 14: 0x2bea14c9 (stale) Mirror 15: 0x2bea14c9 (stale) Mirror 16: 0x2bea14c9 (stale) Mirror 18: 0x2bea14c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d9c: skipped 000787 1783138978.178583 write 0x024c45 thru 0x024dd1 (0x18d bytes) 000788 1783138978.186108 write 0x03566d thru 0x03ffff (0xa993 bytes) 000789 1783138978.198470 write 0x01efd9 thru 0x0277e2 (0x880a bytes) 000790 1783138978.213088 mapwrite 0x0188c6 thru 0x01cbb6 (0x42f1 bytes) 000791 1783138978.292172 read 0x03f081 thru 0x03ffff (0xf7f bytes) 000792 1783138978.300643 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x88362f81 (stale) Mirror 11: 0x88362f81 (stale) Mirror 12: 0x88362f81 (stale) Mirror 13: 0x88362f81 (stale) Mirror 14: 0x88362f81 (stale) Mirror 15: 0x88362f81 (stale) Mirror 16: 0x88362f81 (stale) Mirror 18: 0x88362f81 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x82122150 Mirror 19: 0x82122150 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000793 1783138979.141201 MIRROR_RESYNC 10 mirrors 000794 1783138980.414772 mapread 0x021108 thru 0x0265c8 (0x54c1 bytes) 000796 1783138980.432439 mapwrite 0x03bd18 thru 0x03ffff (0x42e8 bytes) 000798 1783138980.605483 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x82122150 (stale) Mirror 11: 0x82122150 (stale) Mirror 12: 0x82122150 (stale) Mirror 13: 0x82122150 (stale) Mirror 15: 0x82122150 (stale) Mirror 16: 0x82122150 (stale) Mirror 18: 0x82122150 (stale) Mirror 19: 0x82122150 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x82122150 (stale) Mirror 11: 0x82122150 (stale) Mirror 12: 0x82122150 (stale) Mirror 13: 0x82122150 (stale) Mirror 15: 0x82122150 (stale) Mirror 16: 0x82122150 (stale) Mirror 18: 0x82122150 (stale) Mirror 19: 0x82122150 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000799 1783138981.094625 read 0x014227 thru 0x0168ce (0x26a8 bytes) 000800 1783138981.098701 mapwrite 0x01b8db thru 0x024a9c (0x91c2 bytes) 000802 1783138981.187396 write 0x035b4f thru 0x03ffff (0xa4b1 bytes) 000803 1783138981.208462 trunc from 0x040000 to 0x017933 (0x286cc bytes) 000804 1783138981.246964 trunc from 0x017934 to 0x03a3ed (0x22aba bytes) 000806 1783138981.273395 write 0x038178 thru 0x03f391 (0x721a bytes) 000810 1783138981.284330 write 0x00d691 thru 0x018b12 (0xb482 bytes) 000812 1783138981.305331 trunc from 0x03f392 to 0x0036ad (0x3bce4 bytes) 000814 1783138981.327423 MIRROR_RESYNC 9 mirrors 000815 1783138982.395664 mapread 0x002e52 thru 0x0036ad (0x85c bytes) 000816 1783138982.406532 mapwrite 0x003b09 thru 0x00d4ec (0x99e4 bytes) 000819 1783138982.679663 mapread 0x00ce04 thru 0x00d4ec (0x6e9 bytes) 000820 1783138982.683791 trunc from 0x00d4ed to 0x02b8b5 (0x1e3c9 bytes) 000821 1783138982.731291 write 0x039101 thru 0x03ffff (0x6eff bytes) 000822 1783138982.734396 read 0x020499 thru 0x02462f (0x4197 bytes) 000823 1783138982.752815 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x1eade5a0 (stale) Mirror 11: 0x1eade5a0 (stale) Mirror 12: 0x1eade5a0 (stale) Mirror 13: 0x1eade5a0 (stale) Mirror 15: 0x1eade5a0 (stale) Mirror 16: 0x1eade5a0 (stale) Mirror 18: 0x1eade5a0 (stale) Mirror 19: 0x1eade5a0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc0527605 Mirror 20: 0xc0527605 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000825 1783138983.674822 MIRROR_RESYNC 10 mirrors 000826 1783138985.107116 MIRROR_EXTEND to 11 mirrors 000827 1783138986.017098 write 0x01d54d thru 0x02a21e (0xccd2 bytes) 000828 1783138986.122466 read 0x01dcbc thru 0x023e82 (0x61c7 bytes) 000829 1783138986.123205 mapwrite 0x01254a thru 0x013807 (0x12be bytes) 000833 1783138986.202096 trunc from 0x040000 to 0x0133d4 (0x2cc2b bytes) 000834 1783138986.231177 read 0x00426f thru 0x00cb2f (0x88c1 bytes) 000836 1783138986.245144 mapread 0x0111c2 thru 0x0133d4 (0x2213 bytes) 000837 1783138986.246925 trunc from 0x0133d5 to 0x01476f (0x139b bytes) 000838 1783138986.261030 mapread 0x000793 thru 0x00d20d (0xca7b bytes) 000839 1783138986.263048 read 0x013521 thru 0x01476f (0x124f bytes) 000841 1783138986.269936 mapread 0x00ba1d thru 0x01476f (0x8d53 bytes) 000842 1783138986.271441 trunc from 0x014770 to 0x0160a0 (0x1931 bytes) 000843 1783138986.290758 trunc from 0x0160a1 to 0x037445 (0x213a5 bytes) 000844 1783138986.305394 write 0x0039e1 thru 0x0125b2 (0xebd2 bytes) 000848 1783138986.311438 write 0x01274d thru 0x017085 (0x4939 bytes) 000849 1783138986.319407 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x37446): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37446): Mirror 10: 0x99d2b2f8 (stale) Mirror 11: 0x99d2b2f8 (stale) Mirror 12: 0x99d2b2f8 (stale) Mirror 13: 0x99d2b2f8 (stale) Mirror 15: 0x99d2b2f8 (stale) Mirror 16: 0x99d2b2f8 (stale) Mirror 18: 0x99d2b2f8 (stale) Mirror 19: 0x99d2b2f8 (stale) Mirror 20: 0x99d2b2f8 (stale) Mirror 21: 0x99d2b2f8 (stale) CRC-32 checksum value for chunk [0, 0x37446): Mirror 1: 0x856239fc Mirror 22: 0x856239fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37446: skipped 000850 1783138987.219530 mapread 0x014940 thru 0x019043 (0x4704 bytes) 000852 1783138987.281593 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x37446): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37446): Mirror 10: 0x99d2b2f8 (stale) Mirror 11: 0x99d2b2f8 (stale) Mirror 12: 0x99d2b2f8 (stale) Mirror 13: 0x99d2b2f8 (stale) Mirror 15: 0x99d2b2f8 (stale) Mirror 16: 0x99d2b2f8 (stale) Mirror 18: 0x99d2b2f8 (stale) Mirror 19: 0x99d2b2f8 (stale) Mirror 20: 0x99d2b2f8 (stale) Mirror 21: 0x99d2b2f8 (stale) CRC-32 checksum value for chunk [0, 0x37446): Mirror 1: 0x856239fc Mirror 22: 0x856239fc Mirror 23: 0x856239fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37446: skipped 000854 1783138988.255456 write 0x028fe2 thru 0x02c328 (0x3347 bytes) 000856 1783138988.373398 trunc from 0x037446 to 0x019a0b (0x1da3a bytes) 000857 1783138988.415150 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x19a0c): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19a0c): Mirror 10: 0xbe7ca7e6 (stale) Mirror 11: 0xbe7ca7e6 (stale) Mirror 12: 0xbe7ca7e6 (stale) Mirror 13: 0xbe7ca7e6 (stale) Mirror 15: 0xbe7ca7e6 (stale) Mirror 16: 0xbe7ca7e6 (stale) Mirror 18: 0xbe7ca7e6 (stale) Mirror 19: 0xbe7ca7e6 (stale) Mirror 20: 0xbe7ca7e6 (stale) Mirror 21: 0xbe7ca7e6 (stale) Mirror 22: 0x73af4097 (stale) Mirror 23: 0x73af4097 (stale) CRC-32 checksum value for chunk [0, 0x19a0c): Mirror 1: 0x73af4097 Mirror 24: 0x73af4097 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19a0c: skipped 000858 1783138989.468612 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19a0c): Mirror 10: 0xbe7ca7e6 (stale) Mirror 11: 0xbe7ca7e6 (stale) Mirror 12: 0xbe7ca7e6 (stale) Mirror 13: 0xbe7ca7e6 (stale) Mirror 15: 0xbe7ca7e6 (stale) Mirror 16: 0xbe7ca7e6 (stale) Mirror 18: 0xbe7ca7e6 (stale) Mirror 20: 0xbe7ca7e6 (stale) Mirror 21: 0xbe7ca7e6 (stale) Mirror 22: 0x73af4097 (stale) Mirror 23: 0x73af4097 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19a0c): Mirror 10: 0xbe7ca7e6 (stale) Mirror 11: 0xbe7ca7e6 (stale) Mirror 12: 0xbe7ca7e6 (stale) Mirror 13: 0xbe7ca7e6 (stale) Mirror 15: 0xbe7ca7e6 (stale) Mirror 16: 0xbe7ca7e6 (stale) Mirror 18: 0xbe7ca7e6 (stale) Mirror 20: 0xbe7ca7e6 (stale) Mirror 21: 0xbe7ca7e6 (stale) Mirror 22: 0x73af4097 (stale) Mirror 23: 0x73af4097 (stale) CRC-32 checksum value for chunk [0, 0x19a0c): Mirror 1: 0x73af4097 Mirror 24: 0x73af4097 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19a0c: skipped 000862 1783138990.466219 mapwrite 0x01350b thru 0x01bb9f (0x8695 bytes) 000864 1783138990.682724 mapread 0x0045d8 thru 0x00e599 (0x9fc2 bytes) 000866 1783138990.711819 trunc from 0x01bba0 to 0x035855 (0x19cb6 bytes) 000868 1783138990.740125 trunc from 0x035856 to 0x00240c (0x33449 bytes) 000870 1783138990.760321 write 0x038406 thru 0x03ef28 (0x6b23 bytes) 000872 1783138990.765131 write 0x00608c thru 0x00778e (0x1703 bytes) 000873 1783138990.774121 mapread 0x02d86b thru 0x031f1b (0x46b1 bytes) 000874 1783138990.787425 MIRROR_RESYNC 13 mirrors 000875 1783138992.555948 trunc from 0x03ef29 to 0x001a66 (0x3d4c2 bytes) 000879 1783138992.628755 write 0x00a3c3 thru 0x016b2f (0xc76d bytes) 000880 1783138992.630433 mapwrite 0x02d760 thru 0x033af7 (0x6398 bytes) 000882 1783138992.737016 write 0x016fd9 thru 0x01d768 (0x6790 bytes) 000883 1783138992.751116 write 0x0075e0 thru 0x010b33 (0x9554 bytes) 000884 1783138992.767764 MIRROR_RESYNC 13 mirrors 000885 1783138994.371994 mapwrite 0x03dacf thru 0x03ffff (0x2531 bytes) 000886 1783138994.555667 read 0x0284a4 thru 0x037df4 (0xf951 bytes) 000887 1783138994.567243 write 0x018aeb thru 0x0222cb (0x97e1 bytes) 000888 1783138994.583105 trunc from 0x040000 to 0x02a404 (0x15bfb bytes) 000890 1783138994.606686 mapread 0x004c5b thru 0x00976e (0x4b14 bytes) 000893 1783138994.621132 mapwrite 0x02b3c2 thru 0x031cc4 (0x6903 bytes) 000895 1783138994.728271 write 0x03c856 thru 0x03ffff (0x37aa bytes) 000896 1783138994.733541 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa9cd7a5b (stale) Mirror 11: 0xa9cd7a5b (stale) Mirror 12: 0xa9cd7a5b (stale) Mirror 13: 0xa9cd7a5b (stale) Mirror 15: 0xa9cd7a5b (stale) Mirror 16: 0xa9cd7a5b (stale) Mirror 18: 0xa9cd7a5b (stale) Mirror 20: 0xa9cd7a5b (stale) Mirror 21: 0xa9cd7a5b (stale) Mirror 22: 0xa9cd7a5b (stale) Mirror 23: 0xa9cd7a5b (stale) Mirror 24: 0xa9cd7a5b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4c80a641 Mirror 25: 0x4c80a641 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000898 1783138995.872050 write 0x039251 thru 0x03c660 (0x3410 bytes) 000900 1783138995.965104 write 0x0114a8 thru 0x017bfa (0x6753 bytes) 000901 1783138995.975572 write 0x01a1ba thru 0x01b33e (0x1185 bytes) 000902 1783138995.985491 write 0x036996 thru 0x03ffff (0x966a bytes) 000903 1783138995.993980 mapwrite 0x015906 thru 0x01fe42 (0xa53d bytes) 000904 1783138996.060626 mapwrite 0x0390c3 thru 0x03e270 (0x51ae bytes) 000905 1783138996.118256 read 0x01c21d thru 0x024417 (0x81fb bytes) 000907 1783138996.119496 read 0x00deb3 thru 0x012d47 (0x4e95 bytes) 000908 1783138996.142711 mapread 0x004d77 thru 0x00d6ce (0x8958 bytes) 000909 1783138996.143897 mapwrite 0x000770 thru 0x003c87 (0x3518 bytes) 000910 1783138996.181396 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa9cd7a5b (stale) Mirror 11: 0xa9cd7a5b (stale) Mirror 12: 0xa9cd7a5b (stale) Mirror 13: 0xa9cd7a5b (stale) Mirror 15: 0xa9cd7a5b (stale) Mirror 16: 0xa9cd7a5b (stale) Mirror 18: 0xa9cd7a5b (stale) Mirror 20: 0xa9cd7a5b (stale) Mirror 21: 0xa9cd7a5b (stale) Mirror 22: 0xa9cd7a5b (stale) Mirror 23: 0xa9cd7a5b (stale) Mirror 24: 0xa9cd7a5b (stale) Mirror 25: 0x4c80a641 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x59badfff Mirror 26: 0x59badfff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000914 1783138997.193533 MIRROR_SPLIT mirror 13 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa9cd7a5b (stale) Mirror 11: 0xa9cd7a5b (stale) Mirror 12: 0xa9cd7a5b (stale) Mirror 13: 0xa9cd7a5b (stale) Mirror 15: 0xa9cd7a5b (stale) Mirror 16: 0xa9cd7a5b (stale) Mirror 18: 0xa9cd7a5b (stale) Mirror 20: 0xa9cd7a5b (stale) Mirror 21: 0xa9cd7a5b (stale) Mirror 22: 0xa9cd7a5b (stale) Mirror 23: 0xa9cd7a5b (stale) Mirror 24: 0xa9cd7a5b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa9cd7a5b (stale) Mirror 11: 0xa9cd7a5b (stale) Mirror 12: 0xa9cd7a5b (stale) Mirror 13: 0xa9cd7a5b (stale) Mirror 15: 0xa9cd7a5b (stale) Mirror 16: 0xa9cd7a5b (stale) Mirror 18: 0xa9cd7a5b (stale) Mirror 20: 0xa9cd7a5b (stale) Mirror 21: 0xa9cd7a5b (stale) Mirror 22: 0xa9cd7a5b (stale) Mirror 23: 0xa9cd7a5b (stale) Mirror 24: 0xa9cd7a5b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x59badfff Mirror 26: 0x59badfff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000916 1783138998.411290 mapread 0x001c72 thru 0x00cd54 (0xb0e3 bytes) 000917 1783138998.436306 mapwrite 0x03d0fd thru 0x03ffff (0x2f03 bytes) 000918 1783138998.605843 mapwrite 0x01a306 thru 0x02863b (0xe336 bytes) 000919 1783138998.722720 mapwrite 0x034c75 thru 0x03b04d (0x63d9 bytes) 000920 1783138998.781648 write 0x021337 thru 0x02dece (0xcb98 bytes) 000921 1783138998.783209 trunc from 0x040000 to 0x026dd2 (0x1922d bytes) 000924 1783138998.860209 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x26dd3): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26dd3): Mirror 10: 0xe3225d83 (stale) Mirror 11: 0xe3225d83 (stale) Mirror 12: 0xe3225d83 (stale) Mirror 13: 0xe3225d83 (stale) Mirror 15: 0xe3225d83 (stale) Mirror 16: 0xe3225d83 (stale) Mirror 18: 0xe3225d83 (stale) Mirror 20: 0xe3225d83 (stale) Mirror 21: 0xe3225d83 (stale) Mirror 22: 0xe3225d83 (stale) Mirror 23: 0xe3225d83 (stale) Mirror 24: 0xe3225d83 (stale) Mirror 26: 0x2eab5c23 (stale) CRC-32 checksum value for chunk [0, 0x26dd3): Mirror 1: 0x1bef8a20 Mirror 27: 0x1bef8a20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26dd3: skipped 000926 1783138999.933977 trunc from 0x026dd3 to 0x01deda (0x8ef8 bytes) 000929 1783139000.091089 read 0x00bff4 thru 0x018d60 (0xcd6d bytes) 000930 1783139000.108071 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x1dedb): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dedb): Mirror 10: 0x55e52251 (stale) Mirror 11: 0x55e52251 (stale) Mirror 12: 0x55e52251 (stale) Mirror 13: 0x55e52251 (stale) Mirror 15: 0x55e52251 (stale) Mirror 16: 0x55e52251 (stale) Mirror 18: 0x55e52251 (stale) Mirror 20: 0x55e52251 (stale) Mirror 21: 0x55e52251 (stale) Mirror 22: 0x55e52251 (stale) Mirror 23: 0x55e52251 (stale) Mirror 24: 0x55e52251 (stale) Mirror 26: 0xbf93fbe0 (stale) Mirror 27: 0xf3432cf6 (stale) CRC-32 checksum value for chunk [0, 0x1dedb): Mirror 1: 0xf3432cf6 Mirror 28: 0xf3432cf6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dedb: skipped 000931 1783139001.179355 mapwrite 0x01bf56 thru 0x01d9f5 (0x1aa0 bytes) 000932 1783139001.382652 trunc from 0x01dedb to 0x00f69c (0xe83e bytes) 000933 1783139001.413899 MIRROR_RESYNC 16 mirrors 000936 1783139003.452949 MIRROR_SPLIT mirror 4 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000937 1783139003.968135 read 0x000304 thru 0x00f69c (0xf399 bytes) 000939 1783139004.010031 MIRROR_RESYNC 15 mirrors 000941 1783139004.739388 read 0x0031c4 thru 0x00f060 (0xbe9d bytes) 000943 1783139004.741149 read 0x006c86 thru 0x00f69c (0x8a17 bytes) 000944 1783139004.742316 mapwrite 0x022566 thru 0x02a265 (0x7d00 bytes) 000946 1783139005.009498 write 0x03911a thru 0x03ffff (0x6ee6 bytes) 000947 1783139005.014312 mapwrite 0x035dc7 thru 0x039ca5 (0x3edf bytes) 000951 1783139005.077015 write 0x0311f1 thru 0x038c09 (0x7a19 bytes) 000953 1783139005.079771 read 0x01a023 thru 0x0262d7 (0xc2b5 bytes) 000954 1783139005.081044 mapwrite 0x00e944 thru 0x01a686 (0xbd43 bytes) 000955 1783139005.130115 read 0x005ae8 thru 0x0136c7 (0xdbe0 bytes) 000956 1783139005.131327 read 0x0039d0 thru 0x01395e (0xff8f bytes) 000957 1783139005.132363 write 0x02e95b thru 0x035fa2 (0x7648 bytes) 000958 1783139005.134936 mapwrite 0x016b70 thru 0x0263a5 (0xf836 bytes) 000959 1783139005.178140 read 0x03bfc9 thru 0x03f83e (0x3876 bytes) 000960 1783139005.178788 mapwrite 0x0252ee thru 0x03374d (0xe460 bytes) 000963 1783139005.223149 trunc from 0x040000 to 0x016310 (0x29cef bytes) 000964 1783139005.255646 mapwrite 0x02da72 thru 0x03bc51 (0xe1e0 bytes) 000965 1783139005.392996 read 0x010693 thru 0x017e7f (0x77ed bytes) 000966 1783139005.393952 write 0x013d93 thru 0x021382 (0xd5f0 bytes) 000967 1783139005.397233 read 0x02c8cd thru 0x039e97 (0xd5cb bytes) 000971 1783139005.402308 mapread 0x013bbc thru 0x021bc3 (0xe008 bytes) 000972 1783139005.404785 read 0x031773 thru 0x03bc51 (0xa4df bytes) 000975 1783139005.416378 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bc52): Mirror 10: 0x17bb9ad0 (stale) Mirror 11: 0x17bb9ad0 (stale) Mirror 15: 0x17bb9ad0 (stale) Mirror 16: 0x17bb9ad0 (stale) Mirror 18: 0x17bb9ad0 (stale) Mirror 20: 0x17bb9ad0 (stale) Mirror 21: 0x17bb9ad0 (stale) Mirror 22: 0x17bb9ad0 (stale) Mirror 23: 0x17bb9ad0 (stale) Mirror 24: 0x17bb9ad0 (stale) Mirror 26: 0x17bb9ad0 (stale) Mirror 27: 0x17bb9ad0 (stale) Mirror 28: 0x17bb9ad0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc52): Mirror 10: 0x17bb9ad0 (stale) Mirror 11: 0x17bb9ad0 (stale) Mirror 15: 0x17bb9ad0 (stale) Mirror 16: 0x17bb9ad0 (stale) Mirror 18: 0x17bb9ad0 (stale) Mirror 20: 0x17bb9ad0 (stale) Mirror 21: 0x17bb9ad0 (stale) Mirror 22: 0x17bb9ad0 (stale) Mirror 23: 0x17bb9ad0 (stale) Mirror 24: 0x17bb9ad0 (stale) Mirror 26: 0x17bb9ad0 (stale) Mirror 27: 0x17bb9ad0 (stale) Mirror 28: 0x17bb9ad0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc52: skipped 000978 1783139006.060469 read 0x01d3aa thru 0x02c313 (0xef6a bytes) 000980 1783139006.078609 read 0x01e8d4 thru 0x028865 (0x9f92 bytes) 000981 1783139006.080096 trunc from 0x03bc52 to 0x00fb36 (0x2c11b bytes) 000982 1783139006.101204 write 0x03a7a9 thru 0x03e2c0 (0x3b18 bytes) 000984 1783139006.104428 mapwrite 0x038c07 thru 0x03ffff (0x73f9 bytes) 000987 1783139006.229329 read 0x0254d5 thru 0x028ea6 (0x39d2 bytes) 000989 1783139006.233445 write 0x0274d7 thru 0x02948f (0x1fb9 bytes) 000991 1783139006.238837 mapwrite 0x012947 thru 0x01f32c (0xc9e6 bytes) 000995 1783139006.356015 read 0x0353ee thru 0x03ea52 (0x9665 bytes) 000996 1783139006.367604 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xd9c7bc61 (stale) Mirror 11: 0xd9c7bc61 (stale) Mirror 15: 0xd9c7bc61 (stale) Mirror 16: 0xd9c7bc61 (stale) Mirror 18: 0xd9c7bc61 (stale) Mirror 20: 0xd9c7bc61 (stale) Mirror 21: 0xd9c7bc61 (stale) Mirror 22: 0xd9c7bc61 (stale) Mirror 23: 0xd9c7bc61 (stale) Mirror 24: 0xd9c7bc61 (stale) Mirror 26: 0xd9c7bc61 (stale) Mirror 27: 0xd9c7bc61 (stale) Mirror 28: 0xd9c7bc61 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x715514fa Mirror 29: 0x715514fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000997 1783139007.382389 trunc from 0x040000 to 0x02fa25 (0x105da bytes) 001000 1783139007.573144 trunc from 0x02fa26 to 0x0272c3 (0x8762 bytes) All operations completed A-OK! PASS 70a (91s) == sanity-flr test 71: check mirror extend parent fid ==== 00:23:32 (1783139012) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 00:23:34 (1783139014) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.243096 s, 8.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17718 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 00:23:41 (1783139021) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3607264 .. Extending file size to 2010752 .. failed Extending file size to 8548896 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7266016 .. Extending file size to 7452704 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8724096 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8964960 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8949792 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9383520 .. Waiting 56931 56932 56933 56934 56937 done 192.168.203.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre2 (opts:) 192.168.203.160@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 00:24:58 (1783139098) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6191008 .. Extending file size to 2334528 .. Extending file size to 3155776 .. Extending file size to 6386656 .. Extending file size to 8751968 .. Extending file size to 8139904 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8932832 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9046368 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9306912 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58416 58417 58418 58420 58422 done 192.168.203.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre2 (opts:) 192.168.203.160@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 00:26:15 (1783139175) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115871 s, 18.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.284613 s, 7.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa7b5d8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 00:26:32 (1783139192) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0651055 s, 32.2 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 00:26:38 (1783139198) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 00:26:45 (1783139205) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.248326 s, 8.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 00:26:52 (1783139212) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17615 s, 11.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 00:27:00 (1783139220) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.282384 s, 7.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 00:27:31 (1783139251) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg360-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121982 s, 8.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:28:28 (1783139308) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg360-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.317248 s, 6.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:29:27 (1783139367) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg360-server mds-ost sync done. Creating new pool test_204f oleg360-server: Pool lustre.test_204f created Adding targets to pool oleg360-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg360-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.294771 s, 7.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg360-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg360-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg360-server: Pool lustre.test_204f destroyed PASS 204f (72s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 00:30:39 (1783139439) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 00:30:46 (1783139446) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:30:53 (1783139453) oleg360-server: Pool lustre.test_206 created Adding targets to pool oleg360-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg360-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg360-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg360-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg360-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:31:16 (1783139476) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727472 s, 14.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436303 s, 24.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc5c4b245 Mirror 3: 0xc5c4b245 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:31:25 (1783139485) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:31:28 (1783139488) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:31:31 (1783139491) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.12662 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x23f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x870000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x870000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x870000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x960000 Read this (4 bytes): BBBB 192.168.203.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre2 (opts:) PASS 209a (32s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:32:03 (1783139523) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0219483 s, 373 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:32:09 (1783139529) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514486 s, 20.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x218 lmm_fid: [0x200000404:0x218:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x218 lmm_fid: [0x200000404:0x218:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834456 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-04 00:32:11.000000000 -0400 Modify: 2026-07-04 00:32:11.000000000 -0400 Change: 2026-07-04 00:32:12.000000000 -0400 Birth: 2026-07-04 00:32:12.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:32:16 (1783139536) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:32:19 (1783139539) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.501589 s, 81.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0367119 s, 112 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 4 00:32 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 4 00:32 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 00:32:27 (1783139547) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 00:32:35 (1783139555) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1596 sec =========== 00:32:42 (1783139562) === sanity-flr: start cleanup 00:32:44 (1783139564) === === sanity-flr: finish cleanup 00:32:50 (1783139570) ===