-----============= acceptance-small: sanity-flr ============----- Fri Sep 11 06:36:53 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg610-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:37:10 (1789123030) === oleg610-client.virtnet: executing check_config_client /mnt/lustre oleg610-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg610-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff973588c7e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff973588c7e000.idle_timeout=debug disable quota as required oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Setting lod.lustre-MDT0000-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' === sanity-flr: finish setup 06:37:53 (1789123073) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7643] == sanity-flr test 0a: lfs mirror create with -N option == 06:37:57 (1789123077) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:38:18 (1789123098) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:38:22 (1789123102) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:38:25 (1789123105) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:38:50 (1789123130) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:38:54 (1789123134) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 06:38:57 (1789123137) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff973588c7e000.stats=clear osc.lustre-OST0001-osc-ffff973588c7e000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:39:05 (1789123145) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:39:16 (1789123156) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:39:26 (1789123166) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:39:29 (1789123169) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:39:38 (1789123178) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 06:39:50 (1789123190) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:39:59 (1789123199) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:40:03 (1789123203) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:40:11 (1789123211) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:40:23 (1789123223) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:40:29 (1789123229) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.497673 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760124 s, 13.8 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:40:39 (1789123239) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.453899 s, 25.4 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:40:49 (1789123249) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:41:01 (1789123261) Stopping /mnt/lustre-ost1 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff973588c7e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff973588c7e000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg610-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff973588c7e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff973588c7e000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg610-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:42:12 (1789123332) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.431782 s, 26.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff973588c7e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff973588c7e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg610-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:42:52 (1789123372) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff973588c7e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff973588c7e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg610-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff973588c7e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff973588c7e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg610-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:44:07 (1789123447) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.16082 s, 10.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 11 06:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.151s user 0m0.001s sys 0m0.188s Stopping /mnt/lustre-ost1 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff973588c7e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff973588c7e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.023s user 0m0.001s sys 0m0.186s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg610-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff973588c7e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff973588c7e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.403s user 0m0.000s sys 0m0.177s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg610-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg610-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff973588c7e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff973588c7e000.ost_server_uuid in FULL state after 0 sec PASS 33b (86s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:45:32 (1789123532) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:45:36 (1789123536) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:45:38 (1789123538) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:45:41 (1789123541) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 06:45:47 (1789123547) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.527763 s, 23.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:45:58 (1789123558) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439462 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572174 s, 18.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:46:05 (1789123565) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:46:13 (1789123573) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00150403 s, 681 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000992717 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00133025 s, 770 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011929 s, 858 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 06:46:22 (1789123582) create test files with size 15758778 .. e857e648986dacac6b9fd2141330486d - 1f5b4089dd11d586f3f6edd2818fa40a - 84c2f01469f22c2f0783e6c1729a0d62 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg610-client.virtnet /mnt/lustre (opts:) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 06:46:45 (1789123605) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:46:58 (1789123618) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:47:04 (1789123624) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107864 s, 9.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389117 s, 26.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 06:47:13 (1789123633) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147045 s, 14.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.288157 s, 14.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.314086 s, 6.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 06:47:30 (1789123650) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 06:47:33 (1789123653) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:47:36 (1789123656) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338482 s, 31.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 06:47:44 (1789123664) llite.lustre-ffff9735a014d000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (6s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 06:47:51 (1789123671) llite.lustre-ffff9735a014d000.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344925 s, 30.4 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635372 s, 16.5 MB/s **verify components PASS 43d (7s) == sanity-flr test 44a: lfs mirror split check =========== 06:47:59 (1789123679) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:48:02 (1789123682) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:48:12 (1789123692) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.319405 s, 32.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg610-client.virtnet /mnt/lustre (opts:) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:48:24 (1789123704) llite.lustre-ffff97359162f000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.330397 s, 31.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg610-client.virtnet /mnt/lustre (opts:) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff97359883b000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:48:37 (1789123717) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:49:18 (1789123758) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:49:54 (1789123794) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 06:49:57 (1789123797) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.313789 s, 10.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.150936 s, 20.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe56dd7a0 (nosync,stale) Mirror 2: 0xe56dd7a0 (nosync,stale) ** verify mirror 2 doesn't change original checksum: b4f49828ecb15b5194f21dddf33229c8 - mirror 1 checksum: 9c918275ed89c5fb39e7a99ac2109fc0 - mirror 2 checksum: b4f49828ecb15b5194f21dddf33229c8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T06:50:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T06:50:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b4f49828ecb15b5194f21dddf33229c8 - mirror 1 checksum: 9c918275ed89c5fb39e7a99ac2109fc0 - mirror 2 checksum: 9c918275ed89c5fb39e7a99ac2109fc0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T06:50:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T06:50:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:50:13 (1789123813) Failing mds1 on oleg610-server Stopping /mnt/lustre-mds1 (opts:) on oleg610-server 06:50:21 (1789123821) shut down facet: mds1 facet_host: oleg610-server facet_failover_host: oleg610-server Failover mds1 to oleg610-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-MDT0000 06:50:54 (1789123854) targets are mounted 06:50:54 (1789123854) facet_failover done oleg610-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457039 s, 22.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:51:10 (1789123870) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:51:12 (1789123872) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:51:15 (1789123875) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0116885 s, 1.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:51:22 (1789123882) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:51:25 (1789123885) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00783192 s, 523 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0666024 s, 61.5 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:51:34 (1789123894) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:51:58 (1789123918) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:52:27 (1789123947) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:52:51 (1789123971) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:53:14 (1789123994) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 06:53:23 (1789124003) mirror create pid 55794 mirror split pid 55796 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55794 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55796 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 06:54:32 (1789124072) Seed set to 4073 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1789124074.334026 MIRROR_EXTEND to 2 mirrors skipping zero size read skipping zero size read skipping zero size read 000011 1789124074.976006 MIRROR_RESYNC 2 mirrors 000012 1789124075.173173 MIRROR_RESYNC 2 mirrors skipping zero size read truncating to largest ever: 0x3a89b 000014 1789124075.369714 trunc from 00000000 to 0x03a89a (0x3a89b bytes) 000015 1789124075.498598 mapread 0x038dc4 thru 0x03a89a (0x1ad7 bytes) 000018 1789124075.540697 write 0x029d8e thru 0x037509 (0xd77c bytes) 000020 1789124075.566627 mapread 0x036104 thru 0x03a89a (0x4797 bytes) 000022 1789124075.568551 mapwrite 0x00dffd thru 0x014a08 (0x6a0c bytes) 000023 1789124075.679978 mapread 0x02ab73 thru 0x03413d (0x95cb bytes) 000024 1789124075.684774 read 0x008948 thru 0x00ff38 (0x75f1 bytes) 000026 1789124075.687874 mapread 0x026410 thru 0x027106 (0xcf7 bytes) 000028 1789124075.707230 read 0x02d66c thru 0x039f80 (0xc915 bytes) 000029 1789124075.708744 read 0x015755 thru 0x0203cc (0xac78 bytes) 000031 1789124075.727731 read 0x005abd thru 0x0158b4 (0xfdf8 bytes) 000032 1789124075.728488 read 0x0178f3 thru 0x0217d3 (0x9ee1 bytes) 000033 1789124075.729067 mapread 0x000c09 thru 0x002179 (0x1571 bytes) 000035 1789124075.737136 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000037 1789124075.956862 trunc from 0x03a89b to 0x0124ec (0x283ae bytes) 000038 1789124075.982173 mapwrite 0x0245f5 thru 0x024866 (0x272 bytes) 000040 1789124076.074515 write 0x02fa73 thru 0x03ac62 (0xb1f0 bytes) 000043 1789124076.078647 read 0x0117fc thru 0x0184dd (0x6ce2 bytes) 000045 1789124076.088099 mapwrite 0x03f9d3 thru 0x03ffff (0x62d bytes) 000046 1789124076.157228 trunc from 0x040000 to 0x02c33e (0x13cc1 bytes) 000047 1789124076.170898 read 0x00ec00 thru 0x013a2c (0x4e2d bytes) 000051 1789124076.183778 write 0x018b72 thru 0x027d5e (0xf1ed bytes) 000053 1789124076.186128 write 0x037699 thru 0x037983 (0x2eb bytes) 000055 1789124076.188547 trunc from 0x037984 to 0x017127 (0x2085c bytes) 000057 1789124076.204756 mapwrite 0x028f2f thru 0x037ada (0xebac bytes) 000060 1789124076.364850 mapwrite 0x029e8a thru 0x02f618 (0x578f bytes) 000061 1789124076.487929 read 0x011bf7 thru 0x01c429 (0xa833 bytes) 000062 1789124076.526646 mapread 0x033b75 thru 0x034f96 (0x1422 bytes) 000065 1789124076.532538 mapread 0x0361b5 thru 0x037ada (0x1926 bytes) 000066 1789124076.541343 mapread 0x022411 thru 0x0312ac (0xee9c bytes) 000068 1789124076.543827 read 0x0181c9 thru 0x01f6a4 (0x74dc bytes) 000071 1789124076.548013 write 0x03e42e thru 0x03ffff (0x1bd2 bytes) 000073 1789124076.557935 mapwrite 0x008c33 thru 0x014bec (0xbfba bytes) 000074 1789124076.674903 read 0x013c7c thru 0x021852 (0xdbd7 bytes) 000075 1789124076.676076 write 0x032eef thru 0x03a12d (0x723f bytes) 000076 1789124076.686669 trunc from 0x040000 to 0x0169c7 (0x29638 bytes) 000078 1789124076.709007 write 0x021e4e thru 0x029261 (0x7414 bytes) 000080 1789124076.712090 trunc from 0x029262 to 0x01b56b (0xdcf6 bytes) 000081 1789124076.728670 write 0x01e02c thru 0x02819d (0xa172 bytes) 000083 1789124076.732773 mapread 0x009e9d thru 0x018517 (0xe67b bytes) 000084 1789124076.757586 MIRROR_EXTEND to 2 mirrors 000085 1789124077.732961 mapread 0x01274a thru 0x01de56 (0xb70d bytes) 000086 1789124077.788117 write 0x003b3e thru 0x011f19 (0xe3dc bytes) 000087 1789124078.042829 trunc from 0x02819e to 0x0264c7 (0x1cd6 bytes) 000089 1789124078.096673 mapwrite 0x02f575 thru 0x03ed90 (0xf81c bytes) 000090 1789124078.286762 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ed91): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed91): Mirror 2: 0x938f2570 (stale) CRC-32 checksum value for chunk [0, 0x3ed91): Mirror 1: 0xfa55db45 Mirror 3: 0xfa55db45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed91: skipped 000091 1789124079.306730 mapwrite 0x009d86 thru 0x00d152 (0x33cd bytes) 000092 1789124079.481695 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ed91): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed91): Mirror 2: 0x938f2570 (stale) Mirror 3: 0xfa55db45 (stale) CRC-32 checksum value for chunk [0, 0x3ed91): Mirror 1: 0x660f7009 Mirror 4: 0x660f7009 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed91: skipped 000094 1789124080.573824 trunc from 0x03ed91 to 0x01d359 (0x21a37 bytes) 000096 1789124080.674863 mapwrite 0x00d12f thru 0x010844 (0x3716 bytes) 000097 1789124080.797511 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d35a): Mirror 2: 0x5fc92d4b (stale) Mirror 4: 0x900413be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d35a): Mirror 2: 0x5fc92d4b (stale) Mirror 4: 0x900413be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d35a: skipped 000098 1789124081.166994 trunc from 0x01d35a to 0x02ecc0 (0x11967 bytes) 000101 1789124081.192142 trunc from 0x02ecc1 to 0x011a58 (0x1d268 bytes) 000103 1789124081.216943 mapread 0x00cb48 thru 0x00ff20 (0x33d9 bytes) 000104 1789124081.255705 trunc from 0x011a59 to 0x00a609 (0x744f bytes) 000105 1789124081.271202 read 0x0028ff thru 0x00a609 (0x7d0b bytes) 000106 1789124081.300280 write 0x03bdee thru 0x03c0f5 (0x308 bytes) 000108 1789124081.303799 trunc from 0x03c0f6 to 0x01f573 (0x1cb82 bytes) 000111 1789124081.329436 mapwrite 0x03b7e1 thru 0x03cad0 (0x12f0 bytes) 000115 1789124081.448791 MIRROR_RESYNC 3 mirrors 000116 1789124082.614865 trunc from 0x03cad1 to 0x037c05 (0x4ecb bytes) 000117 1789124082.726733 mapwrite 0x035e19 thru 0x0395c8 (0x37b0 bytes) 000120 1789124082.847864 write 0x03bc65 thru 0x03ffff (0x439b bytes) 000121 1789124082.849353 mapwrite 0x03ae04 thru 0x03ffff (0x51fc bytes) 000123 1789124082.917471 mapwrite 0x004172 thru 0x008a91 (0x4920 bytes) 000124 1789124082.984012 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xff3c2f53 (stale) Mirror 4: 0xff3c2f53 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3fe59899 Mirror 5: 0x3fe59899 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000125 1789124084.206228 write 0x023040 thru 0x02d31c (0xa2dd bytes) 000128 1789124084.294153 trunc from 0x040000 to 0x00710e (0x38ef1 bytes) 000130 1789124084.314689 trunc from 0x00710f to 0x016971 (0xf863 bytes) 000131 1789124084.335835 write 0x01f286 thru 0x020c44 (0x19bf bytes) 000133 1789124084.338146 read 0x0185b3 thru 0x020c44 (0x8692 bytes) 000134 1789124084.345501 mapwrite 0x02a1f5 thru 0x02fd1a (0x5b26 bytes) 000135 1789124084.455894 mapread 0x01c631 thru 0x01d0ba (0xa8a bytes) 000136 1789124084.457437 trunc from 0x02fd1b to 0x039413 (0x96f9 bytes) 000137 1789124084.487451 read 0x022656 thru 0x02dce3 (0xb68e bytes) 000139 1789124084.508868 write 0x02d9e3 thru 0x031682 (0x3ca0 bytes) 000140 1789124084.513171 mapread 0x021b54 thru 0x0239b5 (0x1e62 bytes) 000141 1789124084.518171 mapread 0x02ae18 thru 0x0301a3 (0x538c bytes) 000142 1789124084.528830 write 0x01acf8 thru 0x029929 (0xec32 bytes) 000145 1789124084.532812 read 0x007ab0 thru 0x016001 (0xe552 bytes) 000147 1789124084.534866 trunc from 0x039414 to 0x01d0c9 (0x1c34a bytes) 000149 1789124084.563997 write 0x00f75b thru 0x01efd8 (0xf87e bytes) 000151 1789124084.567637 mapread 0x009165 thru 0x00aca0 (0x1b3c bytes) 000152 1789124084.568821 mapwrite 0x03a902 thru 0x03ffff (0x56fe bytes) 000154 1789124084.636104 write 0x022799 thru 0x0299fe (0x7266 bytes) 000156 1789124084.648873 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xff3c2f53 (stale) Mirror 4: 0xff3c2f53 (stale) Mirror 5: 0x3fe59899 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xddb8a000 Mirror 6: 0xddb8a000 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x3b02d 000157 1789124086.086000 trunc from 0x040000 to 0x03b02c (0x4fd3 bytes) 000160 1789124086.188115 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b02d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b02d): Mirror 2: 0xcd3b6a17 (stale) Mirror 4: 0xcd3b6a17 (stale) Mirror 5: 0x25292682 (stale) Mirror 6: 0xe9f96aba (stale) CRC-32 checksum value for chunk [0, 0x3b02d): Mirror 1: 0xe9f96aba Mirror 7: 0xe9f96aba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b02d: skipped 000161 1789124087.583037 write 0x011eeb thru 0x01cb4f (0xac65 bytes) 000162 1789124087.683205 mapwrite 0x01de85 thru 0x020353 (0x24cf bytes) 000163 1789124087.872253 MIRROR_RESYNC 6 mirrors truncating to largest ever: 0x3cdad 000164 1789124089.224416 trunc from 0x03b02d to 0x03cdac (0x1d80 bytes) 000165 1789124089.313854 trunc from 0x03cdad to 0x00dc61 (0x2f14b bytes) 000170 1789124089.359449 read 0x0099b8 thru 0x00dc61 (0x42aa bytes) 000171 1789124089.368418 mapwrite 0x00b6ea thru 0x017760 (0xc077 bytes) 000172 1789124089.477506 MIRROR_RESYNC 6 mirrors 000173 1789124091.063029 trunc from 0x017761 to 0x00037a (0x173e6 bytes) 000174 1789124091.137899 mapread 0x0001fc thru 0x00037a (0x17f bytes) 000175 1789124091.146006 write 0x036cdb thru 0x03c028 (0x534e bytes) 000176 1789124091.148433 read 0x002608 thru 0x00f479 (0xce72 bytes) 000177 1789124091.161708 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c029): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c029): Mirror 2: 0xe70174cb (stale) Mirror 4: 0xe70174cb (stale) Mirror 5: 0xe70174cb (stale) Mirror 6: 0xe70174cb (stale) Mirror 7: 0xe70174cb (stale) CRC-32 checksum value for chunk [0, 0x3c029): Mirror 1: 0x3d27da2c Mirror 8: 0x3d27da2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c029: skipped 000179 1789124092.620540 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c029): Mirror 2: 0xe70174cb (stale) Mirror 4: 0xe70174cb (stale) Mirror 5: 0xe70174cb (stale) Mirror 7: 0xe70174cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c029): Mirror 2: 0xe70174cb (stale) Mirror 4: 0xe70174cb (stale) Mirror 5: 0xe70174cb (stale) Mirror 7: 0xe70174cb (stale) CRC-32 checksum value for chunk [0, 0x3c029): Mirror 1: 0x3d27da2c Mirror 8: 0x3d27da2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c029: skipped 000183 1789124093.702807 mapwrite 0x02f192 thru 0x0364da (0x7349 bytes) 000184 1789124093.906940 read 0x035bc4 thru 0x03c028 (0x6465 bytes) 000185 1789124093.914981 trunc from 0x03c029 to 0x016be9 (0x2543f bytes) 000186 1789124093.980630 MIRROR_RESYNC 6 mirrors 000187 1789124095.774104 MIRROR_RESYNC 6 mirrors 000190 1789124096.568619 MIRROR_EXTEND to 7 mirrors 000191 1789124097.573953 read 0x012796 thru 0x016be9 (0x4454 bytes) 000192 1789124097.582920 write 0x0074aa thru 0x00818c (0xce3 bytes) 000193 1789124097.697006 write 0x036594 thru 0x03946e (0x2edb bytes) 000194 1789124097.699742 read 0x0195bd thru 0x01e638 (0x507c bytes) 000195 1789124097.714547 mapread 0x014aba thru 0x015e04 (0x134b bytes) 000196 1789124097.715987 mapread 0x02c879 thru 0x0306e9 (0x3e71 bytes) 000197 1789124097.724775 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3946f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3946f): Mirror 2: 0x447c14c (stale) Mirror 4: 0x447c14c (stale) Mirror 5: 0x447c14c (stale) Mirror 7: 0x447c14c (stale) Mirror 8: 0x447c14c (stale) Mirror 9: 0x447c14c (stale) CRC-32 checksum value for chunk [0, 0x3946f): Mirror 1: 0x6a592df3 Mirror 10: 0x6a592df3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3946f: skipped 000199 1789124099.200766 mapread 0x01b53f thru 0x02510e (0x9bd0 bytes) 000200 1789124099.236848 read 0x03318f thru 0x03541a (0x228c bytes) 000201 1789124099.244905 mapread 0x034308 thru 0x03946e (0x5167 bytes) 000203 1789124099.268914 read 0x0168d1 thru 0x01dda1 (0x74d1 bytes) 000205 1789124099.271070 write 0x0134b7 thru 0x01bfcc (0x8b16 bytes) 000206 1789124099.485782 mapwrite 0x01e24d thru 0x02beb1 (0xdc65 bytes) 000207 1789124099.592829 mapwrite 0x022711 thru 0x02cc42 (0xa532 bytes) 000208 1789124099.741129 MIRROR_RESYNC 8 mirrors 000209 1789124101.583399 MIRROR_RESYNC 8 mirrors 000210 1789124102.191798 read 0x00b4ee thru 0x0152df (0x9df2 bytes) 000211 1789124102.204496 write 0x000dfd thru 0x00f530 (0xe734 bytes) 000213 1789124102.348594 mapwrite 0x01eb27 thru 0x01fd7e (0x1258 bytes) 000215 1789124102.495543 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3946f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3946f): Mirror 2: 0x516f3883 (stale) Mirror 4: 0x516f3883 (stale) Mirror 5: 0x516f3883 (stale) Mirror 7: 0x516f3883 (stale) Mirror 8: 0x516f3883 (stale) Mirror 9: 0x516f3883 (stale) Mirror 10: 0x516f3883 (stale) CRC-32 checksum value for chunk [0, 0x3946f): Mirror 1: 0x6432fd91 Mirror 11: 0x6432fd91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3946f: skipped 000217 1789124104.321080 write 0x036d8c thru 0x038c8a (0x1eff bytes) 000220 1789124104.510081 mapwrite 0x029c92 thru 0x032469 (0x87d8 bytes) 000221 1789124104.691492 write 0x03070a thru 0x03d04d (0xc944 bytes) 000222 1789124104.701735 trunc from 0x03d04e to 0x01deef (0x1f15e bytes) 000226 1789124104.767930 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1def0): Mirror 2: 0x5fc46975 (stale) Mirror 4: 0x5fc46975 (stale) Mirror 5: 0x5fc46975 (stale) Mirror 8: 0x5fc46975 (stale) Mirror 9: 0x5fc46975 (stale) Mirror 10: 0x5fc46975 (stale) Mirror 11: 0x6f8ce664 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1def0): Mirror 2: 0x5fc46975 (stale) Mirror 4: 0x5fc46975 (stale) Mirror 5: 0x5fc46975 (stale) Mirror 8: 0x5fc46975 (stale) Mirror 9: 0x5fc46975 (stale) Mirror 10: 0x5fc46975 (stale) Mirror 11: 0x6f8ce664 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1def0: skipped 000227 1789124105.320782 read 0x0134e9 thru 0x01deef (0xaa07 bytes) 000229 1789124105.329759 trunc from 0x01def0 to 0x03a89e (0x1c9af bytes) 000231 1789124105.356763 mapwrite 0x0011b3 thru 0x007cfe (0x6b4c bytes) 000232 1789124105.425204 read 0x02eb67 thru 0x03a89e (0xbd38 bytes) 000234 1789124105.455045 mapread 0x006a0d thru 0x00b39b (0x498f bytes) 000235 1789124105.460012 read 0x01f312 thru 0x0284d5 (0x91c4 bytes) 000239 1789124105.464458 write 0x03a1df thru 0x03ffff (0x5e21 bytes) 000240 1789124105.466566 write 0x03503d thru 0x03ef23 (0x9ee7 bytes) 000241 1789124105.469008 mapread 0x00b74e thru 0x015553 (0x9e06 bytes) 000242 1789124105.471602 mapread 0x018a48 thru 0x01cd81 (0x433a bytes) 000244 1789124105.478853 MIRROR_RESYNC 8 mirrors 000245 1789124107.794845 write 0x00a003 thru 0x010a27 (0x6a25 bytes) 000247 1789124107.873023 mapread 0x024a3c thru 0x032ebb (0xe480 bytes) 000251 1789124107.968847 trunc from 0x040000 to 0x007e4f (0x381b0 bytes) 000255 1789124108.024054 read 0x005f1c thru 0x007e4f (0x1f34 bytes) 000258 1789124108.032927 read 0x000250 thru 0x007e4f (0x7c00 bytes) 000259 1789124108.040941 write 0x024e10 thru 0x02f53a (0xa72b bytes) 000261 1789124108.049794 mapwrite 0x01a390 thru 0x01edff (0x4a70 bytes) 000262 1789124108.114565 read 0x00167f thru 0x0057b2 (0x4134 bytes) 000264 1789124108.115701 mapwrite 0x028003 thru 0x02fcb6 (0x7cb4 bytes) 000265 1789124108.225372 trunc from 0x02fcb7 to 0x00b382 (0x24934 bytes) 000269 1789124108.252139 read 0x00a6a4 thru 0x00b382 (0xcdf bytes) 000271 1789124108.256494 read 0x00861e thru 0x00b382 (0x2d65 bytes) 000272 1789124108.258036 mapwrite 0x0188cb thru 0x02130c (0x8a42 bytes) 000273 1789124108.340969 write 0x02a911 thru 0x02ced6 (0x25c6 bytes) 000274 1789124108.343641 write 0x0360f7 thru 0x03f0de (0x8fe8 bytes) 000277 1789124108.347825 write 0x0334fc thru 0x03c52c (0x9031 bytes) 000278 1789124108.361738 read 0x00042e thru 0x00f04c (0xec1f bytes) 000279 1789124108.362828 read 0x026d35 thru 0x0325b3 (0xb87f bytes) 000280 1789124108.375377 MIRROR_RESYNC 8 mirrors 000281 1789124110.183981 read 0x01e30e thru 0x023b21 (0x5814 bytes) 000282 1789124110.204885 mapread 0x02c7fd thru 0x034a21 (0x8225 bytes) 000285 1789124110.228154 mapwrite 0x03020e thru 0x036a7b (0x686e bytes) 000286 1789124110.455067 mapread 0x02e192 thru 0x0382f0 (0xa15f bytes) 000288 1789124110.460808 read 0x01183a thru 0x014776 (0x2f3d bytes) 000289 1789124110.476792 mapwrite 0x026e73 thru 0x03665a (0xf7e8 bytes) 000290 1789124110.595848 read 0x024772 thru 0x02abd2 (0x6461 bytes) 000291 1789124110.596776 write 0x0248f2 thru 0x02796a (0x3079 bytes) 000292 1789124110.602915 mapread 0x01339e thru 0x01b884 (0x84e7 bytes) 000294 1789124110.607791 mapread 0x029b8a thru 0x037777 (0xdbee bytes) 000296 1789124110.626185 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f0df): Mirror 2: 0x288d4d25 (stale) Mirror 4: 0x288d4d25 (stale) Mirror 5: 0x288d4d25 (stale) Mirror 8: 0x288d4d25 (stale) Mirror 10: 0x288d4d25 (stale) Mirror 11: 0x288d4d25 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0df): Mirror 2: 0x288d4d25 (stale) Mirror 4: 0x288d4d25 (stale) Mirror 5: 0x288d4d25 (stale) Mirror 8: 0x288d4d25 (stale) Mirror 10: 0x288d4d25 (stale) Mirror 11: 0x288d4d25 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0df: skipped 000297 1789124111.378955 trunc from 0x03f0df to 0x00bc13 (0x334cb bytes) 000298 1789124111.419923 write 0x0139bf thru 0x01c822 (0x8e64 bytes) 000299 1789124111.428274 trunc from 0x01c823 to 0x0049ab (0x17e77 bytes) 000301 1789124111.476942 trunc from 0x0049ac to 0x03959f (0x34bf4 bytes) 000302 1789124111.513937 read 0x01e7db thru 0x02525e (0x6a84 bytes) 000304 1789124111.532917 read 0x03752f thru 0x03959f (0x2071 bytes) 000305 1789124111.581963 mapread 0x019dfa thru 0x0288c5 (0xeacc bytes) 000308 1789124111.586283 write 0x00c8ae thru 0x018d88 (0xc4db bytes) 000311 1789124111.599923 trunc from 0x0395a0 to 0x01782c (0x21d73 bytes) 000313 1789124111.648858 mapwrite 0x00b438 thru 0x0177f8 (0xc3c1 bytes) 000314 1789124111.723738 mapread 0x007eea thru 0x00ef95 (0x70ac bytes) 000315 1789124111.731206 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1782d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1782d): Mirror 2: 0x3bf8828e (stale) Mirror 4: 0x3bf8828e (stale) Mirror 5: 0x3bf8828e (stale) Mirror 8: 0x3bf8828e (stale) Mirror 10: 0x3bf8828e (stale) Mirror 11: 0x3bf8828e (stale) CRC-32 checksum value for chunk [0, 0x1782d): Mirror 1: 0xc40dec49 Mirror 12: 0xc40dec49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1782d: skipped 000316 1789124113.298715 mapwrite 0x03cf93 thru 0x03ffff (0x306d bytes) 000317 1789124113.511629 mapread 0x032f1e thru 0x03fad5 (0xcbb8 bytes) 000318 1789124113.544125 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9cbd4947 (stale) Mirror 4: 0x9cbd4947 (stale) Mirror 5: 0x9cbd4947 (stale) Mirror 8: 0x9cbd4947 (stale) Mirror 11: 0x9cbd4947 (stale) Mirror 12: 0x85ac8411 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9cbd4947 (stale) Mirror 4: 0x9cbd4947 (stale) Mirror 5: 0x9cbd4947 (stale) Mirror 8: 0x9cbd4947 (stale) Mirror 11: 0x9cbd4947 (stale) Mirror 12: 0x85ac8411 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000319 1789124114.114821 write 0x01c724 thru 0x020e60 (0x473d bytes) 000321 1789124114.129807 read 0x015c8c thru 0x018b44 (0x2eb9 bytes) 000323 1789124114.142533 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9cbd4947 (stale) Mirror 4: 0x9cbd4947 (stale) Mirror 5: 0x9cbd4947 (stale) Mirror 11: 0x9cbd4947 (stale) Mirror 12: 0x85ac8411 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9cbd4947 (stale) Mirror 4: 0x9cbd4947 (stale) Mirror 5: 0x9cbd4947 (stale) Mirror 11: 0x9cbd4947 (stale) Mirror 12: 0x85ac8411 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000324 1789124114.805455 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9cbd4947 (stale) Mirror 4: 0x9cbd4947 (stale) Mirror 5: 0x9cbd4947 (stale) Mirror 11: 0x9cbd4947 (stale) Mirror 12: 0x85ac8411 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb8d0d158 Mirror 13: 0xb8d0d158 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000326 1789124115.967999 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9cbd4947 (stale) Mirror 4: 0x9cbd4947 (stale) Mirror 5: 0x9cbd4947 (stale) Mirror 11: 0x9cbd4947 (stale) Mirror 12: 0x85ac8411 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb8d0d158 Mirror 13: 0xb8d0d158 Mirror 14: 0xb8d0d158 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000328 1789124117.309527 read 0x030827 thru 0x036340 (0x5b1a bytes) 000330 1789124117.320263 write 0x036f82 thru 0x03ffff (0x907e bytes) 000336 1789124117.486405 trunc from 0x040000 to 0x01b057 (0x24fa8 bytes) 000337 1789124117.518970 read 0x017546 thru 0x01b057 (0x3b12 bytes) 000338 1789124117.534690 write 0x01325d thru 0x01d76f (0xa513 bytes) 000341 1789124117.542794 mapwrite 0x014e01 thru 0x01dff6 (0x91f6 bytes) 000342 1789124117.637881 mapwrite 0x02234e thru 0x031135 (0xede8 bytes) 000343 1789124117.750007 mapwrite 0x01f947 thru 0x027a80 (0x813a bytes) 000344 1789124117.812805 read 0x00f00e thru 0x0167b9 (0x77ac bytes) 000350 1789124117.819004 read 0x00605d thru 0x00c143 (0x60e7 bytes) 000352 1789124117.826677 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x31136): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31136): Mirror 2: 0x1332ab3f (stale) Mirror 4: 0x1332ab3f (stale) Mirror 5: 0x1332ab3f (stale) Mirror 11: 0x1332ab3f (stale) Mirror 12: 0x3c122c2b (stale) Mirror 13: 0x334b1321 (stale) Mirror 14: 0x334b1321 (stale) CRC-32 checksum value for chunk [0, 0x31136): Mirror 1: 0x356a004 Mirror 15: 0x356a004 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31136: skipped 000353 1789124119.221710 mapread 0x00c29a thru 0x00c865 (0x5cc bytes) 000354 1789124119.231159 mapwrite 0x02af33 thru 0x03a8a4 (0xf972 bytes) 000360 1789124119.633194 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3a8a5): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a8a5): Mirror 2: 0x465ee89b (stale) Mirror 4: 0x465ee89b (stale) Mirror 5: 0x465ee89b (stale) Mirror 11: 0x465ee89b (stale) Mirror 12: 0xab00c925 (stale) Mirror 13: 0x1e4b34e1 (stale) Mirror 14: 0x1e4b34e1 (stale) Mirror 15: 0xec52ed5 (stale) CRC-32 checksum value for chunk [0, 0x3a8a5): Mirror 1: 0x6cacbd13 Mirror 16: 0x6cacbd13 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a8a5: skipped 000362 1789124121.372150 write 0x02f767 thru 0x03adb3 (0xb64d bytes) 000365 1789124121.549739 mapwrite 0x016104 thru 0x01a4ec (0x43e9 bytes) 000366 1789124121.639142 mapwrite 0x015294 thru 0x01f2ed (0xa05a bytes) 000367 1789124121.692267 read 0x0227a4 thru 0x02e557 (0xbdb4 bytes) 000368 1789124121.705895 mapwrite 0x03d362 thru 0x03ffff (0x2c9e bytes) 000371 1789124121.787595 write 0x0171c4 thru 0x01f27f (0x80bc bytes) 000372 1789124121.789777 trunc from 0x040000 to 0x02f11d (0x10ee2 bytes) 000373 1789124121.811776 read 0x028aa8 thru 0x02f11d (0x6676 bytes) 000374 1789124121.812983 read 0x01e2d5 thru 0x024cf9 (0x6a25 bytes) 000376 1789124121.829880 MIRROR_RESYNC 10 mirrors 000379 1789124124.244104 mapwrite 0x0105cc thru 0x01a365 (0x9d9a bytes) 000380 1789124124.651281 trunc from 0x02f11e to 0x0089d6 (0x26747 bytes) 000382 1789124124.712716 mapwrite 0x0083ee thru 0x00c55b (0x416e bytes) 000383 1789124124.919752 trunc from 0x00c55c to 0x03224f (0x25cf4 bytes) 000386 1789124124.963201 trunc from 0x032250 to 0x01df2f (0x14320 bytes) 000387 1789124125.006949 mapwrite 0x02627c thru 0x0319fc (0xb781 bytes) 000388 1789124125.148608 mapread 0x02894f thru 0x0319fc (0x90ae bytes) 000389 1789124125.157399 trunc from 0x0319fd to 0x0350e8 (0x36ec bytes) 000390 1789124125.214597 read 0x022d50 thru 0x028562 (0x5813 bytes) 000392 1789124125.222642 read 0x033125 thru 0x0350e8 (0x1fc4 bytes) 000393 1789124125.279500 trunc from 0x0350e9 to 0x015e3d (0x1f2ab bytes) 000395 1789124125.347539 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15e3e): Mirror 2: 0x2f534596 (stale) Mirror 4: 0x2f534596 (stale) Mirror 5: 0x2f534596 (stale) Mirror 11: 0x2f534596 (stale) Mirror 13: 0x2f534596 (stale) Mirror 14: 0x2f534596 (stale) Mirror 15: 0x2f534596 (stale) Mirror 16: 0x2f534596 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15e3e): Mirror 2: 0x2f534596 (stale) Mirror 4: 0x2f534596 (stale) Mirror 5: 0x2f534596 (stale) Mirror 11: 0x2f534596 (stale) Mirror 13: 0x2f534596 (stale) Mirror 14: 0x2f534596 (stale) Mirror 15: 0x2f534596 (stale) Mirror 16: 0x2f534596 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15e3e: skipped 000396 1789124126.402032 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x15e3e): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15e3e): Mirror 2: 0x2f534596 (stale) Mirror 4: 0x2f534596 (stale) Mirror 5: 0x2f534596 (stale) Mirror 11: 0x2f534596 (stale) Mirror 13: 0x2f534596 (stale) Mirror 14: 0x2f534596 (stale) Mirror 15: 0x2f534596 (stale) Mirror 16: 0x2f534596 (stale) CRC-32 checksum value for chunk [0, 0x15e3e): Mirror 1: 0xffd25cab Mirror 17: 0xffd25cab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15e3e: skipped 000397 1789124128.123598 write 0x01f35e thru 0x0277e5 (0x8488 bytes) 000398 1789124128.219809 read 0x01e912 thru 0x0277e5 (0x8ed4 bytes) 000399 1789124128.226871 trunc from 0x0277e6 to 0x023861 (0x3f84 bytes) 000400 1789124128.275492 read 0x00c0c3 thru 0x01a530 (0xe46e bytes) 000401 1789124128.301103 trunc from 0x023862 to 0x00b563 (0x182fe bytes) 000402 1789124128.330686 trunc from 0x00b564 to 0x03ae2b (0x2f8c8 bytes) 000404 1789124128.360264 trunc from 0x03ae2c to 0x03605b (0x4dd0 bytes) 000405 1789124128.389843 read 0x00b5ee thru 0x0127c3 (0x71d6 bytes) 000407 1789124128.429354 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3605c): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3605c): Mirror 2: 0xd8a7e10c (stale) Mirror 4: 0xd8a7e10c (stale) Mirror 5: 0xd8a7e10c (stale) Mirror 11: 0xd8a7e10c (stale) Mirror 13: 0xd8a7e10c (stale) Mirror 14: 0xd8a7e10c (stale) Mirror 15: 0xd8a7e10c (stale) Mirror 16: 0xd8a7e10c (stale) Mirror 17: 0x1bd0b050 (stale) CRC-32 checksum value for chunk [0, 0x3605c): Mirror 1: 0xdcdf5cbf Mirror 18: 0xdcdf5cbf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3605c: skipped 000408 1789124129.985647 read 0x02d221 thru 0x03605b (0x8e3b bytes) 000409 1789124129.998845 write 0x0116e7 thru 0x018ea4 (0x77be bytes) 000410 1789124130.194523 mapread 0x016282 thru 0x0250f8 (0xee77 bytes) 000414 1789124130.230229 mapread 0x02601b thru 0x035d30 (0xfd16 bytes) 000415 1789124130.243753 mapwrite 0x00133c thru 0x005e1e (0x4ae3 bytes) 000417 1789124130.371129 read 0x027db0 thru 0x03379a (0xb9eb bytes) 000418 1789124130.377436 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3605c): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3605c): Mirror 2: 0xd8a7e10c (stale) Mirror 4: 0xd8a7e10c (stale) Mirror 5: 0xd8a7e10c (stale) Mirror 11: 0xd8a7e10c (stale) Mirror 13: 0xd8a7e10c (stale) Mirror 14: 0xd8a7e10c (stale) Mirror 15: 0xd8a7e10c (stale) Mirror 16: 0xd8a7e10c (stale) Mirror 17: 0x1bd0b050 (stale) Mirror 18: 0xdcdf5cbf (stale) CRC-32 checksum value for chunk [0, 0x3605c): Mirror 1: 0xfd31493a Mirror 19: 0xfd31493a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3605c: skipped 000419 1789124132.277842 mapwrite 0x0090a3 thru 0x0135f1 (0xa54f bytes) 000422 1789124132.742217 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3605c): Mirror 2: 0xd8a7e10c (stale) Mirror 5: 0xd8a7e10c (stale) Mirror 11: 0xd8a7e10c (stale) Mirror 13: 0xd8a7e10c (stale) Mirror 14: 0xd8a7e10c (stale) Mirror 15: 0xd8a7e10c (stale) Mirror 16: 0xd8a7e10c (stale) Mirror 17: 0x1bd0b050 (stale) Mirror 18: 0xdcdf5cbf (stale) Mirror 19: 0xfd31493a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3605c): Mirror 2: 0xd8a7e10c (stale) Mirror 5: 0xd8a7e10c (stale) Mirror 11: 0xd8a7e10c (stale) Mirror 13: 0xd8a7e10c (stale) Mirror 14: 0xd8a7e10c (stale) Mirror 15: 0xd8a7e10c (stale) Mirror 16: 0xd8a7e10c (stale) Mirror 17: 0x1bd0b050 (stale) Mirror 18: 0xdcdf5cbf (stale) Mirror 19: 0xfd31493a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3605c: skipped 000423 1789124134.055606 read 0x02fc0a thru 0x03605b (0x6452 bytes) 000425 1789124134.079809 trunc from 0x03605c to 0x02c17e (0x9edd bytes) 000426 1789124134.145282 trunc from 0x02c17f to 0x02cecb (0xd4d bytes) 000428 1789124134.189697 trunc from 0x02cecc to 0x005aa3 (0x27428 bytes) 000430 1789124134.305663 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5aa4): Mirror 2: 0xc7fe722d (stale) Mirror 5: 0xc7fe722d (stale) Mirror 11: 0xc7fe722d (stale) Mirror 14: 0xc7fe722d (stale) Mirror 15: 0xc7fe722d (stale) Mirror 16: 0xc7fe722d (stale) Mirror 17: 0xc7fe722d (stale) Mirror 18: 0xc7fe722d (stale) Mirror 19: 0x8b18c600 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5aa4): Mirror 2: 0xc7fe722d (stale) Mirror 5: 0xc7fe722d (stale) Mirror 11: 0xc7fe722d (stale) Mirror 14: 0xc7fe722d (stale) Mirror 15: 0xc7fe722d (stale) Mirror 16: 0xc7fe722d (stale) Mirror 17: 0xc7fe722d (stale) Mirror 18: 0xc7fe722d (stale) Mirror 19: 0x8b18c600 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5aa4: skipped 000431 1789124135.185257 read 0x003efa thru 0x005aa3 (0x1baa bytes) 000433 1789124135.194851 mapwrite 0x010b68 thru 0x012042 (0x14db bytes) 000435 1789124135.345334 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x12043): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12043): Mirror 2: 0x28dcd19f (stale) Mirror 5: 0x28dcd19f (stale) Mirror 11: 0x28dcd19f (stale) Mirror 14: 0x28dcd19f (stale) Mirror 15: 0x28dcd19f (stale) Mirror 16: 0x28dcd19f (stale) Mirror 17: 0xa8bb8d2c (stale) Mirror 18: 0x172d90c5 (stale) Mirror 19: 0xd6b53754 (stale) CRC-32 checksum value for chunk [0, 0x12043): Mirror 1: 0xf58b6661 Mirror 20: 0xf58b6661 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12043: skipped 000436 1789124136.597940 write 0x00c56a thru 0x0145f3 (0x808a bytes) 000437 1789124136.784787 write 0x00928e thru 0x014632 (0xb3a5 bytes) 000438 1789124136.800705 mapwrite 0x0153ba thru 0x0165fc (0x1243 bytes) 000439 1789124137.038955 trunc from 0x0165fd to 0x03651a (0x1ff1e bytes) 000441 1789124137.074836 read 0x005cca thru 0x00c50e (0x6845 bytes) 000442 1789124137.089838 mapread 0x00f068 thru 0x00f622 (0x5bb bytes) 000443 1789124137.090899 trunc from 0x03651b to 0x02aef9 (0xb621 bytes) 000446 1789124137.134645 write 0x0117a0 thru 0x01df1c (0xc77d bytes) 000448 1789124137.154517 read 0x026fc3 thru 0x02aef9 (0x3f37 bytes) 000449 1789124137.181046 trunc from 0x02aefa to 0x01aad1 (0x10428 bytes) 000451 1789124137.225907 trunc from 0x01aad2 to 0x02cfee (0x1251d bytes) 000452 1789124137.275496 trunc from 0x02cfef to 0x00f99c (0x1d652 bytes) 000454 1789124137.304760 write 0x03e535 thru 0x03ffff (0x1acb bytes) 000455 1789124137.306410 write 0x00b40e thru 0x010735 (0x5328 bytes) 000456 1789124137.316715 read 0x010acf thru 0x015d4b (0x527d bytes) 000457 1789124137.330391 mapwrite 0x03f82b thru 0x03ffff (0x7d5 bytes) 000459 1789124137.434218 write 0x01267a thru 0x016d26 (0x46ad bytes) 000461 1789124137.445748 MIRROR_RESYNC 11 mirrors 000462 1789124141.013002 MIRROR_EXTEND to 12 mirrors 000463 1789124142.281436 write 0x037ab2 thru 0x03ffff (0x854e bytes) 000466 1789124142.531276 mapwrite 0x03eb19 thru 0x03ffff (0x14e7 bytes) 000467 1789124142.623763 trunc from 0x040000 to 0x027022 (0x18fdd bytes) 000468 1789124142.654525 write 0x0311d9 thru 0x03d31f (0xc147 bytes) 000469 1789124142.656004 trunc from 0x03d320 to 0x02e730 (0xebef bytes) 000472 1789124142.678351 read 0x029720 thru 0x02e730 (0x5011 bytes) 000473 1789124142.688625 read 0x021004 thru 0x02ccf2 (0xbcef bytes) 000474 1789124142.699790 write 0x025091 thru 0x02b520 (0x6490 bytes) 000476 1789124142.701739 write 0x024e77 thru 0x030455 (0xb5df bytes) 000479 1789124142.706274 mapread 0x008f47 thru 0x0142eb (0xb3a5 bytes) 000480 1789124142.707873 read 0x00a671 thru 0x016dc7 (0xc757 bytes) 000481 1789124142.708610 read 0x02a3d2 thru 0x02c910 (0x253f bytes) 000485 1789124142.712166 trunc from 0x030456 to 0x02f43b (0x101a bytes) 000486 1789124142.732562 mapwrite 0x00bd94 thru 0x00eb1f (0x2d8c bytes) 000489 1789124142.784802 read 0x022ebd thru 0x025358 (0x249c bytes) 000490 1789124142.790455 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f43c): Mirror 2: 0x1f62145d (stale) Mirror 5: 0x1f62145d (stale) Mirror 11: 0x1f62145d (stale) Mirror 15: 0x1f62145d (stale) Mirror 16: 0x1f62145d (stale) Mirror 17: 0x1f62145d (stale) Mirror 18: 0x1f62145d (stale) Mirror 19: 0x1f62145d (stale) Mirror 20: 0x1f62145d (stale) Mirror 21: 0x1f62145d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f43c): Mirror 2: 0x1f62145d (stale) Mirror 5: 0x1f62145d (stale) Mirror 11: 0x1f62145d (stale) Mirror 15: 0x1f62145d (stale) Mirror 16: 0x1f62145d (stale) Mirror 17: 0x1f62145d (stale) Mirror 18: 0x1f62145d (stale) Mirror 19: 0x1f62145d (stale) Mirror 20: 0x1f62145d (stale) Mirror 21: 0x1f62145d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f43c: skipped 000491 1789124143.415928 read 0x0060ae thru 0x00ac3e (0x4b91 bytes) 000493 1789124143.426845 mapwrite 0x02d1ef thru 0x0389c2 (0xb7d4 bytes) 000494 1789124143.512244 trunc from 0x0389c3 to 0x0060cb (0x328f7 bytes) 000495 1789124143.531018 read 0x002e18 thru 0x0060cb (0x32b4 bytes) 000497 1789124143.540093 mapread 0x000845 thru 0x0060cb (0x5887 bytes) 000499 1789124143.545353 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x60cc): Mirror 5: 0xf93b01e0 (stale) Mirror 11: 0xf93b01e0 (stale) Mirror 15: 0xf93b01e0 (stale) Mirror 16: 0xf93b01e0 (stale) Mirror 17: 0xf93b01e0 (stale) Mirror 18: 0xf93b01e0 (stale) Mirror 19: 0xf93b01e0 (stale) Mirror 20: 0xf93b01e0 (stale) Mirror 21: 0xf93b01e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x60cc): Mirror 5: 0xf93b01e0 (stale) Mirror 11: 0xf93b01e0 (stale) Mirror 15: 0xf93b01e0 (stale) Mirror 16: 0xf93b01e0 (stale) Mirror 17: 0xf93b01e0 (stale) Mirror 18: 0xf93b01e0 (stale) Mirror 19: 0xf93b01e0 (stale) Mirror 20: 0xf93b01e0 (stale) Mirror 21: 0xf93b01e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x60cc: skipped 000500 1789124144.028156 read 0x00218c thru 0x0060cb (0x3f40 bytes) 000501 1789124144.037785 read 0x0050f5 thru 0x0060cb (0xfd7 bytes) 000502 1789124144.038555 trunc from 0x0060cc to 0x02f35b (0x29290 bytes) 000503 1789124144.065947 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2f35c): Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f35c): Mirror 5: 0x32a8bd13 (stale) Mirror 11: 0x32a8bd13 (stale) Mirror 15: 0x32a8bd13 (stale) Mirror 16: 0x32a8bd13 (stale) Mirror 17: 0x32a8bd13 (stale) Mirror 18: 0x32a8bd13 (stale) Mirror 19: 0x32a8bd13 (stale) Mirror 20: 0x32a8bd13 (stale) Mirror 21: 0x32a8bd13 (stale) CRC-32 checksum value for chunk [0, 0x2f35c): Mirror 1: 0x9856adef Mirror 22: 0x9856adef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f35c: skipped 000504 1789124146.268453 mapread 0x00c6fc thru 0x013e45 (0x774a bytes) 000505 1789124146.308273 mapread 0x020ab0 thru 0x02244b (0x199c bytes) 000507 1789124146.317941 read 0x00b25a thru 0x00f9a1 (0x4748 bytes) 000509 1789124146.318699 write 0x015c74 thru 0x02282b (0xcbb8 bytes) 000510 1789124146.407857 MIRROR_RESYNC 11 mirrors 000511 1789124148.350826 write 0x016e99 thru 0x0174d4 (0x63c bytes) 000513 1789124148.442127 mapread 0x0168fb thru 0x02137d (0xaa83 bytes) 000514 1789124148.466366 write 0x0240da thru 0x02de19 (0x9d40 bytes) 000517 1789124148.493497 mapwrite 0x029703 thru 0x02ee51 (0x574f bytes) 000518 1789124148.628180 trunc from 0x02f35c to 0x03ad5b (0xba00 bytes) 000519 1789124148.653371 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ad5c): Mirror 5: 0x7c977be4 (stale) Mirror 15: 0x7c977be4 (stale) Mirror 16: 0x7c977be4 (stale) Mirror 17: 0x7c977be4 (stale) Mirror 18: 0x7c977be4 (stale) Mirror 19: 0x7c977be4 (stale) Mirror 20: 0x7c977be4 (stale) Mirror 21: 0x7c977be4 (stale) Mirror 22: 0x7c977be4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad5c): Mirror 5: 0x7c977be4 (stale) Mirror 15: 0x7c977be4 (stale) Mirror 16: 0x7c977be4 (stale) Mirror 17: 0x7c977be4 (stale) Mirror 18: 0x7c977be4 (stale) Mirror 19: 0x7c977be4 (stale) Mirror 20: 0x7c977be4 (stale) Mirror 21: 0x7c977be4 (stale) Mirror 22: 0x7c977be4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad5c: skipped 000522 1789124149.413587 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3ad5c): Mirror 5: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad5c): Mirror 5: 0x7c977be4 (stale) Mirror 15: 0x7c977be4 (stale) Mirror 16: 0x7c977be4 (stale) Mirror 17: 0x7c977be4 (stale) Mirror 18: 0x7c977be4 (stale) Mirror 19: 0x7c977be4 (stale) Mirror 20: 0x7c977be4 (stale) Mirror 21: 0x7c977be4 (stale) Mirror 22: 0x7c977be4 (stale) CRC-32 checksum value for chunk [0, 0x3ad5c): Mirror 1: 0xe8bb3bfd Mirror 23: 0xe8bb3bfd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad5c: skipped 000525 1789124150.984917 write 0x007609 thru 0x00a2d6 (0x2cce bytes) 000526 1789124151.104618 read 0x02c4ea thru 0x035ed2 (0x99e9 bytes) 000527 1789124151.114790 trunc from 0x03ad5c to 0x010392 (0x2a9c9 bytes) 000528 1789124151.142747 trunc from 0x010393 to 0x01361f (0x328d bytes) 000529 1789124151.162271 mapwrite 0x008704 thru 0x00aeab (0x27a8 bytes) truncating to largest ever: 0x3d8c9 000531 1789124151.221471 trunc from 0x013620 to 0x03d8c8 (0x2a2a9 bytes) truncating to largest ever: 0x3f4a1 000532 1789124151.252581 trunc from 0x03d8c9 to 0x03f4a0 (0x1bd8 bytes) 000534 1789124151.274637 trunc from 0x03f4a1 to 0x005333 (0x3a16d bytes) 000537 1789124151.297572 read 0x0046db thru 0x005333 (0xc59 bytes) 000538 1789124151.306174 mapwrite 0x01023f thru 0x0197cb (0x958d bytes) 000542 1789124151.411605 trunc from 0x0197cc to 0x016573 (0x3258 bytes) 000543 1789124151.443191 mapread 0x00c0e0 thru 0x01363e (0x755f bytes) 000544 1789124151.444602 trunc from 0x016574 to 0x00e4b5 (0x80be bytes) 000545 1789124151.465963 trunc from 0x00e4b6 to 0x011b44 (0x368f bytes) 000553 1789124151.498090 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11b45): Mirror 5: 0x85cb1bb4 (stale) Mirror 15: 0x85cb1bb4 (stale) Mirror 16: 0x85cb1bb4 (stale) Mirror 18: 0x85cb1bb4 (stale) Mirror 19: 0x85cb1bb4 (stale) Mirror 20: 0x85cb1bb4 (stale) Mirror 21: 0x85cb1bb4 (stale) Mirror 22: 0x85cb1bb4 (stale) Mirror 23: 0x85cb1bb4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11b45): Mirror 5: 0x85cb1bb4 (stale) Mirror 15: 0x85cb1bb4 (stale) Mirror 16: 0x85cb1bb4 (stale) Mirror 18: 0x85cb1bb4 (stale) Mirror 19: 0x85cb1bb4 (stale) Mirror 20: 0x85cb1bb4 (stale) Mirror 21: 0x85cb1bb4 (stale) Mirror 22: 0x85cb1bb4 (stale) Mirror 23: 0x85cb1bb4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11b45: skipped 000558 1789124152.135796 write 0x02df24 thru 0x03560a (0x76e7 bytes) 000559 1789124152.144809 MIRROR_RESYNC 10 mirrors 000560 1789124155.011101 write 0x02e2be thru 0x033932 (0x5675 bytes) 000561 1789124155.125135 write 0x02a7be thru 0x038059 (0xd89c bytes) 000562 1789124155.133778 read 0x01ef4a thru 0x02a0f2 (0xb1a9 bytes) 000563 1789124155.150490 read 0x024bf0 thru 0x031dae (0xd1bf bytes) 000564 1789124155.151481 write 0x02d847 thru 0x034d70 (0x752a bytes) 000565 1789124155.153639 mapwrite 0x00a6b7 thru 0x01165d (0x6fa7 bytes) 000566 1789124155.234934 read 0x037a30 thru 0x038059 (0x62a bytes) 000567 1789124155.251626 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3805a): Mirror 5: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3805a): Mirror 5: 0xe4199687 (stale) Mirror 15: 0xe4199687 (stale) Mirror 16: 0xe4199687 (stale) Mirror 18: 0xe4199687 (stale) Mirror 19: 0xe4199687 (stale) Mirror 20: 0xe4199687 (stale) Mirror 21: 0xe4199687 (stale) Mirror 22: 0xe4199687 (stale) Mirror 23: 0xe4199687 (stale) CRC-32 checksum value for chunk [0, 0x3805a): Mirror 1: 0x3dcc9fec Mirror 24: 0x3dcc9fec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3805a: skipped 000571 1789124156.269255 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3805a): Mirror 5: 0xe4199687 (stale) Mirror 15: 0xe4199687 (stale) Mirror 16: 0xe4199687 (stale) Mirror 18: 0xe4199687 (stale) Mirror 19: 0xe4199687 (stale) Mirror 20: 0xe4199687 (stale) Mirror 22: 0xe4199687 (stale) Mirror 23: 0xe4199687 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3805a): Mirror 5: 0xe4199687 (stale) Mirror 15: 0xe4199687 (stale) Mirror 16: 0xe4199687 (stale) Mirror 18: 0xe4199687 (stale) Mirror 19: 0xe4199687 (stale) Mirror 20: 0xe4199687 (stale) Mirror 22: 0xe4199687 (stale) Mirror 23: 0xe4199687 (stale) CRC-32 checksum value for chunk [0, 0x3805a): Mirror 1: 0x3dcc9fec Mirror 24: 0x3dcc9fec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3805a: skipped 000573 1789124157.257818 mapread 0x02a92a thru 0x0376ef (0xcdc6 bytes) 000575 1789124157.280061 mapwrite 0x03ad6f thru 0x03ffff (0x5291 bytes) 000579 1789124157.491174 trunc from 0x040000 to 0x038400 (0x7bff bytes) 000580 1789124157.544533 MIRROR_RESYNC 10 mirrors 000581 1789124159.504595 read 0x019036 thru 0x028936 (0xf901 bytes) 000582 1789124159.517906 trunc from 0x038401 to 0x02885c (0xfba4 bytes) 000583 1789124159.625653 write 0x035436 thru 0x03ffff (0xabca bytes) 000585 1789124159.647123 MIRROR_RESYNC 10 mirrors 000586 1789124161.429883 mapwrite 0x00e338 thru 0x015d45 (0x7a0e bytes) 000587 1789124161.642677 trunc from 0x040000 to 0x03a8d0 (0x572f bytes) 000588 1789124161.666570 read 0x01fb21 thru 0x029fe9 (0xa4c9 bytes) 000589 1789124161.683027 read 0x00f5d2 thru 0x019739 (0xa168 bytes) 000592 1789124161.703170 mapwrite 0x02cdc1 thru 0x03893e (0xbb7e bytes) 000594 1789124161.749266 mapread 0x039d4d thru 0x03a6ed (0x9a1 bytes) 000595 1789124161.751084 read 0x023faf thru 0x024222 (0x274 bytes) 000596 1789124161.752346 write 0x0191fa thru 0x0192cd (0xd4 bytes) 000599 1789124161.755692 trunc from 0x03a8d1 to 0x008056 (0x3287a bytes) 000601 1789124161.778141 trunc from 0x008057 to 0x0261b9 (0x1e163 bytes) 000602 1789124161.809786 mapread 0x01b3db thru 0x01ecc5 (0x38eb bytes) 000605 1789124161.838105 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x261ba): Mirror 5: 0xb39a2368 (stale) Mirror 16: 0xb39a2368 (stale) Mirror 18: 0xb39a2368 (stale) Mirror 19: 0xb39a2368 (stale) Mirror 20: 0xb39a2368 (stale) Mirror 22: 0xb39a2368 (stale) Mirror 23: 0xb39a2368 (stale) Mirror 24: 0xb39a2368 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x261ba): Mirror 5: 0xb39a2368 (stale) Mirror 16: 0xb39a2368 (stale) Mirror 18: 0xb39a2368 (stale) Mirror 19: 0xb39a2368 (stale) Mirror 20: 0xb39a2368 (stale) Mirror 22: 0xb39a2368 (stale) Mirror 23: 0xb39a2368 (stale) Mirror 24: 0xb39a2368 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x261ba: skipped 000606 1789124162.394385 write 0x015bce thru 0x01eae3 (0x8f16 bytes) 000607 1789124162.407185 trunc from 0x0261ba to 0x020982 (0x5837 bytes) 000608 1789124162.432744 trunc from 0x020983 to 0x014416 (0xc56c bytes) 000609 1789124162.452835 read 0x000d76 thru 0x003625 (0x28b0 bytes) 000611 1789124162.461197 read 0x0080da thru 0x00c0b2 (0x3fd9 bytes) 000612 1789124162.470803 mapread 0x000336 thru 0x0075db (0x72a6 bytes) 000613 1789124162.471960 write 0x02cfe0 thru 0x0304fd (0x351e bytes) 000615 1789124162.479884 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x304fe): Mirror 5: 0x7230e39f (stale) Mirror 16: 0x7230e39f (stale) Mirror 18: 0x7230e39f (stale) Mirror 19: 0x7230e39f (stale) Mirror 20: 0x7230e39f (stale) Mirror 23: 0x7230e39f (stale) Mirror 24: 0x7230e39f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x304fe): Mirror 5: 0x7230e39f (stale) Mirror 16: 0x7230e39f (stale) Mirror 18: 0x7230e39f (stale) Mirror 19: 0x7230e39f (stale) Mirror 20: 0x7230e39f (stale) Mirror 23: 0x7230e39f (stale) Mirror 24: 0x7230e39f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x304fe: skipped 000617 1789124163.101460 write 0x01cc59 thru 0x029340 (0xc6e8 bytes) 000618 1789124163.133123 read 0x028f99 thru 0x0304fd (0x7565 bytes) 000619 1789124163.163665 trunc from 0x0304fe to 0x017a38 (0x18ac5 bytes) 000620 1789124163.182563 mapwrite 0x009785 thru 0x00d482 (0x3cfe bytes) 000622 1789124163.296624 read 0x012b8b thru 0x01721f (0x4695 bytes) 000625 1789124163.315252 write 0x028b4b thru 0x03018c (0x7642 bytes) 000627 1789124163.325295 mapwrite 0x03ab4c thru 0x03d761 (0x2c16 bytes) 000628 1789124163.467267 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d762): Mirror 16: 0xb976d0ee (stale) Mirror 18: 0xb976d0ee (stale) Mirror 19: 0xb976d0ee (stale) Mirror 20: 0xb976d0ee (stale) Mirror 23: 0xb976d0ee (stale) Mirror 24: 0xb976d0ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d762): Mirror 16: 0xb976d0ee (stale) Mirror 18: 0xb976d0ee (stale) Mirror 19: 0xb976d0ee (stale) Mirror 20: 0xb976d0ee (stale) Mirror 23: 0xb976d0ee (stale) Mirror 24: 0xb976d0ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d762: skipped 000629 1789124164.035842 mapwrite 0x001385 thru 0x00fed0 (0xeb4c bytes) 000630 1789124164.138745 mapwrite 0x01c3d5 thru 0x024a38 (0x8664 bytes) 000633 1789124164.215913 write 0x01db35 thru 0x022343 (0x480f bytes) 000635 1789124164.219070 mapread 0x01aa38 thru 0x027973 (0xcf3c bytes) 000637 1789124164.249485 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d762): Mirror 16: 0xb976d0ee (stale) Mirror 18: 0xb976d0ee (stale) Mirror 19: 0xb976d0ee (stale) Mirror 20: 0xb976d0ee (stale) Mirror 24: 0xb976d0ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d762): Mirror 16: 0xb976d0ee (stale) Mirror 18: 0xb976d0ee (stale) Mirror 19: 0xb976d0ee (stale) Mirror 20: 0xb976d0ee (stale) Mirror 24: 0xb976d0ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d762: skipped 000640 1789124164.696747 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d762): Mirror 16: 0xb976d0ee (stale) Mirror 18: 0xb976d0ee (stale) Mirror 20: 0xb976d0ee (stale) Mirror 24: 0xb976d0ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d762): Mirror 16: 0xb976d0ee (stale) Mirror 18: 0xb976d0ee (stale) Mirror 20: 0xb976d0ee (stale) Mirror 24: 0xb976d0ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d762: skipped 000641 1789124165.138784 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d762): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d762): Mirror 16: 0xb976d0ee (stale) Mirror 18: 0xb976d0ee (stale) Mirror 20: 0xb976d0ee (stale) Mirror 24: 0xb976d0ee (stale) CRC-32 checksum value for chunk [0, 0x3d762): Mirror 1: 0x7231f328 Mirror 25: 0x7231f328 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d762: skipped 000643 1789124166.344702 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d762): Mirror 16: 0xb976d0ee (stale) Mirror 20: 0xb976d0ee (stale) Mirror 24: 0xb976d0ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d762): Mirror 16: 0xb976d0ee (stale) Mirror 20: 0xb976d0ee (stale) Mirror 24: 0xb976d0ee (stale) CRC-32 checksum value for chunk [0, 0x3d762): Mirror 1: 0x7231f328 Mirror 25: 0x7231f328 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d762: skipped 000645 1789124167.123523 mapread 0x0322b0 thru 0x03d761 (0xb4b2 bytes) 000647 1789124167.185672 mapwrite 0x02a371 thru 0x02be09 (0x1a99 bytes) 000648 1789124167.604672 mapwrite 0x005635 thru 0x008c5b (0x3627 bytes) 000651 1789124167.741072 mapread 0x0365fa thru 0x03d761 (0x7168 bytes) 000653 1789124167.760853 mapwrite 0x013321 thru 0x0195ee (0x62ce bytes) 000654 1789124167.870868 read 0x0144df thru 0x017fbf (0x3ae1 bytes) 000656 1789124167.873003 mapwrite 0x03c801 thru 0x03ffff (0x37ff bytes) 000657 1789124168.044981 mapwrite 0x03b2dc thru 0x03ffff (0x4d24 bytes) 000658 1789124168.114073 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x68130775 (stale) Mirror 24: 0x68130775 (stale) Mirror 25: 0xadf4fe5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x68130775 (stale) Mirror 24: 0x68130775 (stale) Mirror 25: 0xadf4fe5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000659 1789124168.540894 MIRROR_RESYNC 4 mirrors 000661 1789124169.544244 mapread 0x02582a thru 0x02c1d0 (0x69a7 bytes) 000662 1789124169.579267 mapwrite 0x02ba54 thru 0x039850 (0xddfd bytes) 000663 1789124169.847643 mapwrite 0x037300 thru 0x03bd30 (0x4a31 bytes) 000664 1789124169.938594 write 0x0254c3 thru 0x027cee (0x282c bytes) 000665 1789124169.940201 mapread 0x03e5c8 thru 0x03ffff (0x1a38 bytes) 000667 1789124169.965136 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: 0x986f42b0 (stale) Mirror 25: 0x986f42b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 24: 0x986f42b0 (stale) Mirror 25: 0x986f42b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000671 1789124170.287175 trunc from 0x040000 to 0x0375d2 (0x8a2d bytes) 000672 1789124170.303041 trunc from 0x0375d3 to 0x0281ff (0xf3d3 bytes) 000673 1789124170.333117 MIRROR_RESYNC 3 mirrors 000674 1789124171.355818 mapwrite 0x01ee36 thru 0x02594b (0x6b16 bytes) 000676 1789124171.569847 mapwrite 0x005858 thru 0x014969 (0xf112 bytes) 000678 1789124171.670685 mapwrite 0x002f40 thru 0x00d052 (0xa113 bytes) 000680 1789124171.727596 mapwrite 0x03c33b thru 0x03ffff (0x3cc5 bytes) 000684 1789124171.812038 mapread 0x019cb6 thru 0x01face (0x5e19 bytes) 000687 1789124171.817768 trunc from 0x040000 to 0x023504 (0x1cafb bytes) 000690 1789124171.857612 read 0x00c316 thru 0x017b4a (0xb835 bytes) 000692 1789124171.859378 read 0x013a49 thru 0x018405 (0x49bd bytes) 000694 1789124171.861622 mapread 0x00b04e thru 0x0174ad (0xc460 bytes) 000695 1789124171.865494 mapwrite 0x028bc0 thru 0x02d412 (0x4853 bytes) 000696 1789124172.017709 mapread 0x01e35b thru 0x024be5 (0x688b bytes) 000699 1789124172.037190 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d413): Mirror 25: 0x840f123e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d413): Mirror 25: 0x840f123e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d413: skipped 000700 1789124172.456772 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d413): Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d413): Mirror 25: 0x840f123e (stale) CRC-32 checksum value for chunk [0, 0x2d413): Mirror 1: 0x11cc92bd Mirror 26: 0x11cc92bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d413: skipped 000701 1789124173.404465 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000702 1789124173.767762 trunc from 0x02d413 to 0x027ab4 (0x595e bytes) 000708 1789124173.887209 write 0x019957 thru 0x01f7ef (0x5e99 bytes) 000711 1789124173.898848 trunc from 0x027ab5 to 0x03b80e (0x13d5a bytes) 000713 1789124173.923789 trunc from 0x03b80f to 0x002db1 (0x38a5d bytes) 000714 1789124173.943291 mapwrite 0x0033c1 thru 0x00a328 (0x6f68 bytes) 000715 1789124174.054610 mapwrite 0x02442f thru 0x0263b6 (0x1f88 bytes) 000717 1789124174.139230 read 0x023797 thru 0x0263b6 (0x2c20 bytes) 000718 1789124174.140097 trunc from 0x0263b7 to 0x03032d (0x9f77 bytes) 000719 1789124174.172586 read 0x014b3f thru 0x0199d2 (0x4e94 bytes) 000720 1789124174.188473 mapwrite 0x003294 thru 0x00d883 (0xa5f0 bytes) 000722 1789124174.285985 mapread 0x020161 thru 0x0244d1 (0x4371 bytes) 000724 1789124174.288667 read 0x01bd8a thru 0x02bd5d (0xffd4 bytes) 000726 1789124174.290509 write 0x02812f thru 0x03563b (0xd50d bytes) 000730 1789124174.299670 write 0x02f741 thru 0x03922f (0x9aef bytes) 000733 1789124174.303323 mapread 0x03735e thru 0x03922f (0x1ed2 bytes) 000734 1789124174.304660 read 0x0178da thru 0x018586 (0xcad bytes) 000736 1789124174.306410 write 0x0100ea thru 0x012dff (0x2d16 bytes) 000738 1789124174.308283 write 0x00cd09 thru 0x012932 (0x5c2a bytes) 000740 1789124174.310163 mapwrite 0x036724 thru 0x03ffff (0x98dc bytes) 000741 1789124174.408436 read 0x013e9a thru 0x0176c2 (0x3829 bytes) 000744 1789124174.410083 read 0x020352 thru 0x021631 (0x12e0 bytes) 000746 1789124174.412512 mapwrite 0x03d977 thru 0x03ffff (0x2689 bytes) 000747 1789124174.452262 read 0x030dd9 thru 0x0389cf (0x7bf7 bytes) 000748 1789124174.457186 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: 0x4463baba (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa18e4e7c Mirror 27: 0xa18e4e7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000749 1789124175.260050 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000750 1789124175.708105 mapread 0x01ce8b thru 0x024562 (0x76d8 bytes) 000751 1789124175.736477 trunc from 0x040000 to 0x00e374 (0x31c8b bytes) 000753 1789124175.910350 mapread 0x00d295 thru 0x00e374 (0x10e0 bytes) 000754 1789124175.926619 mapwrite 0x01453e thru 0x01d609 (0x90cc bytes) 000755 1789124176.083177 read 0x01a223 thru 0x01d609 (0x33e7 bytes) 000756 1789124176.085341 mapread 0x015a8b thru 0x01d609 (0x7b7f bytes) 000759 1789124176.095438 mapread 0x0099bd thru 0x0119d4 (0x8018 bytes) 000760 1789124176.098652 mapread 0x019e0f thru 0x01d609 (0x37fb bytes) 000761 1789124176.101273 trunc from 0x01d60a to 0x0301f3 (0x12bea bytes) 000763 1789124176.163661 trunc from 0x0301f4 to 0x028acc (0x7727 bytes) 000764 1789124176.196466 mapwrite 0x02e44b thru 0x03cf7e (0xeb34 bytes) 000765 1789124176.360539 read 0x00aa4f thru 0x00d533 (0x2ae5 bytes) 000766 1789124176.361628 read 0x00f24d thru 0x01cd63 (0xdb17 bytes) 000769 1789124176.365579 read 0x00cf9f thru 0x013aef (0x6b51 bytes) 000770 1789124176.371528 MIRROR_RESYNC 2 mirrors 000772 1789124177.302182 mapwrite 0x0215d2 thru 0x030f25 (0xf954 bytes) 000773 1789124177.512663 MIRROR_RESYNC 2 mirrors 000776 1789124178.485387 mapwrite 0x010e88 thru 0x016ae6 (0x5c5f bytes) 000779 1789124178.699842 trunc from 0x03cf7f to 0x02c71a (0x10864 bytes) 000780 1789124178.736980 mapwrite 0x00e72a thru 0x015682 (0x6f59 bytes) 000783 1789124178.808819 write 0x0174a0 thru 0x0221c6 (0xad27 bytes) 000784 1789124178.816460 write 0x03173b thru 0x03b332 (0x9bf8 bytes) 000785 1789124178.818517 mapread 0x022efc thru 0x023c57 (0xd5c bytes) 000786 1789124178.831019 read 0x035d7f thru 0x03b332 (0x55b4 bytes) 000790 1789124178.834530 write 0x0397b4 thru 0x03dca1 (0x44ee bytes) 000792 1789124178.842983 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000793 1789124179.103757 trunc from 0x03dca2 to 0x0106c8 (0x2d5d9 bytes) 000794 1789124179.164008 MIRROR_EXTEND to 2 mirrors 000797 1789124180.360573 read 0x00f954 thru 0x0106c8 (0xd75 bytes) 000798 1789124180.365072 mapread 0x0007eb thru 0x00392f (0x3145 bytes) 000799 1789124180.386639 trunc from 0x0106c9 to 0x03afe4 (0x2a91c bytes) 000801 1789124180.518810 trunc from 0x03afe5 to 0x0137a3 (0x27841 bytes) 000803 1789124180.567229 mapwrite 0x031f2e thru 0x0348db (0x29ae bytes) 000804 1789124180.681088 mapread 0x014e1e thru 0x017028 (0x220b bytes) 000805 1789124180.694049 trunc from 0x0348dc to 0x03d2de (0x8a03 bytes) 000806 1789124180.719038 trunc from 0x03d2df to 0x009681 (0x33c5d bytes) 000807 1789124180.742292 mapwrite 0x029aee thru 0x02ca98 (0x2fab bytes) 000808 1789124180.844645 read 0x00964a thru 0x017f20 (0xe8d7 bytes) 000809 1789124180.857027 mapread 0x007e0b thru 0x00948e (0x1684 bytes) 000811 1789124180.858620 write 0x00b456 thru 0x00b9c5 (0x570 bytes) 000816 1789124180.864460 trunc from 0x02ca99 to 0x00d573 (0x1f525 bytes) 000817 1789124180.895578 trunc from 0x00d574 to 0x029a16 (0x1c4a3 bytes) 000819 1789124180.927928 mapwrite 0x00ee1b thru 0x013d9e (0x4f84 bytes) 000820 1789124181.028977 mapread 0x00ebfa thru 0x016307 (0x770e bytes) 000822 1789124181.075198 trunc from 0x029a17 to 0x00a383 (0x1f693 bytes) 000824 1789124181.098660 write 0x01b607 thru 0x021470 (0x5e6a bytes) 000828 1789124181.106982 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000831 1789124181.325266 trunc from 0x021471 to 0x02ad93 (0x9923 bytes) 000834 1789124181.366254 write 0x02736d thru 0x036113 (0xeda7 bytes) 000836 1789124181.402446 MIRROR_EXTEND to 2 mirrors 000839 1789124182.464003 mapread 0x01f333 thru 0x028ee6 (0x9bb4 bytes) 000841 1789124182.485416 trunc from 0x036114 to 0x0316e5 (0x4a2e bytes) 000843 1789124182.565558 mapread 0x0038de thru 0x00e0c4 (0xa7e7 bytes) 000844 1789124182.575645 mapwrite 0x026c64 thru 0x029fac (0x3349 bytes) 000845 1789124182.632246 mapread 0x015d22 thru 0x01802d (0x230c bytes) 000846 1789124182.651774 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000847 1789124182.787515 trunc from 0x0316e6 to 0x01c49d (0x15248 bytes) 000848 1789124182.804133 read 0x0096a1 thru 0x00f8e2 (0x6242 bytes) 000849 1789124182.819273 mapwrite 0x00c6b5 thru 0x01b93c (0xf288 bytes) 000850 1789124182.976038 read 0x0013ed thru 0x00649c (0x50b0 bytes) 000851 1789124182.986654 write 0x025f56 thru 0x03531b (0xf3c6 bytes) 000853 1789124182.991012 mapwrite 0x02ff46 thru 0x03342b (0x34e6 bytes) 000854 1789124183.070220 mapread 0x02829e thru 0x034630 (0xc393 bytes) 000855 1789124183.072224 write 0x00317e thru 0x011465 (0xe2e8 bytes) 000856 1789124183.074175 mapwrite 0x00f0ab thru 0x0169ae (0x7904 bytes) 000860 1789124183.148527 read 0x02fab2 thru 0x0304b6 (0xa05 bytes) 000862 1789124183.153238 mapread 0x02ba88 thru 0x02e560 (0x2ad9 bytes) 000866 1789124183.161746 mapwrite 0x02383f thru 0x029132 (0x58f4 bytes) 000870 1789124183.243226 write 0x00cc5e thru 0x00de4a (0x11ed bytes) 000873 1789124183.245504 read 0x00c18f thru 0x019db1 (0xdc23 bytes) 000874 1789124183.246098 read 0x0240d8 thru 0x029859 (0x5782 bytes) 000876 1789124183.247360 write 0x027077 thru 0x029146 (0x20d0 bytes) 000879 1789124183.249737 write 0x01695f thru 0x01f4ba (0x8b5c bytes) 000880 1789124183.250858 write 0x01f352 thru 0x024966 (0x5615 bytes) 000881 1789124183.251966 write 0x037d39 thru 0x03ffff (0x82c7 bytes) 000882 1789124183.253222 trunc from 0x040000 to 0x013e1e (0x2c1e1 bytes) 000883 1789124183.272134 trunc from 0x013e1f to 0x029649 (0x1582b bytes) 000885 1789124183.294280 write 0x025cde thru 0x02d0b8 (0x73db bytes) 000888 1789124183.312417 read 0x00efa2 thru 0x016aa4 (0x7b03 bytes) 000890 1789124183.315568 mapwrite 0x023d56 thru 0x02d33b (0x95e6 bytes) 000891 1789124183.413291 mapread 0x02b2ab thru 0x02d33b (0x2091 bytes) 000892 1789124183.414580 write 0x00aaf2 thru 0x00cda1 (0x22b0 bytes) 000893 1789124183.417039 read 0x0287a3 thru 0x02d33b (0x4b99 bytes) 000899 1789124183.424609 mapwrite 0x002d1a thru 0x00b275 (0x855c bytes) 000901 1789124183.460980 mapread 0x014354 thru 0x0172aa (0x2f57 bytes) 000902 1789124183.463046 read 0x000567 thru 0x00768b (0x7125 bytes) 000909 1789124183.469538 mapread 0x015022 thru 0x01a71b (0x56fa bytes) 000914 1789124183.473789 mapread 0x00112c thru 0x00f19c (0xe071 bytes) 000915 1789124183.477767 read 0x001abd thru 0x005341 (0x3885 bytes) 000917 1789124183.480349 write 0x03b190 thru 0x03ffff (0x4e70 bytes) 000918 1789124183.482643 trunc from 0x040000 to 0x039f66 (0x6099 bytes) 000919 1789124183.506786 mapread 0x014825 thru 0x019e2c (0x5608 bytes) 000920 1789124183.508859 write 0x02960a thru 0x033822 (0xa219 bytes) 000924 1789124183.520254 trunc from 0x039f67 to 0x03df4b (0x3fe5 bytes) 000926 1789124183.544602 write 0x02d6ea thru 0x02eebb (0x17d2 bytes) 000927 1789124183.545339 mapread 0x03b48d thru 0x03df4b (0x2abf bytes) 000929 1789124183.564687 mapwrite 0x031eb3 thru 0x0365f7 (0x4745 bytes) 000930 1789124183.642379 MIRROR_EXTEND to 2 mirrors 000931 1789124184.640128 mapread 0x03cfc9 thru 0x03df4b (0xf83 bytes) 000933 1789124184.669758 trunc from 0x03df4c to 0x01b3f1 (0x22b5a bytes) 000935 1789124184.822784 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000936 1789124185.046455 mapread 0x002dc3 thru 0x00f56a (0xc7a8 bytes) 000938 1789124185.087429 mapread 0x0012a9 thru 0x00920e (0x7f66 bytes) 000940 1789124185.093863 mapread 0x010c5a thru 0x01b3f1 (0xa798 bytes) 000941 1789124185.127027 mapread 0x001ee5 thru 0x00a805 (0x8921 bytes) 000943 1789124185.131630 mapread 0x00a7f9 thru 0x00c9ee (0x21f6 bytes) 000944 1789124185.133224 trunc from 0x01b3f2 to 0x02e53b (0x1314a bytes) 000946 1789124185.172549 read 0x005f91 thru 0x0115b6 (0xb626 bytes) 000947 1789124185.174108 write 0x0343a5 thru 0x03f0ec (0xad48 bytes) 000949 1789124185.178162 mapwrite 0x027e3f thru 0x030d78 (0x8f3a bytes) 000954 1789124185.284020 MIRROR_EXTEND to 2 mirrors 000955 1789124186.054558 trunc from 0x03f0ed to 0x02db90 (0x1155c bytes) 000957 1789124186.273140 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000959 1789124186.532283 write 0x01a763 thru 0x01c1af (0x1a4d bytes) 000961 1789124186.553856 write 0x0344f9 thru 0x03485e (0x366 bytes) 000962 1789124186.558691 mapread 0x016143 thru 0x01962f (0x34ed bytes) 000964 1789124186.626056 trunc from 0x03485f to 0x00c4bd (0x283a1 bytes) 000966 1789124186.690681 trunc from 0x00c4be to 0x01f6bb (0x131fe bytes) 000967 1789124186.718432 read 0x00a056 thru 0x00c151 (0x20fc bytes) 000968 1789124186.731202 trunc from 0x01f6bc to 0x021b9c (0x24e1 bytes) 000970 1789124186.769745 read 0x001e39 thru 0x00c66a (0xa832 bytes) 000971 1789124186.787967 write 0x020b28 thru 0x024b3b (0x4014 bytes) 000972 1789124186.797218 MIRROR_EXTEND to 2 mirrors 000973 1789124188.088516 mapread 0x0210d6 thru 0x0218cd (0x7f8 bytes) 000978 1789124188.099847 trunc from 0x024b3c to 0x02b6da (0x6b9f bytes) 000979 1789124188.201237 mapwrite 0x00354e thru 0x006e2f (0x38e2 bytes) 000981 1789124188.256683 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ac:0x0]: Invalid argument (22) 000982 1789124188.423702 trunc from 0x02b6db to 0x0002e1 (0x2b3f9 bytes) 000985 1789124188.463771 mapwrite 0x0000a3 thru 0x00f348 (0xf2a6 bytes) 000986 1789124188.595640 read 0x00b695 thru 0x00dc4a (0x25b6 bytes) 000989 1789124188.598134 mapwrite 0x03a095 thru 0x03ffff (0x5f6b bytes) 000991 1789124188.695454 read 0x024ad1 thru 0x02dee9 (0x9419 bytes) 000994 1789124188.713984 read 0x021643 thru 0x022004 (0x9c2 bytes) 000995 1789124188.715222 mapwrite 0x033182 thru 0x036b57 (0x39d6 bytes) 000996 1789124188.811400 trunc from 0x040000 to 0x00ba70 (0x3458f bytes) 000997 1789124188.832406 trunc from 0x00ba71 to 0x030003 (0x24593 bytes) 000998 1789124188.851777 mapread 0x02a5d5 thru 0x030003 (0x5a2f bytes) All operations completed A-OK! PASS 70a (122s) == sanity-flr test 71: check mirror extend parent fid ==== 06:56:34 (1789124194) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 06:56:39 (1789124199) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.420903 s, 5.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20153 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 06:56:48 (1789124208) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre2 Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6123456 .. Extending file size to 6687040 .. Extending file size to 6898944 .. Extending file size to 8201792 .. done Extending file size to 8598080 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6927072 .. Extending file size to 7739904 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8694784 .. Extending file size to 8035680 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8325504 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8957440 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9195872 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58932 58933 58935 58936 58939 done 192.168.206.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre2 (opts:) 192.168.206.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:58:08 (1789124288) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre2 Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5283200 .. done Extending file size to 378208 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6335776 .. Extending file size to 8297856 .. Extending file size to 6693792 .. Extending file size to 9033472 .. Extending file size to 7058528 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7193984 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9044992 .. Extending file size to 8388224 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8449568 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8686048 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9062208 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 60656 60657 60658 60661 60664 192.168.206.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre2 (opts:) 192.168.206.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:59:25 (1789124365) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155782 s, 13.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.408884 s, 5.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xc3a63e2f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.206.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:59:45 (1789124385) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0931538 s, 22.5 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 06:59:52 (1789124392) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fd:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:59:58 (1789124398) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.271234 s, 7.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:00:06 (1789124406) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.240775 s, 8.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:00:16 (1789124416) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.529526 s, 4.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:00:51 (1789124451) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg610-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142653 s, 7.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:01:50 (1789124510) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg610-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.360046 s, 5.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:02:52 (1789124572) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg610-server mds-ost sync done. Creating new pool test_204f oleg610-server: Pool lustre.test_204f created Adding targets to pool oleg610-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg610-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21314 s, 9.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg610-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg610-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg610-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:04:09 (1789124649) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:04:18 (1789124658) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:04:25 (1789124665) oleg610-server: Pool lustre.test_206 created Adding targets to pool oleg610-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg610-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg610-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg610-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg610-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:04:51 (1789124691) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101574 s, 10.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843878 s, 12.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xa7185830 Mirror 3: 0xa7185830 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:05:02 (1789124702) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:05:06 (1789124706) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:05:09 (1789124709) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.185486 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xaf0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb00000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb00000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb00000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2440000 Read this (4 bytes): BBBB 192.168.206.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:05:48 (1789124748) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0297513 s, 275 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:05:55 (1789124755) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490636 s, 21.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x222 lmm_fid: [0x200000404:0x222:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x222 lmm_fid: [0x200000404:0x222:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834466 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 07:05:57.000000000 -0400 Modify: 2026-09-11 07:05:57.000000000 -0400 Change: 2026-09-11 07:05:58.000000000 -0400 Birth: 2026-09-11 07:05:57.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:06:03 (1789124763) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:06:06 (1789124766) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.629854 s, 65.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0659034 s, 62.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 11 07:06 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x224:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 11 07:06 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:06:14 (1789124774) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:06:22 (1789124782) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1775 sec =========== 07:06:30 (1789124790) osc.lustre-OST0000-osc-ffff97359883b000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff97359883b000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1789124793.528595120 secs.nsecs start_time 1789123477.206994261 secs.nsecs elapsed_time 1316.321600859 secs.nsecs uncompressed_objects 2595 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1789124793.528924853 secs.nsecs start_time 1789123508.596843338 secs.nsecs elapsed_time 1284.932081515 secs.nsecs uncompressed_objects 2587 samples [reqs] === sanity-flr: start cleanup 07:06:33 (1789124793) === === sanity-flr: finish cleanup 07:06:39 (1789124799) ===