-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 20:32:59 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg326-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:33:13 (1787790793) === oleg326-client.virtnet: executing check_config_client /mnt/lustre oleg326-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg326-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d8a05289800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d8a05289800.idle_timeout=debug disable quota as required oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:33:36 (1787790816) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7634] == sanity-flr test 0a: lfs mirror create with -N option == 20:33:37 (1787790817) PASS 0a (11s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:33:48 (1787790828) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:33:50 (1787790830) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:33:52 (1787790832) PASS 0d (17s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:34:09 (1787790849) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:34:11 (1787790851) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:34:14 (1787790854) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d8a05289800.stats=clear osc.lustre-OST0001-osc-ffff8d8a05289800.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:34:19 (1787790859) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:34:27 (1787790867) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:34:33 (1787790873) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:34:36 (1787790876) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:34:42 (1787790882) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 20:34:50 (1787790890) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:34:56 (1787790896) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:34:58 (1787790898) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:35:04 (1787790904) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:35:13 (1787790913) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:35:18 (1787790918) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.70404 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116088 s, 9.0 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:35:27 (1787790927) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.367181 s, 45.7 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:35:34 (1787790934) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:35:42 (1787790942) Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8a05289800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8a05289800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d8a05289800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8a05289800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (50s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:36:32 (1787790992) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.245832 s, 25.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8a05289800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8a05289800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (30s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:37:03 (1787791023) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8a05289800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8a05289800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d8a05289800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8a05289800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (59s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:38:01 (1787791081) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.68485 s, 14.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 20:38 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.703s user 0m0.001s sys 0m0.171s Stopping /mnt/lustre-ost1 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8a05289800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8a05289800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.872s user 0m0.000s sys 0m0.176s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg326-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg326-server oleg326-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d8a05289800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8a05289800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.838s user 0m0.000s sys 0m0.156s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg326-server: oleg326-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg326-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d8a05289800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8a05289800.ost_server_uuid in FULL state after 0 sec PASS 33b (68s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:39:10 (1787791150) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:39:12 (1787791152) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:39:14 (1787791154) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 20:39:17 (1787791157) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 20:39:22 (1787791162) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.436283 s, 28.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:39:32 (1787791172) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292494 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270906 s, 38.7 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:39:38 (1787791178) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:39:45 (1787791185) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00158105 s, 648 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121555 s, 842 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00131301 s, 780 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000789005 s, 1.3 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 20:39:53 (1787791193) create test files with size 15759287 .. 87e4295e3db24f2884f9142da6ba0e86 - 5f422ee9cf0fc7f62c47ac1b07816b9b - a4eea5cffff5b686726d422ce119d737 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 20:40:13 (1787791213) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:40:24 (1787791224) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:40:30 (1787791230) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142169 s, 7.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201134 s, 52.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 20:40:38 (1787791238) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155275 s, 13.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.281749 s, 14.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.259557 s, 8.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 20:40:51 (1787791251) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 20:40:53 (1787791253) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:40:56 (1787791256) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399257 s, 26.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 43c: read prefer uncompressed mirror == 20:41:01 (1787791261) llite.lustre-ffff8d8a098d3000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (5s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 20:41:06 (1787791266) llite.lustre-ffff8d8a098d3000.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272138 s, 38.5 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466884 s, 22.5 MB/s **verify components PASS 43d (5s) == sanity-flr test 44a: lfs mirror split check =========== 20:41:11 (1787791271) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:41:14 (1787791274) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:41:21 (1787791281) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.214688 s, 48.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:41:31 (1787791291) llite.lustre-ffff8d8a06ae9800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.20911 s, 50.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8d8a098d7000.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:41:41 (1787791301) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (32s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:42:13 (1787791333) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:42:43 (1787791363) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 20:42:45 (1787791365) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.240249 s, 13.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.137602 s, 22.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb62093fc (nosync,stale) Mirror 2: 0xb62093fc (nosync,stale) ** verify mirror 2 doesn't change original checksum: cf2908560f2b9b33af934882dc2041ad - mirror 1 checksum: d9d9943a871fa063bb521057b6eb1f22 - mirror 2 checksum: cf2908560f2b9b33af934882dc2041ad - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T20:42:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T20:42:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: cf2908560f2b9b33af934882dc2041ad - mirror 1 checksum: d9d9943a871fa063bb521057b6eb1f22 - mirror 2 checksum: d9d9943a871fa063bb521057b6eb1f22 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T20:42:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T20:42:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } PASS 48 (8s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:42:54 (1787791374) Failing mds1 on oleg326-server Stopping /mnt/lustre-mds1 (opts:) on oleg326-server 20:42:59 (1787791379) shut down facet: mds1 facet_host: oleg326-server facet_failover_host: oleg326-server Failover mds1 to oleg326-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-MDT0000 20:43:19 (1787791399) targets are mounted 20:43:19 (1787791399) facet_failover done oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220205 s, 47.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xeb:0x0] } PASS 50A (37s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:43:31 (1787791411) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:43:33 (1787791413) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:43:36 (1787791416) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00507157 s, 3.2 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:43:41 (1787791421) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:43:44 (1787791424) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00516576 s, 793 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0561331 s, 73.0 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:43:51 (1787791431) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (17s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:44:08 (1787791448) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (24s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:44:33 (1787791473) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:44:54 (1787791494) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:45:16 (1787791516) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 20:45:22 (1787791522) mirror create pid 55527 mirror split pid 55529 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55527 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55529 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 20:46:28 (1787791588) Seed set to 1589 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1787791589.948492 write 0x02499c thru 0x02aa8d (0x60f2 bytes) 000005 1787791589.951125 mapread 0x01b53c thru 0x025849 (0xa30e bytes) 000006 1787791589.965756 write 0x016970 thru 0x01a1fa (0x388b bytes) 000007 1787791589.975944 mapread 0x00df04 thru 0x01345d (0x555a bytes) 000010 1787791589.990733 MIRROR_EXTEND to 2 mirrors 000014 1787791590.880524 mapread 0x011f70 thru 0x0168a8 (0x4939 bytes) 000015 1787791590.907838 write 0x03c085 thru 0x03ffff (0x3f7b bytes) 000017 1787791591.001243 mapread 0x0174ec thru 0x01767f (0x194 bytes) 000018 1787791591.015221 read 0x02e9e7 thru 0x0395cb (0xabe5 bytes) 000019 1787791591.023930 read 0x00cb6a thru 0x01c128 (0xf5bf bytes) 000025 1787791591.039688 mapwrite 0x000054 thru 0x007a10 (0x79bd bytes) 000026 1787791591.108115 mapread 0x02fa12 thru 0x0334a4 (0x3a93 bytes) 000027 1787791591.109089 write 0x0364a4 thru 0x03ffff (0x9b5c bytes) 000028 1787791591.111113 read 0x011a5b thru 0x013fc3 (0x2569 bytes) 000029 1787791591.116406 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x56df83d9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8ce0e2b6 Mirror 3: 0x8ce0e2b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000030 1787791592.167196 read 0x03f949 thru 0x03ffff (0x6b7 bytes) 000032 1787791592.178939 mapwrite 0x019a5b thru 0x01a0ff (0x6a5 bytes) 000034 1787791592.413474 write 0x03e74c thru 0x03f988 (0x123d bytes) 000038 1787791592.428090 mapread 0x0391fd thru 0x03ffff (0x6e03 bytes) truncating to largest ever: 0x36627 000039 1787791592.445857 trunc from 0x040000 to 0x036626 (0x99d9 bytes) truncating to largest ever: 0x3ec96 000040 1787791592.517786 trunc from 0x036627 to 0x03ec95 (0x866f bytes) 000041 1787791592.544478 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ec96): Mirror 3: 0x97c4fc17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ec96): Mirror 3: 0x97c4fc17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ec96: skipped 000042 1787791592.889995 mapread 0x0027b8 thru 0x00313d (0x986 bytes) 000044 1787791592.905858 mapwrite 0x00e088 thru 0x015f1f (0x7e98 bytes) 000045 1787791592.999719 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000046 1787791593.148431 trunc from 0x03ec96 to 0x0254ea (0x197ab bytes) 000048 1787791593.170015 read 0x01bc33 thru 0x0251b6 (0x9584 bytes) 000051 1787791593.201228 write 0x03f47b thru 0x03ffff (0xb85 bytes) 000053 1787791593.204118 write 0x014775 thru 0x024715 (0xffa1 bytes) 000054 1787791593.216849 read 0x01526e thru 0x018207 (0x2f9a bytes) 000057 1787791593.221657 write 0x036cce thru 0x038457 (0x178a bytes) 000058 1787791593.235417 read 0x0387f7 thru 0x03ffff (0x7809 bytes) 000059 1787791593.236282 trunc from 0x040000 to 0x035a86 (0xa579 bytes) 000062 1787791593.270972 trunc from 0x035a87 to 0x030dc4 (0x4cc2 bytes) 000065 1787791593.318750 MIRROR_EXTEND to 2 mirrors 000066 1787791594.083391 mapwrite 0x0009ba thru 0x00a2ad (0x98f4 bytes) 000068 1787791594.259796 read 0x014424 thru 0x0215f7 (0xd1d4 bytes) 000070 1787791594.274551 mapread 0x02988b thru 0x029bce (0x344 bytes) 000071 1787791594.285923 read 0x025ada thru 0x030dc4 (0xb2eb bytes) 000072 1787791594.287246 trunc from 0x030dc5 to 0x004a77 (0x2c34d bytes) 000075 1787791594.346652 mapread 0x002362 thru 0x003cae (0x194d bytes) 000076 1787791594.347467 trunc from 0x004a78 to 0x026223 (0x217ac bytes) 000077 1787791594.376193 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000078 1787791594.501704 write 0x0301ab thru 0x03ef34 (0xed8a bytes) 000079 1787791594.503233 write 0x03f890 thru 0x03ffff (0x770 bytes) 000080 1787791594.505908 mapwrite 0x008290 thru 0x00f6b9 (0x742a bytes) 000083 1787791594.596387 mapwrite 0x0019e4 thru 0x00d9cb (0xbfe8 bytes) 000084 1787791594.673077 mapwrite 0x01eb06 thru 0x0252cb (0x67c6 bytes) 000086 1787791594.777050 read 0x02bc19 thru 0x03acde (0xf0c6 bytes) 000089 1787791594.793103 mapwrite 0x00b9b5 thru 0x00ca0f (0x105b bytes) 000090 1787791594.858064 MIRROR_EXTEND to 2 mirrors 000095 1787791595.716836 trunc from 0x040000 to 0x02dc76 (0x12389 bytes) 000098 1787791595.812431 mapwrite 0x02111d thru 0x02559c (0x4480 bytes) 000099 1787791595.897437 mapwrite 0x001b04 thru 0x00f3c4 (0xd8c1 bytes) 000100 1787791596.046826 write 0x01cf2f thru 0x021779 (0x484b bytes) 000101 1787791596.047913 write 0x03f0f2 thru 0x03ffff (0xf0e bytes) 000102 1787791596.050023 read 0x02c353 thru 0x030b4d (0x47fb bytes) 000106 1787791596.062704 write 0x039669 thru 0x03ffff (0x6997 bytes) 000107 1787791596.073654 read 0x02ac17 thru 0x02eb39 (0x3f23 bytes) 000109 1787791596.075184 mapwrite 0x01a13a thru 0x029918 (0xf7df bytes) 000110 1787791596.170934 mapwrite 0x008138 thru 0x00c1da (0x40a3 bytes) 000113 1787791596.241252 mapread 0x03497f thru 0x03ffff (0xb681 bytes) 000114 1787791596.242173 trunc from 0x040000 to 0x01af4b (0x250b4 bytes) 000116 1787791596.276523 trunc from 0x01af4c to 0x00f9e7 (0xb564 bytes) 000118 1787791596.314287 trunc from 0x00f9e8 to 0x039370 (0x29989 bytes) 000119 1787791596.327905 trunc from 0x039371 to 0x013ef7 (0x25479 bytes) 000122 1787791596.353596 trunc from 0x013ef8 to 0x0100fe (0x3df9 bytes) 000123 1787791596.374721 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x100ff): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x100ff): Mirror 2: 0x47bfc04d (stale) CRC-32 checksum value for chunk [0, 0x100ff): Mirror 1: 0xdc909a3f Mirror 3: 0xdc909a3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x100ff: skipped 000124 1787791597.156303 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x100ff): Mirror 2: 0x47bfc04d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x100ff): Mirror 2: 0x47bfc04d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x100ff: skipped 000125 1787791597.386622 mapread 0x002796 thru 0x003131 (0x99c bytes) 000126 1787791597.397588 mapwrite 0x03d813 thru 0x03ffff (0x27ed bytes) 000127 1787791597.523948 MIRROR_RESYNC 2 mirrors 000129 1787791598.173614 mapwrite 0x01a763 thru 0x022205 (0x7aa3 bytes) 000130 1787791598.344066 trunc from 0x040000 to 0x023384 (0x1cc7b bytes) 000131 1787791598.413784 MIRROR_RESYNC 2 mirrors 000135 1787791599.237714 trunc from 0x023385 to 0x0008b4 (0x22ad0 bytes) 000137 1787791599.319207 mapread 0x000232 thru 0x0008b4 (0x683 bytes) 000138 1787791599.324315 mapread 0x0003fa thru 0x0008b4 (0x4bb bytes) 000140 1787791599.333957 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x8b5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8b5): Mirror 2: 0x5dea860b (stale) CRC-32 checksum value for chunk [0, 0x8b5): Mirror 1: 0x5dea860b Mirror 3: 0x5dea860b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8b5: skipped 000141 1787791600.071366 mapread 0x00005e thru 0x0008b4 (0x857 bytes) 000142 1787791600.078965 mapread 0x0004d4 thru 0x0008b4 (0x3e1 bytes) 000144 1787791600.080988 trunc from 0x0008b5 to 0x020019 (0x1f765 bytes) 000145 1787791600.189880 MIRROR_RESYNC 3 mirrors 000146 1787791600.948101 mapread 0x0100e7 thru 0x01a19c (0xa0b6 bytes) 000148 1787791600.970255 mapwrite 0x01c3ca thru 0x02b695 (0xf2cc bytes) 000150 1787791601.132682 MIRROR_RESYNC 3 mirrors 000151 1787791601.907435 MIRROR_RESYNC 3 mirrors 000153 1787791602.339212 mapwrite 0x016fc2 thru 0x020535 (0x9574 bytes) 000157 1787791602.531194 mapwrite 0x00fbf5 thru 0x0162fb (0x6707 bytes) 000159 1787791602.588167 write 0x026191 thru 0x02731e (0x118e bytes) 000160 1787791602.589410 write 0x00f04e thru 0x0161bb (0x716e bytes) 000161 1787791602.594741 MIRROR_RESYNC 3 mirrors 000164 1787791603.223851 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000166 1787791603.453338 write 0x010caf thru 0x016ce6 (0x6038 bytes) 000167 1787791603.540444 MIRROR_RESYNC 2 mirrors 000169 1787791604.133699 mapwrite 0x0019af thru 0x004ac7 (0x3119 bytes) 000170 1787791604.271984 read 0x016708 thru 0x022a90 (0xc389 bytes) 000171 1787791604.289017 mapwrite 0x032baa thru 0x03fa2e (0xce85 bytes) 000172 1787791604.391707 read 0x00a027 thru 0x0175f6 (0xd5d0 bytes) 000173 1787791604.413479 MIRROR_RESYNC 2 mirrors 000174 1787791604.897178 trunc from 0x03fa2f to 0x010459 (0x2f5d5 bytes) 000175 1787791605.009606 write 0x00f32c thru 0x0190f4 (0x9dc9 bytes) 000176 1787791605.018598 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x190f5): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x190f5): Mirror 3: 0x9428f728 (stale) CRC-32 checksum value for chunk [0, 0x190f5): Mirror 1: 0x1b7f2188 Mirror 4: 0x1b7f2188 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x190f5: skipped 000177 1787791605.651587 trunc from 0x0190f5 to 0x01bf78 (0x2e84 bytes) 000180 1787791605.712712 read 0x0097a7 thru 0x012814 (0x906e bytes) 000182 1787791605.726184 mapwrite 0x032ed2 thru 0x03b335 (0x8464 bytes) 000183 1787791605.817647 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b336): Mirror 4: 0xd58f69d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b336): Mirror 4: 0xd58f69d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b336: skipped 000185 1787791606.132431 write 0x03034c thru 0x03a07d (0x9d32 bytes) 000186 1787791606.170745 MIRROR_RESYNC 2 mirrors 000188 1787791607.011475 write 0x030455 thru 0x031eec (0x1a98 bytes) 000190 1787791607.082617 mapwrite 0x015efa thru 0x025e4f (0xff56 bytes) 000196 1787791607.168791 read 0x00ff35 thru 0x014a53 (0x4b1f bytes) 000199 1787791607.176528 write 0x009f69 thru 0x019880 (0xf918 bytes) 000200 1787791607.185216 write 0x000afd thru 0x00d3e7 (0xc8eb bytes) 000203 1787791607.199262 mapwrite 0x03fa34 thru 0x03ffff (0x5cc bytes) 000204 1787791607.295280 mapwrite 0x002275 thru 0x006582 (0x430e bytes) 000208 1787791607.377499 read 0x039b56 thru 0x03ffff (0x64aa bytes) 000210 1787791607.380166 mapread 0x002048 thru 0x00bb42 (0x9afb bytes) 000212 1787791607.384342 mapwrite 0x00d728 thru 0x00efa9 (0x1882 bytes) 000213 1787791607.461977 read 0x0163c4 thru 0x02172e (0xb36b bytes) 000214 1787791607.468581 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6b22c2f0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x81088790 Mirror 5: 0x81088790 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000215 1787791608.342781 mapwrite 0x0278e7 thru 0x031f8e (0xa6a8 bytes) 000219 1787791608.524409 read 0x0187fa thru 0x01a546 (0x1d4d bytes) 000221 1787791608.532355 mapread 0x01f382 thru 0x02583e (0x64bd bytes) 000225 1787791608.544218 mapread 0x000324 thru 0x006325 (0x6002 bytes) 000226 1787791608.559595 trunc from 0x040000 to 0x034033 (0xbfcc bytes) 000228 1787791608.621251 MIRROR_RESYNC 3 mirrors 000229 1787791609.513243 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000231 1787791609.805497 mapwrite 0x0195d0 thru 0x01c5e1 (0x3012 bytes) 000234 1787791609.960893 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34034): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34034): Mirror 4: 0x60e04745 (stale) CRC-32 checksum value for chunk [0, 0x34034): Mirror 1: 0x118844a3 Mirror 5: 0x118844a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34034: skipped 000235 1787791610.754626 read 0x02348c thru 0x0308ef (0xd464 bytes) 000240 1787791610.770084 write 0x017eab thru 0x01b828 (0x397e bytes) 000241 1787791610.843426 write 0x00bf48 thru 0x00fae4 (0x3b9d bytes) 000242 1787791610.857198 mapread 0x0005cc thru 0x004013 (0x3a48 bytes) 000244 1787791610.887437 read 0x031b24 thru 0x034033 (0x2510 bytes) 000245 1787791610.911568 MIRROR_RESYNC 3 mirrors 000249 1787791611.668387 mapread 0x00e274 thru 0x011949 (0x36d6 bytes) 000250 1787791611.706605 mapread 0x02a208 thru 0x02de78 (0x3c71 bytes) 000251 1787791611.716475 mapread 0x02e44f thru 0x0325dc (0x418e bytes) 000252 1787791611.717172 mapread 0x02e004 thru 0x02ee07 (0xe04 bytes) 000253 1787791611.717698 trunc from 0x034034 to 0x005540 (0x2eaf3 bytes) 000254 1787791611.817395 mapread 0x00006f thru 0x0011ec (0x117e bytes) 000256 1787791611.835684 mapwrite 0x03c42a thru 0x03ffff (0x3bd6 bytes) 000257 1787791611.925882 mapread 0x00fe65 thru 0x01b446 (0xb5e2 bytes) 000259 1787791611.946513 mapwrite 0x0245b3 thru 0x027ae4 (0x3532 bytes) 000260 1787791612.014088 trunc from 0x040000 to 0x01da0f (0x225f0 bytes) 000261 1787791612.028386 write 0x037904 thru 0x03ffff (0x86fc bytes) 000262 1787791612.030396 write 0x03d863 thru 0x03e799 (0xf37 bytes) 000263 1787791612.031503 trunc from 0x040000 to 0x037e3f (0x81c0 bytes) 000264 1787791612.048446 mapread 0x00f124 thru 0x011395 (0x2272 bytes) 000265 1787791612.049580 read 0x0041d3 thru 0x004d2b (0xb59 bytes) 000268 1787791612.051431 read 0x00f782 thru 0x011ec5 (0x2744 bytes) 000269 1787791612.052425 trunc from 0x037e40 to 0x036192 (0x1cad bytes) 000270 1787791612.068634 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36193): Mirror 4: 0x28da2f01 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36193): Mirror 4: 0x28da2f01 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36193: skipped 000273 1787791612.278727 read 0x02047d thru 0x02a849 (0xa3cd bytes) 000274 1787791612.287899 trunc from 0x036193 to 0x01456e (0x21c24 bytes) 000275 1787791612.310118 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1456f): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1456f): Mirror 4: 0xffab824e (stale) CRC-32 checksum value for chunk [0, 0x1456f): Mirror 1: 0xf1349ecb Mirror 5: 0xf1349ecb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1456f: skipped 000276 1787791613.098718 mapwrite 0x01c9a4 thru 0x01f5b2 (0x2c0f bytes) 000277 1787791613.263473 mapread 0x0026b9 thru 0x00fe3f (0xd787 bytes) 000278 1787791613.276273 read 0x01b96d thru 0x01f5b2 (0x3c46 bytes) 000279 1787791613.277164 write 0x03f708 thru 0x03ffff (0x8f8 bytes) 000281 1787791613.281237 write 0x01cd54 thru 0x02b5ae (0xe85b bytes) 000284 1787791613.289219 read 0x03d7e8 thru 0x03ffff (0x2818 bytes) 000285 1787791613.298470 mapwrite 0x01682b thru 0x016849 (0x1f bytes) 000286 1787791613.342381 trunc from 0x040000 to 0x01fab0 (0x2054f bytes) 000289 1787791613.357717 mapwrite 0x02045b thru 0x028d08 (0x88ae bytes) 000290 1787791613.461943 mapread 0x020836 thru 0x028d08 (0x84d3 bytes) 000291 1787791613.465852 mapread 0x0274fd thru 0x028d08 (0x180c bytes) 000294 1787791613.468473 mapread 0x0031b5 thru 0x011221 (0xe06d bytes) 000295 1787791613.473411 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28d09): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28d09): Mirror 4: 0xd948a18a (stale) Mirror 5: 0x8a37173b (stale) CRC-32 checksum value for chunk [0, 0x28d09): Mirror 1: 0xd54b9394 Mirror 6: 0xd54b9394 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28d09: skipped 000296 1787791614.185668 write 0x01e87b thru 0x021d16 (0x349c bytes) 000297 1787791614.277697 trunc from 0x028d09 to 0x038028 (0xf320 bytes) 000298 1787791614.322375 trunc from 0x038029 to 0x019de3 (0x1e245 bytes) 000299 1787791614.345824 read 0x006b38 thru 0x015708 (0xebd1 bytes) 000303 1787791614.377332 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x19de4): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19de4): Mirror 4: 0x838f18a6 (stale) Mirror 5: 0xc84a67d9 (stale) Mirror 6: 0x7d39c8f1 (stale) CRC-32 checksum value for chunk [0, 0x19de4): Mirror 1: 0x7d39c8f1 Mirror 7: 0x7d39c8f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19de4: skipped 000306 1787791615.191568 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19de4): Mirror 5: 0xc84a67d9 (stale) Mirror 6: 0x7d39c8f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19de4): Mirror 5: 0xc84a67d9 (stale) Mirror 6: 0x7d39c8f1 (stale) CRC-32 checksum value for chunk [0, 0x19de4): Mirror 1: 0x7d39c8f1 Mirror 7: 0x7d39c8f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19de4: skipped 000308 1787791615.519449 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x19de4): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19de4): Mirror 5: 0xc84a67d9 (stale) Mirror 6: 0x7d39c8f1 (stale) CRC-32 checksum value for chunk [0, 0x19de4): Mirror 1: 0x7d39c8f1 Mirror 7: 0x7d39c8f1 Mirror 8: 0x7d39c8f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19de4: skipped 000309 1787791616.462119 write 0x00213b thru 0x00d9ef (0xb8b5 bytes) 000311 1787791616.567650 read 0x009ce0 thru 0x00b22b (0x154c bytes) 000313 1787791616.569079 mapwrite 0x016c35 thru 0x01ba5b (0x4e27 bytes) 000315 1787791616.689874 read 0x003b75 thru 0x0048e4 (0xd70 bytes) 000316 1787791616.690719 read 0x00df5b thru 0x00fc96 (0x1d3c bytes) 000318 1787791616.704575 write 0x01f4f3 thru 0x0205b3 (0x10c1 bytes) 000320 1787791616.706770 mapwrite 0x02c940 thru 0x02de6f (0x1530 bytes) 000321 1787791616.774657 mapread 0x004f0b thru 0x0095a3 (0x4699 bytes) 000322 1787791616.775604 mapread 0x01596c thru 0x01fae8 (0xa17d bytes) 000323 1787791616.781291 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2de70): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2de70): Mirror 5: 0x30793afc (stale) Mirror 6: 0xb6ab7f03 (stale) Mirror 7: 0x2fa34f42 (stale) Mirror 8: 0x2fa34f42 (stale) CRC-32 checksum value for chunk [0, 0x2de70): Mirror 1: 0xfaa0efa3 Mirror 9: 0xfaa0efa3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2de70: skipped 000325 1787791617.563555 trunc from 0x02de70 to 0x031514 (0x36a5 bytes) 000329 1787791617.648404 write 0x01225d thru 0x01414f (0x1ef3 bytes) 000330 1787791617.662241 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31515): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31515): Mirror 5: 0x3a64d0ee (stale) Mirror 6: 0x13d25d2c (stale) Mirror 7: 0xaaa4c2fd (stale) Mirror 8: 0xaaa4c2fd (stale) Mirror 9: 0x820223c4 (stale) CRC-32 checksum value for chunk [0, 0x31515): Mirror 1: 0xf67016df Mirror 10: 0xf67016df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31515: skipped 000331 1787791618.640108 mapread 0x007d2e thru 0x0102c7 (0x859a bytes) 000334 1787791618.656912 mapread 0x00dbec thru 0x01d07b (0xf490 bytes) 000335 1787791618.674789 mapwrite 0x02fe72 thru 0x02febc (0x4b bytes) 000336 1787791618.810021 write 0x0366ac thru 0x03d0c8 (0x6a1d bytes) 000337 1787791618.811495 trunc from 0x03d0c9 to 0x00ea9f (0x2e629 bytes) 000339 1787791618.840206 write 0x030f81 thru 0x038b94 (0x7c14 bytes) 000341 1787791618.842671 mapread 0x012a8d thru 0x01d51d (0xaa91 bytes) 000343 1787791618.874186 read 0x0118c5 thru 0x0193d1 (0x7b0d bytes) 000345 1787791618.875934 trunc from 0x038b95 to 0x017593 (0x21601 bytes) 000346 1787791618.896174 mapread 0x00acc9 thru 0x00bf72 (0x12aa bytes) 000347 1787791618.910244 MIRROR_RESYNC 7 mirrors 000348 1787791620.160455 read 0x0167a9 thru 0x017593 (0xdeb bytes) 000351 1787791620.166985 read 0x013e27 thru 0x017593 (0x376d bytes) 000353 1787791620.178957 mapwrite 0x01f157 thru 0x025b64 (0x6a0e bytes) 000355 1787791620.337012 mapwrite 0x02e1b8 thru 0x03aad4 (0xc91d bytes) 000356 1787791620.413963 write 0x00af08 thru 0x019d88 (0xee81 bytes) 000358 1787791620.421017 read 0x023445 thru 0x029cec (0x68a8 bytes) 000359 1787791620.421787 read 0x00f8e6 thru 0x0113f6 (0x1b11 bytes) 000360 1787791620.425180 mapwrite 0x020b53 thru 0x021e20 (0x12ce bytes) 000361 1787791620.484623 trunc from 0x03aad5 to 0x007ac1 (0x33013 bytes) 000362 1787791620.497376 write 0x01d17a thru 0x025b6e (0x89f5 bytes) 000363 1787791620.498824 read 0x01e993 thru 0x025b6e (0x71dc bytes) 000365 1787791620.500643 write 0x02f741 thru 0x031bc3 (0x2483 bytes) 000366 1787791620.502469 read 0x02c12e thru 0x02f8e2 (0x37b5 bytes) 000367 1787791620.513475 write 0x0372b0 thru 0x038a4b (0x179c bytes) 000368 1787791620.514453 mapwrite 0x03384f thru 0x0352bb (0x1a6d bytes) 000371 1787791620.573862 mapwrite 0x039dbc thru 0x03ed3c (0x4f81 bytes) 000373 1787791620.672521 mapread 0x03347c thru 0x03851f (0x50a4 bytes) 000374 1787791620.675656 mapread 0x00f385 thru 0x013d72 (0x49ee bytes) 000375 1787791620.678823 mapread 0x01f53f thru 0x029870 (0xa332 bytes) 000376 1787791620.680199 write 0x00e4ac thru 0x01d14d (0xeca2 bytes) 000378 1787791620.686850 mapread 0x02945b thru 0x02a723 (0x12c9 bytes) 000382 1787791620.698087 MIRROR_RESYNC 7 mirrors 000383 1787791621.756974 write 0x00180f thru 0x0042e8 (0x2ada bytes) 000388 1787791621.838400 read 0x029326 thru 0x02b520 (0x21fb bytes) 000389 1787791621.845200 write 0x0138e9 thru 0x01ee15 (0xb52d bytes) 000390 1787791621.854389 read 0x0220d5 thru 0x02fae2 (0xda0e bytes) 000391 1787791621.875537 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed3d): Mirror 5: 0xfd683dfc (stale) Mirror 6: 0xfd683dfc (stale) Mirror 8: 0xfd683dfc (stale) Mirror 9: 0xfd683dfc (stale) Mirror 10: 0xfd683dfc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed3d): Mirror 5: 0xfd683dfc (stale) Mirror 6: 0xfd683dfc (stale) Mirror 8: 0xfd683dfc (stale) Mirror 9: 0xfd683dfc (stale) Mirror 10: 0xfd683dfc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed3d: skipped 000394 1787791622.227937 mapwrite 0x00f7e9 thru 0x0156cf (0x5ee7 bytes) 000397 1787791622.312444 mapwrite 0x03a685 thru 0x03d115 (0x2a91 bytes) 000400 1787791622.375714 trunc from 0x03ed3d to 0x02d5a8 (0x11794 bytes) 000401 1787791622.394088 mapwrite 0x0216a2 thru 0x02f6e2 (0xe041 bytes) 000403 1787791622.483670 trunc from 0x02f6e3 to 0x006958 (0x28d8a bytes) 000404 1787791622.506381 read 0x002403 thru 0x006958 (0x4556 bytes) 000405 1787791622.521537 mapwrite 0x037de2 thru 0x03ffff (0x821e bytes) 000406 1787791622.595670 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x11dbee1a (stale) Mirror 6: 0x11dbee1a (stale) Mirror 8: 0x11dbee1a (stale) Mirror 9: 0x11dbee1a (stale) Mirror 10: 0x11dbee1a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbf3bf54c Mirror 11: 0xbf3bf54c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000409 1787791623.473732 trunc from 0x040000 to 0x024469 (0x1bb96 bytes) 000410 1787791623.595358 read 0x01c05b thru 0x020d6c (0x4d12 bytes) 000411 1787791623.607642 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2446a): Mirror 6: 0xe9e097ae (stale) Mirror 8: 0xe9e097ae (stale) Mirror 9: 0xe9e097ae (stale) Mirror 10: 0xe9e097ae (stale) Mirror 11: 0x1bda7060 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2446a): Mirror 6: 0xe9e097ae (stale) Mirror 8: 0xe9e097ae (stale) Mirror 9: 0xe9e097ae (stale) Mirror 10: 0xe9e097ae (stale) Mirror 11: 0x1bda7060 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2446a: skipped 000412 1787791623.929950 mapwrite 0x02c36b thru 0x0338e4 (0x757a bytes) 000414 1787791624.010487 mapwrite 0x01253a thru 0x017827 (0x52ee bytes) 000415 1787791624.103330 mapread 0x003670 thru 0x004627 (0xfb8 bytes) 000416 1787791624.116612 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x338e5): Mirror 8: 0x82065647 (stale) Mirror 9: 0x82065647 (stale) Mirror 10: 0x82065647 (stale) Mirror 11: 0x68622de1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x338e5): Mirror 8: 0x82065647 (stale) Mirror 9: 0x82065647 (stale) Mirror 10: 0x82065647 (stale) Mirror 11: 0x68622de1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x338e5: skipped 000417 1787791624.452737 write 0x02a513 thru 0x03974d (0xf23b bytes) 000418 1787791624.461467 mapread 0x0227d8 thru 0x02b727 (0x8f50 bytes) 000422 1787791624.496668 read 0x028645 thru 0x02c7f8 (0x41b4 bytes) 000424 1787791624.500376 mapread 0x031958 thru 0x03336d (0x1a16 bytes) 000425 1787791624.501249 mapread 0x010f25 thru 0x012abd (0x1b99 bytes) 000426 1787791624.510622 write 0x0325e1 thru 0x03ffff (0xda1f bytes) 000427 1787791624.511988 write 0x0200a3 thru 0x02b8fa (0xb858 bytes) 000428 1787791624.513590 mapwrite 0x02bb6f thru 0x02c7c7 (0xc59 bytes) 000429 1787791624.581305 mapread 0x0291c7 thru 0x02db88 (0x49c2 bytes) 000430 1787791624.582231 mapread 0x03c56a thru 0x03ffff (0x3a96 bytes) 000431 1787791624.583352 trunc from 0x040000 to 0x0378cd (0x8732 bytes) 000433 1787791624.598329 read 0x00ba93 thru 0x012476 (0x69e4 bytes) 000435 1787791624.602826 MIRROR_RESYNC 5 mirrors 000436 1787791626.067062 mapwrite 0x039208 thru 0x03ffff (0x6df8 bytes) 000438 1787791626.177208 trunc from 0x040000 to 0x002581 (0x3da7e bytes) 000440 1787791626.207623 write 0x031b94 thru 0x037a6c (0x5ed9 bytes) 000442 1787791626.213340 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37a6d): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a6d): Mirror 8: 0x8cfc5297 (stale) Mirror 9: 0x8cfc5297 (stale) Mirror 10: 0x8cfc5297 (stale) Mirror 11: 0x8cfc5297 (stale) CRC-32 checksum value for chunk [0, 0x37a6d): Mirror 1: 0x2b7b527e Mirror 12: 0x2b7b527e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a6d: skipped 000446 1787791627.018876 write 0x0381b5 thru 0x03a223 (0x206f bytes) 000451 1787791627.077434 mapread 0x020cab thru 0x02b670 (0xa9c6 bytes) 000452 1787791627.123401 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a224): Mirror 9: 0xbc4e14a0 (stale) Mirror 10: 0xbc4e14a0 (stale) Mirror 11: 0xbc4e14a0 (stale) Mirror 12: 0xaad2678a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a224): Mirror 9: 0xbc4e14a0 (stale) Mirror 10: 0xbc4e14a0 (stale) Mirror 11: 0xbc4e14a0 (stale) Mirror 12: 0xaad2678a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a224: skipped 000454 1787791627.489526 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a224): Mirror 9: 0xbc4e14a0 (stale) Mirror 10: 0xbc4e14a0 (stale) Mirror 11: 0xbc4e14a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a224): Mirror 9: 0xbc4e14a0 (stale) Mirror 10: 0xbc4e14a0 (stale) Mirror 11: 0xbc4e14a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a224: skipped 000455 1787791627.753097 trunc from 0x03a224 to 0x014ea9 (0x2537a bytes) 000456 1787791627.774080 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x14eaa): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14eaa): Mirror 9: 0x4d9734f3 (stale) Mirror 10: 0x4d9734f3 (stale) Mirror 11: 0x4d9734f3 (stale) CRC-32 checksum value for chunk [0, 0x14eaa): Mirror 1: 0xe75551c3 Mirror 12: 0xe75551c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14eaa: skipped 000463 1787791628.651936 write 0x033d13 thru 0x03ffff (0xc2ed bytes) 000464 1787791628.715482 write 0x030f10 thru 0x03c989 (0xba7a bytes) 000470 1787791628.730159 trunc from 0x040000 to 0x03ea83 (0x157c bytes) 000472 1787791628.751471 write 0x01a4e6 thru 0x027515 (0xd030 bytes) 000473 1787791628.763534 trunc from 0x03ea84 to 0x031f5b (0xcb28 bytes) 000476 1787791628.784954 mapread 0x02a222 thru 0x031f5b (0x7d3a bytes) 000477 1787791628.806640 mapread 0x00e11d thru 0x010208 (0x20ec bytes) 000478 1787791628.815975 read 0x0301ec thru 0x031f5b (0x1d70 bytes) 000481 1787791628.817582 read 0x02dabe thru 0x031f5b (0x449e bytes) 000484 1787791628.820797 mapwrite 0x00f2fc thru 0x01b761 (0xc466 bytes) 000485 1787791628.886793 trunc from 0x031f5c to 0x030504 (0x1a57 bytes) 000486 1787791628.901934 mapread 0x0039da thru 0x0121ea (0xe811 bytes) 000487 1787791628.904063 mapwrite 0x00dae5 thru 0x018e9c (0xb3b8 bytes) 000489 1787791628.951371 write 0x00c610 thru 0x01b5ed (0xefde bytes) 000490 1787791628.954506 write 0x012a12 thru 0x021aa4 (0xf093 bytes) 000492 1787791628.957568 write 0x01d23f thru 0x01f1d4 (0x1f96 bytes) 000493 1787791628.958854 mapwrite 0x03802f thru 0x03d2b5 (0x5287 bytes) 000496 1787791629.023694 write 0x0334e4 thru 0x035621 (0x213e bytes) 000498 1787791629.034498 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d2b6): Mirror 10: 0xe1f710c5 (stale) Mirror 11: 0xe1f710c5 (stale) Mirror 12: 0x6613e692 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d2b6): Mirror 10: 0xe1f710c5 (stale) Mirror 11: 0xe1f710c5 (stale) Mirror 12: 0x6613e692 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d2b6: skipped 000499 1787791629.340525 write 0x02198a thru 0x024e31 (0x34a8 bytes) 000501 1787791629.354452 read 0x01266a thru 0x01b392 (0x8d29 bytes) 000502 1787791629.361828 trunc from 0x03d2b6 to 0x0256ef (0x17bc6 bytes) 000503 1787791629.373589 read 0x0135f3 thru 0x0185f3 (0x5001 bytes) 000505 1787791629.375366 read 0x022900 thru 0x0256ef (0x2df0 bytes) 000506 1787791629.392674 trunc from 0x0256f0 to 0x01f8b2 (0x5e3d bytes) 000507 1787791629.409925 mapwrite 0x03d5a1 thru 0x03ffff (0x2a5f bytes) 000509 1787791629.490817 read 0x0026ad thru 0x002804 (0x158 bytes) 000510 1787791629.491517 write 0x03abaa thru 0x03ffff (0x5456 bytes) 000511 1787791629.493441 write 0x030032 thru 0x032b1e (0x2aed bytes) 000512 1787791629.498449 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x976856c9 (stale) Mirror 11: 0x976856c9 (stale) Mirror 12: 0x3e221862 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8de1a4b4 Mirror 13: 0x8de1a4b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000514 1787791630.467438 trunc from 0x040000 to 0x0190b7 (0x26f48 bytes) 000515 1787791630.543389 write 0x02a7ff thru 0x035182 (0xa984 bytes) 000517 1787791630.545840 write 0x018508 thru 0x025f40 (0xda39 bytes) 000518 1787791630.555882 mapwrite 0x012d72 thru 0x022a49 (0xfcd8 bytes) 000522 1787791630.627933 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35183): Mirror 11: 0xbf5b8a0 (stale) Mirror 12: 0x8d30fba4 (stale) Mirror 13: 0xf406003a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35183): Mirror 11: 0xbf5b8a0 (stale) Mirror 12: 0x8d30fba4 (stale) Mirror 13: 0xf406003a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35183: skipped 000524 1787791630.933280 read 0x01ade4 thru 0x023c7b (0x8e98 bytes) 000525 1787791630.941518 mapwrite 0x02f26f thru 0x030ecf (0x1c61 bytes) 000527 1787791630.984581 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35183): Mirror 12: 0x8d30fba4 (stale) Mirror 13: 0xf406003a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35183): Mirror 12: 0x8d30fba4 (stale) Mirror 13: 0xf406003a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35183: skipped 000528 1787791631.254526 mapwrite 0x0274d4 thru 0x034319 (0xce46 bytes) 000530 1787791631.324187 trunc from 0x035183 to 0x024fa2 (0x101e0 bytes) 000532 1787791631.344849 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24fa3): Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24fa3): Mirror 12: 0xac52651 (stale) Mirror 13: 0x2605e9e8 (stale) CRC-32 checksum value for chunk [0, 0x24fa3): Mirror 1: 0xf17af72f Mirror 14: 0xf17af72f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24fa3: skipped 000534 1787791632.075764 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24fa3): Mirror 12: 0xac52651 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24fa3): Mirror 12: 0xac52651 (stale) CRC-32 checksum value for chunk [0, 0x24fa3): Mirror 1: 0xf17af72f Mirror 14: 0xf17af72f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24fa3: skipped 000535 1787791632.456494 trunc from 0x024fa3 to 0x01ac51 (0xa351 bytes) 000537 1787791632.530442 read 0x002062 thru 0x005669 (0x3608 bytes) 000538 1787791632.541409 MIRROR_RESYNC 3 mirrors 000541 1787791633.151731 read 0x016811 thru 0x01ac51 (0x4441 bytes) 000542 1787791633.162378 write 0x02604b thru 0x02ff6e (0x9f24 bytes) 000544 1787791633.220742 write 0x0382d5 thru 0x03ffff (0x7d2b bytes) 000545 1787791633.221904 read 0x006193 thru 0x008e63 (0x2cd1 bytes) 000547 1787791633.253731 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x51c40538 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x51c40538 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000548 1787791633.508878 trunc from 0x040000 to 0x0361de (0x9e21 bytes) 000549 1787791633.546348 write 0x0276b9 thru 0x0371c2 (0xfb0a bytes) 000550 1787791633.552342 mapread 0x011f87 thru 0x0157ae (0x3828 bytes) 000553 1787791633.581474 mapread 0x036132 thru 0x0371c2 (0x1091 bytes) 000555 1787791633.583636 read 0x021f47 thru 0x023ca4 (0x1d5e bytes) 000556 1787791633.589249 mapread 0x015ac5 thru 0x02257a (0xcab6 bytes) 000559 1787791633.597546 write 0x03754e thru 0x037be0 (0x693 bytes) 000560 1787791633.598511 read 0x00d1de thru 0x0194e6 (0xc309 bytes) 000563 1787791633.601930 mapwrite 0x026519 thru 0x03078f (0xa277 bytes) 000565 1787791633.691299 trunc from 0x037be1 to 0x032890 (0x5350 bytes) 000568 1787791633.721556 write 0x004571 thru 0x012619 (0xe0a9 bytes) 000569 1787791633.730816 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32891): Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32891): Mirror 14: 0x7417b04e (stale) CRC-32 checksum value for chunk [0, 0x32891): Mirror 1: 0xa25c9504 Mirror 15: 0xa25c9504 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32891: skipped 000570 1787791634.386580 write 0x00683a thru 0x012d87 (0xc54e bytes) 000571 1787791634.506283 mapwrite 0x031a36 thru 0x035acc (0x4097 bytes) 000572 1787791634.617544 write 0x022e06 thru 0x02828b (0x5486 bytes) 000573 1787791634.630588 mapread 0x00f687 thru 0x01244d (0x2dc7 bytes) 000575 1787791634.632742 trunc from 0x035acd to 0x0368cb (0xdff bytes) 000576 1787791634.652531 trunc from 0x0368cc to 0x00f741 (0x2718a bytes) 000577 1787791634.673576 mapwrite 0x006408 thru 0x016187 (0xfd80 bytes) 000578 1787791634.790426 mapwrite 0x00fcfe thru 0x0119f6 (0x1cf9 bytes) 000579 1787791634.829359 mapwrite 0x03ebfb thru 0x03ffff (0x1405 bytes) 000581 1787791634.895815 trunc from 0x040000 to 0x0043f7 (0x3bc08 bytes) 000582 1787791634.913419 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x43f8): Mirror 15: 0x18f5a832 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x43f8): Mirror 15: 0x18f5a832 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x43f8: skipped 000583 1787791635.172317 mapread 0x000d3c thru 0x0043f7 (0x36bc bytes) 000584 1787791635.209636 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000587 1787791635.340468 trunc from 0x0043f8 to 0x004974 (0x57d bytes) 000588 1787791635.357627 mapread 0x00146c thru 0x0027f4 (0x1389 bytes) 000592 1787791635.375317 write 0x01d222 thru 0x026ae8 (0x98c7 bytes) 000593 1787791635.376989 write 0x0180c0 thru 0x026f99 (0xeeda bytes) 000594 1787791635.381585 write 0x01cb1d thru 0x025fe6 (0x94ca bytes) 000595 1787791635.383239 mapwrite 0x00de97 thru 0x01b395 (0xd4ff bytes) 000596 1787791635.446788 MIRROR_EXTEND to 2 mirrors 000597 1787791635.909336 trunc from 0x026f9a to 0x03eb31 (0x17b98 bytes) 000598 1787791636.054000 mapwrite 0x001cdf thru 0x01197f (0xfca1 bytes) 000599 1787791636.141503 mapread 0x019b6a thru 0x028043 (0xe4da bytes) 000602 1787791636.163370 read 0x03ca86 thru 0x03eb31 (0x20ac bytes) 000603 1787791636.170367 trunc from 0x03eb32 to 0x00c964 (0x321cd bytes) 000610 1787791636.189374 trunc from 0x00c965 to 0x00db83 (0x121f bytes) 000615 1787791636.212569 read 0x0035c3 thru 0x00db83 (0xa5c1 bytes) 000620 1787791636.223792 read 0x002a98 thru 0x00db83 (0xb0ec bytes) 000622 1787791636.225055 mapread 0x00c573 thru 0x00db83 (0x1611 bytes) 000624 1787791636.226303 read 0x000e3e thru 0x00286a (0x1a2d bytes) 000625 1787791636.227343 read 0x004fc7 thru 0x00db83 (0x8bbd bytes) 000626 1787791636.231433 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xdb84): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdb84): Mirror 2: 0xf3ab3762 (stale) CRC-32 checksum value for chunk [0, 0xdb84): Mirror 1: 0xb894347 Mirror 3: 0xb894347 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdb84: skipped 000631 1787791636.903472 trunc from 0x00db84 to 0x01d43b (0xf8b8 bytes) 000632 1787791637.016663 trunc from 0x01d43c to 0x036c2a (0x197ef bytes) 000633 1787791637.040583 mapread 0x02873a thru 0x03686d (0xe134 bytes) 000634 1787791637.068589 mapread 0x0195da thru 0x01b608 (0x202f bytes) 000635 1787791637.079399 write 0x039995 thru 0x03df2b (0x4597 bytes) 000637 1787791637.081818 write 0x036e9d thru 0x03d409 (0x656d bytes) 000639 1787791637.083528 trunc from 0x03df2c to 0x00ce26 (0x31105 bytes) 000642 1787791637.099972 write 0x02d43c thru 0x031367 (0x3f2c bytes) truncating to largest ever: 0x3f225 000643 1787791637.101616 trunc from 0x031368 to 0x03f224 (0xdebd bytes) 000646 1787791637.125080 mapwrite 0x01db8a thru 0x022aeb (0x4f62 bytes) 000649 1787791637.190721 mapwrite 0x03fdd8 thru 0x03ffff (0x228 bytes) 000650 1787791637.239453 read 0x0082d3 thru 0x00c0d3 (0x3e01 bytes) 000651 1787791637.250314 mapread 0x008480 thru 0x017b82 (0xf703 bytes) 000652 1787791637.267683 write 0x000345 thru 0x0058f9 (0x55b5 bytes) 000653 1787791637.275520 read 0x03c886 thru 0x03e15a (0x18d5 bytes) 000654 1787791637.283463 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbf6b6c54 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbf6b6c54 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000657 1787791637.555933 write 0x01cf60 thru 0x01e627 (0x16c8 bytes) 000658 1787791637.569623 read 0x023865 thru 0x027529 (0x3cc5 bytes) 000659 1787791637.582439 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000660 1787791637.731472 trunc from 0x040000 to 0x013ef8 (0x2c107 bytes) 000661 1787791637.757455 trunc from 0x013ef9 to 0x01565f (0x1767 bytes) 000664 1787791637.779669 read 0x0075c0 thru 0x0148be (0xd2ff bytes) 000670 1787791637.797135 read 0x00037b thru 0x00115b (0xde1 bytes) 000671 1787791637.811305 mapread 0x0128de thru 0x01565f (0x2d82 bytes) 000672 1787791637.812181 trunc from 0x015660 to 0x0081b7 (0xd4a8 bytes) 000673 1787791637.833129 write 0x02e91f thru 0x03a9bb (0xc09d bytes) 000675 1787791637.835862 mapread 0x0046c2 thru 0x00e2b2 (0x9bf1 bytes) 000676 1787791637.868131 read 0x006bef thru 0x007f8a (0x139c bytes) 000677 1787791637.869426 trunc from 0x03a9bc to 0x02b7c9 (0xf1f2 bytes) 000678 1787791637.884113 trunc from 0x02b7ca to 0x02ad17 (0xab2 bytes) 000680 1787791637.908995 mapwrite 0x007442 thru 0x011854 (0xa413 bytes) 000681 1787791637.973581 mapread 0x001867 thru 0x0081ec (0x6986 bytes) 000683 1787791637.984431 MIRROR_EXTEND to 2 mirrors 000685 1787791638.742512 mapread 0x00d0b3 thru 0x01a68c (0xd5da bytes) 000686 1787791638.778309 read 0x020632 thru 0x0212fd (0xccc bytes) 000688 1787791638.795607 MIRROR_RESYNC 2 mirrors 000689 1787791639.089626 trunc from 0x02ad18 to 0x033b8b (0x8e74 bytes) 000690 1787791639.167937 trunc from 0x033b8c to 0x0143e4 (0x1f7a7 bytes) 000691 1787791639.186097 write 0x0116fc thru 0x01721e (0x5b23 bytes) 000692 1787791639.194092 write 0x020dae thru 0x02e5ce (0xd821 bytes) 000693 1787791639.199357 MIRROR_RESYNC 2 mirrors 000694 1787791639.848025 trunc from 0x02e5cf to 0x017e48 (0x16786 bytes) 000695 1787791639.957898 read 0x00d4ea thru 0x00e0a2 (0xbb9 bytes) 000697 1787791639.975467 trunc from 0x017e49 to 0x027936 (0xfaee bytes) 000698 1787791640.000227 write 0x038d88 thru 0x03ffff (0x7278 bytes) 000699 1787791640.001449 read 0x017008 thru 0x022e83 (0xbe7c bytes) 000700 1787791640.019722 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000701 1787791640.185559 mapread 0x003ab7 thru 0x0046c1 (0xc0b bytes) 000703 1787791640.198012 mapwrite 0x01a771 thru 0x01ea27 (0x42b7 bytes) 000704 1787791640.255444 MIRROR_EXTEND to 2 mirrors 000705 1787791640.939873 MIRROR_RESYNC 2 mirrors 000707 1787791641.212656 write 0x003771 thru 0x0116da (0xdf6a bytes) 000708 1787791641.292455 MIRROR_RESYNC 2 mirrors 000709 1787791641.945138 read 0x03520d thru 0x038ddd (0x3bd1 bytes) 000710 1787791641.950440 trunc from 0x040000 to 0x0121a1 (0x2de5e bytes) 000713 1787791642.071807 mapread 0x00d699 thru 0x0121a1 (0x4b09 bytes) 000714 1787791642.107580 MIRROR_RESYNC 2 mirrors 000716 1787791642.697056 mapread 0x002b2b thru 0x0056d1 (0x2ba7 bytes) 000717 1787791642.714409 mapread 0x009a46 thru 0x0121a1 (0x875c bytes) 000718 1787791642.721150 trunc from 0x0121a2 to 0x0364cc (0x2432b bytes) 000719 1787791642.779812 write 0x01c5b9 thru 0x023282 (0x6cca bytes) 000720 1787791642.789417 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x364cd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364cd): Mirror 2: 0x94ccf589 (stale) CRC-32 checksum value for chunk [0, 0x364cd): Mirror 1: 0x29ac8263 Mirror 3: 0x29ac8263 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364cd: skipped 000721 1787791643.650607 mapwrite 0x013827 thru 0x018b14 (0x52ee bytes) 000722 1787791643.822690 mapwrite 0x01b2a2 thru 0x0202fa (0x5059 bytes) 000731 1787791643.882775 read 0x00c362 thru 0x018826 (0xc4c5 bytes) 000733 1787791643.895266 read 0x014b90 thru 0x01f264 (0xa6d5 bytes) 000736 1787791643.899023 trunc from 0x0364cd to 0x00c1a3 (0x2a329 bytes) 000737 1787791643.931667 trunc from 0x00c1a4 to 0x010fc5 (0x4e22 bytes) 000739 1787791643.944104 mapread 0x00b2cc thru 0x010fc5 (0x5cfa bytes) 000742 1787791643.953160 mapread 0x008ed2 thru 0x010548 (0x7677 bytes) 000743 1787791643.957524 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x10fc6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10fc6): Mirror 2: 0xa245af16 (stale) Mirror 3: 0xa245af16 (stale) CRC-32 checksum value for chunk [0, 0x10fc6): Mirror 1: 0x57c49771 Mirror 4: 0x57c49771 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10fc6: skipped 000744 1787791644.824847 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10fc6): Mirror 3: 0xa245af16 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10fc6): Mirror 3: 0xa245af16 (stale) CRC-32 checksum value for chunk [0, 0x10fc6): Mirror 1: 0x57c49771 Mirror 4: 0x57c49771 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10fc6: skipped 000750 1787791645.171892 mapread 0x00709f thru 0x00af84 (0x3ee6 bytes) 000752 1787791645.189226 trunc from 0x010fc6 to 0x011eaf (0xeea bytes) 000754 1787791645.296956 trunc from 0x011eb0 to 0x0035fd (0xe8b2 bytes) 000755 1787791645.317024 read 0x000389 thru 0x0008ce (0x546 bytes) 000756 1787791645.323300 trunc from 0x0035fe to 0x018520 (0x14f23 bytes) 000757 1787791645.342505 mapread 0x008eb8 thru 0x00ae10 (0x1f59 bytes) 000758 1787791645.359311 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18521): Mirror 3: 0xe511a36d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18521): Mirror 3: 0xe511a36d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18521: skipped 000759 1787791645.575303 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x18521): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18521): Mirror 3: 0xe511a36d (stale) CRC-32 checksum value for chunk [0, 0x18521): Mirror 1: 0x593a2a8e Mirror 4: 0x593a2a8e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18521: skipped 000760 1787791646.446472 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18521): Mirror 3: 0xe511a36d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18521): Mirror 3: 0xe511a36d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18521: skipped 000762 1787791646.692678 mapread 0x0012ae thru 0x00a1b6 (0x8f09 bytes) 000765 1787791646.709674 write 0x00ec02 thru 0x00ee99 (0x298 bytes) 000766 1787791646.727737 MIRROR_RESYNC 2 mirrors 000769 1787791647.352258 read 0x00378d thru 0x005361 (0x1bd5 bytes) 000770 1787791647.362524 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000774 1787791647.503419 mapread 0x00c567 thru 0x018048 (0xbae2 bytes) 000776 1787791647.538702 mapread 0x012d2b thru 0x0149f0 (0x1cc6 bytes) 000779 1787791647.540783 mapread 0x0008f4 thru 0x00fa1f (0xf12c bytes) 000780 1787791647.556506 write 0x02535d thru 0x02d465 (0x8109 bytes) 000782 1787791647.575429 trunc from 0x02d466 to 0x0115f2 (0x1be73 bytes) 000783 1787791647.592582 write 0x03318d thru 0x038eb7 (0x5d2b bytes) 000784 1787791647.593824 trunc from 0x038eb8 to 0x036289 (0x2c2e bytes) 000786 1787791647.616748 trunc from 0x03628a to 0x01d6c2 (0x18bc7 bytes) 000787 1787791647.643469 mapread 0x01162c thru 0x01d6c2 (0xc097 bytes) 000788 1787791647.665980 read 0x003a31 thru 0x00e907 (0xaed7 bytes) 000791 1787791647.681148 write 0x032434 thru 0x03d46d (0xb03a bytes) 000795 1787791647.698175 MIRROR_EXTEND to 2 mirrors 000796 1787791648.533693 read 0x01de47 thru 0x029f46 (0xc100 bytes) 000797 1787791648.544954 trunc from 0x03d46e to 0x022327 (0x1b146 bytes) 000799 1787791648.669506 mapwrite 0x00e3e1 thru 0x017144 (0x8d64 bytes) 000800 1787791648.749537 mapwrite 0x00daa2 thru 0x01788c (0x9deb bytes) 000801 1787791648.797980 write 0x02b6c3 thru 0x030cea (0x5628 bytes) 000803 1787791648.800062 mapwrite 0x03e0f5 thru 0x03ffff (0x1f0b bytes) 000804 1787791648.885455 read 0x016bc1 thru 0x01c424 (0x5864 bytes) 000805 1787791648.910310 write 0x01ab53 thru 0x01b7df (0xc8d bytes) 000806 1787791648.912834 write 0x03152a thru 0x03ae7e (0x9955 bytes) 000810 1787791648.918392 MIRROR_RESYNC 2 mirrors 000813 1787791649.565314 write 0x0359c5 thru 0x03bd66 (0x63a2 bytes) 000815 1787791649.634414 write 0x023f32 thru 0x0265d2 (0x26a1 bytes) 000816 1787791649.642531 mapwrite 0x0224c8 thru 0x02909c (0x6bd5 bytes) 000818 1787791649.714958 read 0x0088da thru 0x013be0 (0xb307 bytes) 000819 1787791649.727405 read 0x01f2f0 thru 0x027062 (0x7d73 bytes) 000820 1787791649.736481 mapwrite 0x00a023 thru 0x00a8b9 (0x897 bytes) 000824 1787791649.780849 read 0x002956 thru 0x00807b (0x5726 bytes) 000825 1787791649.781877 mapwrite 0x017bb6 thru 0x022760 (0xabab bytes) 000827 1787791649.824339 read 0x02118d thru 0x02be3c (0xacb0 bytes) 000828 1787791649.829076 MIRROR_RESYNC 2 mirrors 000829 1787791650.401640 read 0x00d6c9 thru 0x015624 (0x7f5c bytes) 000832 1787791650.410071 trunc from 0x040000 to 0x00154d (0x3eab2 bytes) 000833 1787791650.494586 mapwrite 0x03104e thru 0x033662 (0x2615 bytes) 000835 1787791650.580290 mapread 0x019c88 thru 0x01ca21 (0x2d9a bytes) 000836 1787791650.596183 mapread 0x016e30 thru 0x01e114 (0x72e5 bytes) 000837 1787791650.598793 mapwrite 0x02a9cc thru 0x02d4d3 (0x2b08 bytes) 000839 1787791650.648412 write 0x02d95f thru 0x036192 (0x8834 bytes) 000840 1787791650.649709 write 0x03aaa8 thru 0x03e04f (0x35a8 bytes) 000842 1787791650.651473 trunc from 0x03e050 to 0x001f7f (0x3c0d0 bytes) 000845 1787791650.669560 read 0x000543 thru 0x0006a2 (0x160 bytes) 000846 1787791650.674744 mapwrite 0x002041 thru 0x003df4 (0x1db4 bytes) 000848 1787791650.726123 write 0x00d821 thru 0x010809 (0x2fe9 bytes) 000849 1787791650.727172 trunc from 0x01080a to 0x00ec15 (0x1bf4 bytes) 000850 1787791650.737766 trunc from 0x00ec16 to 0x006dfa (0x7e1b bytes) 000851 1787791650.751195 mapwrite 0x02e11b thru 0x03ae40 (0xcd26 bytes) 000856 1787791650.833389 MIRROR_RESYNC 2 mirrors 000857 1787791651.523083 MIRROR_RESYNC 2 mirrors 000859 1787791651.831324 trunc from 0x03ae41 to 0x02478d (0x166b3 bytes) 000860 1787791651.896846 mapwrite 0x03e6d6 thru 0x03ffff (0x192a bytes) 000861 1787791651.966090 trunc from 0x040000 to 0x02cd17 (0x132e8 bytes) 000862 1787791651.986278 trunc from 0x02cd18 to 0x02a3b8 (0x295f bytes) 000863 1787791652.006283 trunc from 0x02a3b9 to 0x01d585 (0xce33 bytes) 000865 1787791652.028789 write 0x03d298 thru 0x03ffff (0x2d68 bytes) 000866 1787791652.029922 mapread 0x0252fb thru 0x030ef0 (0xbbf6 bytes) 000867 1787791652.042016 write 0x03ccf7 thru 0x03f928 (0x2c32 bytes) 000868 1787791652.046678 write 0x025358 thru 0x02eb56 (0x97ff bytes) 000871 1787791652.048980 mapwrite 0x0117b2 thru 0x020f7f (0xf7ce bytes) 000872 1787791652.102221 read 0x02c7a9 thru 0x03845c (0xbcb4 bytes) 000873 1787791652.108379 mapread 0x0078a9 thru 0x00d838 (0x5f90 bytes) 000874 1787791652.123024 MIRROR_RESYNC 2 mirrors 000877 1787791652.768937 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000879 1787791652.889758 mapwrite 0x02a09f thru 0x02c25f (0x21c1 bytes) 000882 1787791652.986111 read 0x0394a3 thru 0x03d9ae (0x450c bytes) 000886 1787791652.999625 mapwrite 0x02e96a thru 0x03e7ab (0xfe42 bytes) 000887 1787791653.069013 read 0x024861 thru 0x025f71 (0x1711 bytes) 000889 1787791653.082661 write 0x020121 thru 0x027148 (0x7028 bytes) 000890 1787791653.085381 mapwrite 0x02ad20 thru 0x038459 (0xd73a bytes) 000893 1787791653.148714 mapread 0x0211e6 thru 0x02ac9d (0x9ab8 bytes) 000894 1787791653.150464 read 0x01361c thru 0x01ba48 (0x842d bytes) 000895 1787791653.151814 trunc from 0x040000 to 0x01ee0b (0x211f4 bytes) 000899 1787791653.208391 read 0x014501 thru 0x01ee0b (0xa90b bytes) 000901 1787791653.210772 write 0x03adad thru 0x03b123 (0x377 bytes) 000902 1787791653.212309 write 0x02e325 thru 0x033fd6 (0x5cb2 bytes) 000905 1787791653.221268 write 0x02c2ae thru 0x02e22c (0x1f7f bytes) 000906 1787791653.222309 write 0x02c660 thru 0x033ae0 (0x7481 bytes) 000907 1787791653.224188 write 0x023fbb thru 0x02877b (0x47c1 bytes) 000910 1787791653.227858 mapread 0x02dea1 thru 0x03a587 (0xc6e7 bytes) 000912 1787791653.229602 mapread 0x00f914 thru 0x01c9fd (0xd0ea bytes) 000914 1787791653.231871 trunc from 0x03b124 to 0x00475c (0x369c7 bytes) 000915 1787791653.268733 mapwrite 0x023ecb thru 0x026661 (0x2797 bytes) 000918 1787791653.352246 write 0x03450f thru 0x03d38d (0x8e7f bytes) 000923 1787791653.357782 mapwrite 0x024510 thru 0x024e9d (0x98e bytes) 000924 1787791653.398274 mapwrite 0x0396d4 thru 0x03f2c5 (0x5bf2 bytes) 000926 1787791653.478298 trunc from 0x03f2c6 to 0x000c9d (0x3e628 bytes) 000927 1787791653.492617 write 0x0210ab thru 0x0243d4 (0x332a bytes) 000928 1787791653.494684 trunc from 0x0243d5 to 0x00adc9 (0x1960b bytes) 000932 1787791653.514369 read 0x007760 thru 0x00adc9 (0x366a bytes) 000935 1787791653.526952 write 0x00969a thru 0x0118d7 (0x823e bytes) 000940 1787791653.532951 mapread 0x0066db thru 0x00c997 (0x62bd bytes) 000941 1787791653.534218 trunc from 0x0118d8 to 0x032903 (0x2102c bytes) 000942 1787791653.550437 mapwrite 0x0372b0 thru 0x038295 (0xfe6 bytes) 000943 1787791653.619708 mapwrite 0x023faf thru 0x033646 (0xf698 bytes) 000948 1787791653.678974 mapread 0x00243e thru 0x00282b (0x3ee bytes) 000949 1787791653.680732 mapwrite 0x027683 thru 0x03340c (0xbd8a bytes) 000952 1787791653.732895 mapwrite 0x015584 thru 0x01a1e7 (0x4c64 bytes) 000954 1787791653.766221 MIRROR_EXTEND to 2 mirrors 000955 1787791654.362449 trunc from 0x038296 to 0x03a73c (0x24a7 bytes) 000956 1787791654.448599 mapread 0x018de2 thru 0x0241b4 (0xb3d3 bytes) 000959 1787791654.491389 write 0x032788 thru 0x03ffff (0xd878 bytes) 000960 1787791654.501156 read 0x02c409 thru 0x030a77 (0x466f bytes) 000961 1787791654.521021 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) 000963 1787791654.690484 mapwrite 0x01cfb8 thru 0x0200ea (0x3133 bytes) 000964 1787791654.792963 read 0x01b775 thru 0x01b77c (0x8 bytes) 000966 1787791654.796741 mapwrite 0x03cbfe thru 0x03ffff (0x3402 bytes) 000967 1787791654.864851 mapread 0x034a56 thru 0x03e22c (0x97d7 bytes) 000968 1787791654.873846 trunc from 0x040000 to 0x035734 (0xa8cb bytes) 000969 1787791654.891599 write 0x031ead thru 0x03ee22 (0xcf76 bytes) 000970 1787791654.896775 read 0x0310b4 thru 0x03484d (0x379a bytes) 000971 1787791654.898176 write 0x033277 thru 0x037cce (0x4a58 bytes) 000972 1787791654.899962 mapwrite 0x023216 thru 0x02ec29 (0xba14 bytes) 000975 1787791654.984604 trunc from 0x03ee23 to 0x02bd36 (0x130ec bytes) 000976 1787791655.004211 mapread 0x00625d thru 0x00a6dc (0x4480 bytes) 000977 1787791655.005425 read 0x004ef6 thru 0x01071a (0xb825 bytes) 000978 1787791655.006055 trunc from 0x02bd37 to 0x02404a (0x7cec bytes) 000981 1787791655.030782 read 0x020af9 thru 0x02404a (0x3552 bytes) 000986 1787791655.034401 mapread 0x00440c thru 0x00550f (0x1104 bytes) 000989 1787791655.036681 read 0x004ad5 thru 0x00ade9 (0x6315 bytes) 000990 1787791655.037809 write 0x029bd9 thru 0x02eff8 (0x5420 bytes) 000991 1787791655.039492 read 0x00c617 thru 0x0172eb (0xacd5 bytes) 000993 1787791655.041127 read 0x0172c6 thru 0x0258ad (0xe5e8 bytes) 000995 1787791655.048849 write 0x0304cf thru 0x0347c0 (0x42f2 bytes) 000996 1787791655.049902 mapwrite 0x01ab1b thru 0x01b5a4 (0xa8a bytes) 000997 1787791655.111894 trunc from 0x0347c1 to 0x013947 (0x20e79 bytes) All operations completed A-OK! PASS 70a (70s) == sanity-flr test 71: check mirror extend parent fid ==== 20:47:38 (1787791658) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 20:47:40 (1787791660) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x328:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19089 s, 11.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x328:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137387 s, 7.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x329:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x328:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x329:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x328:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 20:47:47 (1787791667) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4569344 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4129344 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5067808 .. Extending file size to 6360768 .. Extending file size to 8451680 .. Extending file size to 7065888 .. Extending file size to 8554624 .. Extending file size to 8759456 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8855744 .. Extending file size to 8612416 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9158176 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9192544 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8834560 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58801 58802 58805 58806 58807 failed 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) 192.168.203.126@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (72s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:48:59 (1787791739) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3658720 .. Extending file size to 3007552 .. failed Extending file size to 6857504 .. Extending file size to 6222784 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8666208 .. Extending file size to 6867424 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8611168 .. Extending file size to 8960160 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8768992 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9213152 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9242048 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60905 60906 60907 60908 60910 done 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) 192.168.203.126@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (73s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:50:12 (1787791812) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0701316 s, 29.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229985 s, 9.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xd44b8b65 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) PASS 200c (12s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:50:25 (1787791825) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0490619 s, 42.7 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 20:50:30 (1787791830) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22d:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x345:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x345:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x346:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x341:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:50:34 (1787791834) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166187 s, 12.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x347:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:50:39 (1787791839) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.188247 s, 11.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x349:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x349:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x345:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:50:46 (1787791846) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138319 s, 15.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:51:15 (1787791875) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172782 s, 6.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:52:11 (1787791931) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186988 s, 11.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:53:02 (1787791982) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. Creating new pool test_204f oleg326-server: Pool lustre.test_204f created Adding targets to pool oleg326-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg326-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149732 s, 14.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg326-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg326-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg326-server: Pool lustre.test_204f destroyed PASS 204f (68s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:54:10 (1787792050) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:54:16 (1787792056) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:54:22 (1787792062) oleg326-server: Pool lustre.test_206 created Adding targets to pool oleg326-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg326-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg326-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg326-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg326-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:54:43 (1787792083) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058467 s, 17.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274284 s, 38.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb03b134b Mirror 3: 0xb03b134b PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:54:51 (1787792091) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:54:53 (1787792093) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:54:56 (1787792096) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.159854 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x9f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x860000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x860000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x860000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1610000 Read this (4 bytes): BBBB 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:55:30 (1787792130) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00981818 s, 834 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:55:35 (1787792135) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045687 s, 23.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x252 lmm_fid: [0x200000404:0x252:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x252 lmm_fid: [0x200000404:0x252:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x358:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834514 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 20:55:36.000000000 -0400 Modify: 2026-08-26 20:55:36.000000000 -0400 Change: 2026-08-26 20:55:37.000000000 -0400 Birth: 2026-08-26 20:55:35.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:55:40 (1787792140) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:55:42 (1787792142) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.440973 s, 92.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0393157 s, 104 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 20:55 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x254:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 20:55 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:55:48 (1787792148) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:55:53 (1787792153) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1379 sec =========== 20:56:00 (1787792160) === sanity-flr: start cleanup 20:56:01 (1787792161) === === sanity-flr: finish cleanup 20:56:07 (1787792167) ===