-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 08:26:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg318-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:26:48 (1787574408) === oleg318-client.virtnet: executing check_config_client /mnt/lustre oleg318-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg318-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e3c18a4d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e3c18a4d000.idle_timeout=debug disable quota as required oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:27:27 (1787574447) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7618] == sanity-flr test 0a: lfs mirror create with -N option == 08:27:30 (1787574450) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:27:48 (1787574468) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:27:52 (1787574472) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:27:56 (1787574476) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:28:23 (1787574503) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:28:27 (1787574507) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 08:28:31 (1787574511) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8e3c18a4d000.stats=clear osc.lustre-OST0001-osc-ffff8e3c18a4d000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:28:40 (1787574520) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:28:53 (1787574533) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:29:03 (1787574543) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:29:06 (1787574546) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:29:16 (1787574556) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 08:29:25 (1787574565) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:29:33 (1787574573) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:29:36 (1787574576) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:29:43 (1787574583) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:29:53 (1787574593) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:30:00 (1787574600) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 1.19711 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123553 s, 8.5 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:30:11 (1787574611) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.621565 s, 33.7 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:30:22 (1787574622) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:30:34 (1787574634) Stopping /mnt/lustre-ost1 (opts:) on oleg318-server oleg318-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e3c18a4d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e3c18a4d000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg318-server: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg318-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg318-server oleg318-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e3c18a4d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e3c18a4d000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg318-server: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg318-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (70s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:31:44 (1787574704) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.63999 s, 14.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg318-server oleg318-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e3c18a4d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e3c18a4d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg318-server: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg318-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:32:35 (1787574755) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg318-server oleg318-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e3c18a4d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e3c18a4d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg318-server: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg318-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg318-server oleg318-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e3c18a4d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e3c18a4d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg318-server: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg318-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (89s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:34:05 (1787574845) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.96217 s, 8.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 08:34 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.404s user 0m0.000s sys 0m0.192s Stopping /mnt/lustre-ost1 (opts:) on oleg318-server oleg318-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e3c18a4d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e3c18a4d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.094s user 0m0.000s sys 0m0.160s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg318-server: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg318-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg318-server oleg318-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e3c18a4d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e3c18a4d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.447s user 0m0.000s sys 0m0.214s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg318-server: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg318-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e3c18a4d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e3c18a4d000.ost_server_uuid in FULL state after 0 sec PASS 33b (108s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:35:52 (1787574952) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:35:56 (1787574956) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:36:01 (1787574961) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 08:36:05 (1787574965) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 08:36:13 (1787574973) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.724047 s, 17.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:36:26 (1787574986) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467578 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338062 s, 31.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:36:33 (1787574993) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:36:41 (1787575001) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00374395 s, 274 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00305053 s, 336 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000934825 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116139 s, 882 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 08:36:53 (1787575013) create test files with size 15732519 .. dc464afc1bdad4c8c3e590e8ad8b935c - cbdcc5a18966fe4ff5e6968efa5c75f8 - b706b85bcf7cd9f358fa9e268cbd6b1c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 08:37:20 (1787575040) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:37:39 (1787575059) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:37:45 (1787575065) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130904 s, 8.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314896 s, 33.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 08:37:53 (1787575073) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130798 s, 16.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.313803 s, 13.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.267949 s, 7.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 08:38:09 (1787575089) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 08:38:12 (1787575092) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:38:16 (1787575096) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530444 s, 19.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 08:38:23 (1787575103) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:38:27 (1787575107) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:38:38 (1787575118) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.397728 s, 26.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:38:50 (1787575130) llite.lustre-ffff8e3c3aab1800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.28669 s, 36.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8e3c18b29000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:39:05 (1787575145) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:39:47 (1787575187) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:40:27 (1787575227) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 08:40:31 (1787575231) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.316719 s, 9.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.14581 s, 21.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x38f20ce1 (nosync,stale) Mirror 2: 0x38f20ce1 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 23c25ed4e837ab1c912e6313cdfd3eb8 - mirror 1 checksum: 7fb9e1c7ba1a31a2d8a7056307b1e5e9 - mirror 2 checksum: 23c25ed4e837ab1c912e6313cdfd3eb8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T08:40:36-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T08:40:36-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 23c25ed4e837ab1c912e6313cdfd3eb8 - mirror 1 checksum: 7fb9e1c7ba1a31a2d8a7056307b1e5e9 - mirror 2 checksum: 7fb9e1c7ba1a31a2d8a7056307b1e5e9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T08:40:36-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T08:40:36-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:40:45 (1787575245) Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 08:40:53 (1787575253) shut down facet: mds1 facet_host: oleg318-server facet_failover_host: oleg318-server Failover mds1 to oleg318-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 08:41:16 (1787575276) targets are mounted 08:41:17 (1787575277) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551862 s, 19.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (51s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:41:36 (1787575296) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:41:41 (1787575301) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:41:45 (1787575305) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0137289 s, 1.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:41:55 (1787575315) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:41:58 (1787575318) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0181852 s, 225 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0744283 s, 55.0 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:42:08 (1787575328) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:42:34 (1787575354) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:43:06 (1787575386) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:43:31 (1787575411) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:43:56 (1787575436) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 08:44:05 (1787575445) mirror create pid 54312 mirror split pid 54314 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54312 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54314 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 08:45:14 (1787575514) Seed set to 5516 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1787575516.590936 write 0x0065ea thru 0x010aef (0xa506 bytes) 000004 1787575516.597122 mapread 0x003ce2 thru 0x00f9e2 (0xbd01 bytes) 000007 1787575516.614345 write 0x022603 thru 0x022b43 (0x541 bytes) truncating to largest ever: 0x3659 000008 1787575516.630987 trunc from 0x022b44 to 0x003658 (0x1f4eb bytes) 000009 1787575516.655655 read 0x000447 thru 0x003658 (0x3212 bytes) 000012 1787575516.674276 mapwrite 0x00f4fc thru 0x0129d5 (0x34da bytes) 000013 1787575516.790412 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x1fffa 000016 1787575517.703210 trunc from 0x0129d6 to 0x01fff9 (0xd624 bytes) 000018 1787575517.841960 read 0x00a4fc thru 0x013cfa (0x97ff bytes) 000019 1787575517.853312 mapread 0x002ab7 thru 0x012050 (0xf59a bytes) 000020 1787575517.881215 mapwrite 0x02c893 thru 0x03921d (0xc98b bytes) 000023 1787575517.993750 mapread 0x00f5c0 thru 0x01dedc (0xe91d bytes) 000026 1787575518.024987 mapread 0x03107d thru 0x03921d (0x81a1 bytes) 000028 1787575518.026348 trunc from 0x03921e to 0x01a29f (0x1ef7e bytes) 000029 1787575518.043993 mapwrite 0x029977 thru 0x036710 (0xcd9a bytes) truncating to largest ever: 0x27234 000032 1787575518.170061 trunc from 0x036711 to 0x027233 (0xf4dd bytes) 000034 1787575518.192072 read 0x01624a thru 0x01c21c (0x5fd3 bytes) 000037 1787575518.210812 mapwrite 0x02bd07 thru 0x030a1f (0x4d19 bytes) 000038 1787575518.308341 mapread 0x00642c thru 0x0108e5 (0xa4ba bytes) 000039 1787575518.309989 read 0x0141f6 thru 0x01e336 (0xa141 bytes) 000043 1787575518.315229 write 0x00ccd9 thru 0x013a55 (0x6d7d bytes) 000044 1787575518.316897 read 0x0048eb thru 0x00d9d3 (0x90e9 bytes) 000048 1787575518.319527 mapread 0x0048d9 thru 0x00f8ec (0xb014 bytes) 000049 1787575518.332244 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000051 1787575518.625018 read 0x023687 thru 0x02b4a2 (0x7e1c bytes) 000053 1787575518.653598 MIRROR_EXTEND to 2 mirrors 000055 1787575519.583520 trunc from 0x030a20 to 0x00bd30 (0x24cef bytes) 000058 1787575519.674183 trunc from 0x00bd31 to 0x013b44 (0x7e14 bytes) 000059 1787575519.695961 mapread 0x001ff4 thru 0x004e74 (0x2e81 bytes) 000060 1787575519.715783 write 0x0161ea thru 0x01f55d (0x9374 bytes) 000062 1787575519.718923 write 0x01824c thru 0x01a4a7 (0x225c bytes) 000063 1787575519.724227 MIRROR_RESYNC 2 mirrors 000064 1787575520.573281 write 0x0377c1 thru 0x03b9ea (0x422a bytes) 000065 1787575520.689233 trunc from 0x03b9eb to 0x024051 (0x17999 bytes) 000067 1787575520.737798 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24052): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24052): Mirror 2: 0xf4794c7a (stale) CRC-32 checksum value for chunk [0, 0x24052): Mirror 1: 0xf4794c7a Mirror 3: 0xf4794c7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24052: skipped 000068 1787575521.992595 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000069 1787575522.342546 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000070 1787575522.557362 read 0x0028e5 thru 0x00896f (0x608b bytes) 000073 1787575522.574016 write 0x01e92b thru 0x02a075 (0xb74b bytes) 000076 1787575522.602852 read 0x00b8dc thru 0x00e25e (0x2983 bytes) 000084 1787575522.617883 read 0x029d2b thru 0x02a075 (0x34b bytes) 000086 1787575522.620039 mapwrite 0x00cbcc thru 0x00e2fe (0x1733 bytes) 000087 1787575522.696294 write 0x021bae thru 0x027c47 (0x609a bytes) 000092 1787575522.703392 write 0x00d820 thru 0x014ec8 (0x76a9 bytes) 000093 1787575522.704991 mapread 0x00f635 thru 0x016ad5 (0x74a1 bytes) 000097 1787575522.712336 mapread 0x01bd11 thru 0x0221f6 (0x64e6 bytes) 000098 1787575522.717388 write 0x027ad7 thru 0x02fbf8 (0x8122 bytes) 000099 1787575522.718650 mapread 0x012144 thru 0x017e6b (0x5d28 bytes) 000103 1787575522.727322 trunc from 0x02fbf9 to 0x01dd35 (0x11ec3 bytes) truncating to largest ever: 0x2d71d 000105 1787575522.772459 trunc from 0x01dd36 to 0x02d71c (0xf9e7 bytes) 000107 1787575522.853101 mapread 0x02505e thru 0x025c40 (0xbe3 bytes) 000108 1787575522.879868 mapwrite 0x03476b thru 0x03ffff (0xb895 bytes) 000109 1787575523.041101 trunc from 0x040000 to 0x018f6f (0x27090 bytes) 000110 1787575523.097265 read 0x00112b thru 0x00f7f6 (0xe6cc bytes) 000113 1787575523.106470 mapwrite 0x01d40b thru 0x026a0a (0x9600 bytes) 000114 1787575523.261653 read 0x0221aa thru 0x026a0a (0x4861 bytes) 000116 1787575523.263557 write 0x014458 thru 0x01b1db (0x6d84 bytes) 000117 1787575523.265507 read 0x000e0e thru 0x00e3e6 (0xd5d9 bytes) 000118 1787575523.271193 MIRROR_EXTEND to 2 mirrors 000119 1787575524.084748 write 0x024054 thru 0x02c058 (0x8005 bytes) 000121 1787575524.257976 read 0x011d85 thru 0x020042 (0xe2be bytes) 000125 1787575524.282613 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000126 1787575524.462336 trunc from 0x02c059 to 0x004501 (0x27b57 bytes) 000130 1787575524.510324 write 0x000609 thru 0x008a0c (0x8404 bytes) 000131 1787575524.516468 trunc from 0x008a0d to 0x004c59 (0x3db3 bytes) 000133 1787575524.535622 mapread 0x003ff4 thru 0x004c59 (0xc66 bytes) 000134 1787575524.536915 read 0x0034db thru 0x004c59 (0x177f bytes) 000135 1787575524.537906 mapread 0x00270f thru 0x004c59 (0x254b bytes) 000136 1787575524.541177 mapread 0x000058 thru 0x004c59 (0x4c02 bytes) 000137 1787575524.545192 write 0x0085fc thru 0x015dce (0xd7d3 bytes) truncating to largest ever: 0x37322 000139 1787575524.547147 trunc from 0x015dcf to 0x037321 (0x21553 bytes) 000140 1787575524.570590 read 0x02ec3d thru 0x033fda (0x539e bytes) 000141 1787575524.586633 mapwrite 0x00b063 thru 0x0103fb (0x5399 bytes) 000143 1787575524.668991 mapwrite 0x0017bf thru 0x010e7a (0xf6bc bytes) 000145 1787575524.798929 write 0x018a94 thru 0x023a61 (0xafce bytes) 000147 1787575524.803211 read 0x02ced6 thru 0x02da26 (0xb51 bytes) 000148 1787575524.823398 MIRROR_EXTEND to 2 mirrors 000149 1787575526.253749 write 0x03d64f thru 0x03ffff (0x29b1 bytes) 000150 1787575526.394682 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000151 1787575526.697317 MIRROR_EXTEND to 2 mirrors 000152 1787575528.103662 mapwrite 0x018f4d thru 0x02191a (0x89ce bytes) 000153 1787575528.430555 read 0x0072b2 thru 0x008d08 (0x1a57 bytes) 000154 1787575528.443913 mapread 0x03d6bc thru 0x03ffff (0x2944 bytes) 000155 1787575528.467232 mapread 0x02e1ae thru 0x03c872 (0xe6c5 bytes) 000156 1787575528.488093 read 0x034bec thru 0x03c510 (0x7925 bytes) 000160 1787575528.494876 write 0x02c314 thru 0x02c597 (0x284 bytes) 000162 1787575528.496816 read 0x037a30 thru 0x03ffff (0x85d0 bytes) 000163 1787575528.503617 mapread 0x021afc thru 0x027905 (0x5e0a bytes) 000164 1787575528.523314 write 0x03049e thru 0x030a9a (0x5fd bytes) 000165 1787575528.524757 trunc from 0x040000 to 0x035028 (0xafd7 bytes) 000166 1787575528.578847 write 0x036ad8 thru 0x03ffff (0x9528 bytes) 000169 1787575528.584357 mapread 0x03262a thru 0x03be52 (0x9829 bytes) 000170 1787575528.586205 write 0x030f43 thru 0x039472 (0x8530 bytes) 000171 1787575528.587506 read 0x02c933 thru 0x02fbfb (0x32c9 bytes) 000173 1787575528.589732 write 0x028621 thru 0x02878b (0x16b bytes) 000177 1787575528.601695 read 0x0268f0 thru 0x034d21 (0xe432 bytes) 000179 1787575528.602859 mapread 0x03787e thru 0x03cd7d (0x5500 bytes) 000180 1787575528.603715 write 0x01ff55 thru 0x02048f (0x53b bytes) 000183 1787575528.611156 write 0x017008 thru 0x01a57d (0x3576 bytes) 000184 1787575528.612842 mapwrite 0x01eaf7 thru 0x02c358 (0xd862 bytes) 000185 1787575528.776240 write 0x035801 thru 0x038205 (0x2a05 bytes) 000186 1787575528.777378 write 0x0074e1 thru 0x00de24 (0x6944 bytes) 000189 1787575528.789289 mapwrite 0x039184 thru 0x03a4a8 (0x1325 bytes) 000190 1787575528.908534 read 0x01a043 thru 0x01e6a8 (0x4666 bytes) 000191 1787575528.909497 trunc from 0x040000 to 0x012db6 (0x2d249 bytes) 000192 1787575528.948535 MIRROR_RESYNC 2 mirrors 000193 1787575530.308078 mapread 0x00e390 thru 0x012db6 (0x4a27 bytes) 000194 1787575530.351269 mapread 0x00edd0 thru 0x012db6 (0x3fe7 bytes) 000195 1787575530.353082 write 0x02b6dc thru 0x032339 (0x6c5e bytes) 000197 1787575530.563251 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000201 1787575530.825648 write 0x03c329 thru 0x03eed7 (0x2baf bytes) 000202 1787575530.851902 mapread 0x0341cf thru 0x03eed7 (0xad09 bytes) 000203 1787575530.889623 mapwrite 0x03cd98 thru 0x03ffff (0x3268 bytes) 000204 1787575531.013407 mapread 0x0207e8 thru 0x0286c7 (0x7ee0 bytes) 000205 1787575531.025791 read 0x019a80 thru 0x02473c (0xacbd bytes) 000206 1787575531.029151 trunc from 0x040000 to 0x026a65 (0x1959a bytes) 000208 1787575531.052977 mapread 0x00f915 thru 0x016b62 (0x724e bytes) 000209 1787575531.072472 read 0x003c20 thru 0x00a92a (0x6d0b bytes) 000212 1787575531.104873 mapread 0x0039a6 thru 0x010b0e (0xd169 bytes) truncating to largest ever: 0x3839d 000216 1787575531.116136 trunc from 0x026a66 to 0x03839c (0x11937 bytes) 000217 1787575531.167341 trunc from 0x03839d to 0x02f298 (0x9104 bytes) 000218 1787575531.228211 mapread 0x005ede thru 0x008f0f (0x3032 bytes) 000219 1787575531.229790 write 0x014bf3 thru 0x017cb1 (0x30bf bytes) 000222 1787575531.247259 read 0x0119db thru 0x01881c (0x6e42 bytes) 000223 1787575531.253164 mapwrite 0x013705 thru 0x022295 (0xeb91 bytes) 000224 1787575531.378464 mapread 0x01e49d thru 0x0238d7 (0x543b bytes) 000225 1787575531.384399 write 0x006f57 thru 0x015f9e (0xf048 bytes) 000227 1787575531.390353 trunc from 0x02f299 to 0x014dc8 (0x1a4d0 bytes) 000230 1787575531.461869 mapwrite 0x011c4f thru 0x014249 (0x25fb bytes) 000232 1787575531.578237 trunc from 0x014dc9 to 0x030da0 (0x1bfd8 bytes) 000233 1787575531.636491 mapwrite 0x020bcb thru 0x029756 (0x8b8c bytes) 000234 1787575531.784963 mapread 0x0197a9 thru 0x019dec (0x644 bytes) 000237 1787575531.812251 MIRROR_EXTEND to 2 mirrors 000238 1787575533.302845 write 0x002937 thru 0x007a5f (0x5129 bytes) 000242 1787575533.476059 mapwrite 0x01502b thru 0x01544a (0x420 bytes) 000246 1787575533.584120 read 0x02c0be thru 0x02ccec (0xc2f bytes) 000248 1787575533.601394 trunc from 0x030da1 to 0x034f9a (0x41fa bytes) 000250 1787575533.638162 trunc from 0x034f9b to 0x015883 (0x1f717 bytes) 000251 1787575533.665116 mapread 0x000e06 thru 0x00c870 (0xba6b bytes) 000252 1787575533.686056 write 0x013d53 thru 0x0171ff (0x34ad bytes) 000254 1787575533.705302 mapwrite 0x003850 thru 0x012094 (0xe845 bytes) 000257 1787575533.783379 mapread 0x007392 thru 0x011504 (0xa173 bytes) 000258 1787575533.784401 mapread 0x00a3b1 thru 0x012eca (0x8b1a bytes) 000261 1787575533.787763 read 0x01425d thru 0x0171ff (0x2fa3 bytes) 000262 1787575533.789865 write 0x037da2 thru 0x03e19b (0x63fa bytes) 000263 1787575533.792190 trunc from 0x03e19c to 0x02a717 (0x13a84 bytes) 000264 1787575533.810770 mapread 0x002583 thru 0x0043e1 (0x1e5f bytes) 000265 1787575533.818532 MIRROR_RESYNC 2 mirrors 000267 1787575534.710025 write 0x0159a0 thru 0x01f1d2 (0x9833 bytes) 000268 1787575534.831081 write 0x02305c thru 0x0295c9 (0x656e bytes) 000269 1787575534.842940 mapread 0x00b2e9 thru 0x00bd99 (0xab1 bytes) 000271 1787575534.850642 mapread 0x00cafa thru 0x00d0b3 (0x5ba bytes) 000274 1787575534.868985 trunc from 0x02a718 to 0x02ccff (0x25e8 bytes) 000275 1787575534.902120 MIRROR_RESYNC 2 mirrors 000277 1787575535.813278 write 0x0243cb thru 0x032183 (0xddb9 bytes) 000278 1787575535.967985 mapwrite 0x002f49 thru 0x005e80 (0x2f38 bytes) 000279 1787575536.100696 mapwrite 0x003027 thru 0x009d27 (0x6d01 bytes) 000280 1787575536.150771 read 0x0166fa thru 0x019918 (0x321f bytes) 000281 1787575536.159696 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000286 1787575536.352057 write 0x03e39e thru 0x03ffff (0x1c62 bytes) 000289 1787575536.360129 write 0x03e50c thru 0x03ffff (0x1af4 bytes) 000291 1787575536.363255 write 0x02f6ba thru 0x03f4e3 (0xfe2a bytes) 000293 1787575536.371697 mapread 0x03eca0 thru 0x03ffff (0x1360 bytes) 000294 1787575536.373018 write 0x00412b thru 0x004658 (0x52e bytes) 000295 1787575536.382332 trunc from 0x040000 to 0x029af9 (0x16506 bytes) 000297 1787575536.400984 write 0x007dfe thru 0x0127d4 (0xa9d7 bytes) 000299 1787575536.411896 mapwrite 0x03338d thru 0x03649c (0x3110 bytes) 000300 1787575536.508337 write 0x0314b4 thru 0x0333c0 (0x1f0d bytes) 000301 1787575536.515728 mapwrite 0x01baaf thru 0x01c7b3 (0xd05 bytes) 000302 1787575536.677907 trunc from 0x03649d to 0x01c200 (0x1a29c bytes) 000305 1787575536.732258 mapwrite 0x01e2fc thru 0x0244bd (0x61c2 bytes) 000306 1787575536.844722 read 0x017339 thru 0x021219 (0x9ee1 bytes) 000308 1787575536.851294 mapwrite 0x01b3c3 thru 0x021e1f (0x6a5d bytes) 000311 1787575536.941703 mapread 0x010c9d thru 0x011442 (0x7a6 bytes) 000316 1787575536.946813 read 0x01e3db thru 0x02265f (0x4285 bytes) 000317 1787575536.947790 mapread 0x0028cb thru 0x010284 (0xd9ba bytes) 000318 1787575536.958704 write 0x017ab5 thru 0x01afef (0x353b bytes) 000320 1787575536.961336 trunc from 0x0244be to 0x0288eb (0x442e bytes) 000322 1787575536.980772 write 0x0247be thru 0x02ecde (0xa521 bytes) 000323 1787575536.983345 mapread 0x01b70e thru 0x0287df (0xd0d2 bytes) 000329 1787575536.988884 trunc from 0x02ecdf to 0x000da0 (0x2df3e bytes) 000330 1787575537.011779 mapwrite 0x0044da thru 0x010335 (0xbe5c bytes) 000331 1787575537.164191 trunc from 0x010336 to 0x031d0d (0x219d8 bytes) 000333 1787575537.186980 trunc from 0x031d0e to 0x006f78 (0x2ad95 bytes) 000335 1787575537.205980 read 0x00457c thru 0x006f78 (0x29fd bytes) 000337 1787575537.207113 mapread 0x004a05 thru 0x006f78 (0x2574 bytes) 000340 1787575537.212285 mapread 0x004027 thru 0x006f78 (0x2f52 bytes) 000341 1787575537.221494 MIRROR_EXTEND to 2 mirrors 000344 1787575538.004911 read 0x002a84 thru 0x006f78 (0x44f5 bytes) 000346 1787575538.015997 write 0x00f567 thru 0x013aa2 (0x453c bytes) 000347 1787575538.130348 trunc from 0x013aa3 to 0x02e17a (0x1a6d8 bytes) 000351 1787575538.186497 mapread 0x0108a9 thru 0x01eb3a (0xe292 bytes) 000352 1787575538.207700 write 0x01b742 thru 0x02840c (0xcccb bytes) 000354 1787575538.214228 trunc from 0x02e17b to 0x0104e3 (0x1dc97 bytes) 000356 1787575538.238761 write 0x009734 thru 0x00d506 (0x3dd3 bytes) 000357 1787575538.250817 write 0x01566f thru 0x021f0e (0xc8a0 bytes) 000358 1787575538.262840 mapread 0x005fd8 thru 0x0150d9 (0xf102 bytes) 000359 1787575538.307842 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000361 1787575538.527810 read 0x00c2ea thru 0x016439 (0xa150 bytes) 000362 1787575538.565610 MIRROR_EXTEND to 2 mirrors 000363 1787575539.789994 mapread 0x01dd62 thru 0x01f769 (0x1a08 bytes) 000365 1787575539.815581 mapwrite 0x021eb4 thru 0x02551a (0x3667 bytes) 000366 1787575540.142312 trunc from 0x02551b to 0x02dd6b (0x8851 bytes) 000367 1787575540.178483 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000368 1787575540.354734 write 0x035a72 thru 0x03c78d (0x6d1c bytes) 000369 1787575540.357173 read 0x00b311 thru 0x019366 (0xe056 bytes) 000370 1787575540.380267 trunc from 0x03c78e to 0x0138a8 (0x28ee5 bytes) 000373 1787575540.406956 write 0x0343ac thru 0x03ffff (0xbc54 bytes) 000376 1787575540.410469 mapwrite 0x02f2de thru 0x032798 (0x34bb bytes) 000378 1787575540.484738 MIRROR_EXTEND to 2 mirrors 000380 1787575541.222027 MIRROR_RESYNC 2 mirrors 000381 1787575541.729764 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000383 1787575542.003597 mapwrite 0x00f0b0 thru 0x01e85e (0xf7af bytes) 000386 1787575542.226610 mapread 0x008b3b thru 0x00e483 (0x5949 bytes) 000388 1787575542.242956 write 0x01ba28 thru 0x0218d2 (0x5eab bytes) 000389 1787575542.249252 MIRROR_EXTEND to 2 mirrors 000390 1787575543.691059 mapread 0x02ca34 thru 0x03b7dc (0xeda9 bytes) 000391 1787575543.750478 trunc from 0x040000 to 0x011413 (0x2ebec bytes) truncating to largest ever: 0x3f970 000393 1787575543.959340 trunc from 0x011414 to 0x03f96f (0x2e55c bytes) 000394 1787575544.006340 read 0x0082fb thru 0x00a337 (0x203d bytes) 000395 1787575544.018395 mapread 0x023e97 thru 0x0319d1 (0xdb3b bytes) 000396 1787575544.037471 write 0x029c92 thru 0x02b596 (0x1905 bytes) 000397 1787575544.038833 trunc from 0x03f970 to 0x00637a (0x395f5 bytes) 000398 1787575544.067636 trunc from 0x00637b to 0x0125bc (0xc242 bytes) 000399 1787575544.098241 mapwrite 0x00357a thru 0x012545 (0xefcc bytes) 000400 1787575544.168240 read 0x00032c thru 0x002f42 (0x2c17 bytes) 000401 1787575544.172871 write 0x035161 thru 0x03ffff (0xae9f bytes) 000404 1787575544.175597 read 0x035288 thru 0x039be5 (0x495e bytes) 000406 1787575544.176820 read 0x0059d0 thru 0x006483 (0xab4 bytes) 000407 1787575544.177804 write 0x0359af thru 0x03fadd (0xa12f bytes) 000410 1787575544.180231 write 0x0073b2 thru 0x00a276 (0x2ec5 bytes) 000412 1787575544.182483 mapwrite 0x03257e thru 0x033564 (0xfe7 bytes) 000413 1787575544.371201 mapread 0x01cca7 thru 0x02b3c9 (0xe723 bytes) 000414 1787575544.372553 mapread 0x01bd7a thru 0x024ce0 (0x8f67 bytes) 000421 1787575544.382726 mapwrite 0x01dc50 thru 0x028761 (0xab12 bytes) 000424 1787575544.515778 mapread 0x03d978 thru 0x03e3a9 (0xa32 bytes) 000426 1787575544.524210 mapwrite 0x006bd0 thru 0x010f04 (0xa335 bytes) 000427 1787575544.654950 MIRROR_RESYNC 2 mirrors 000429 1787575545.597744 trunc from 0x040000 to 0x016780 (0x2987f bytes) 000430 1787575545.695403 mapread 0x016418 thru 0x016780 (0x369 bytes) 000432 1787575545.704286 mapread 0x010508 thru 0x016780 (0x6279 bytes) 000434 1787575545.732026 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x16781): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16781): Mirror 2: 0x1c7f267e (stale) CRC-32 checksum value for chunk [0, 0x16781): Mirror 1: 0x1c7f267e Mirror 3: 0x1c7f267e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16781: skipped 000435 1787575547.038863 MIRROR_RESYNC 3 mirrors 000436 1787575548.000671 read 0x0085a0 thru 0x0141ca (0xbc2b bytes) 000437 1787575548.008958 write 0x03b03f thru 0x03ffff (0x4fc1 bytes) 000439 1787575548.097862 mapread 0x03e4b8 thru 0x03ffff (0x1b48 bytes) 000440 1787575548.099051 read 0x01db26 thru 0x029e67 (0xc342 bytes) 000442 1787575548.114358 read 0x0006d2 thru 0x00ff9c (0xf8cb bytes) 000444 1787575548.115572 mapread 0x0087d8 thru 0x0091fc (0xa25 bytes) 000445 1787575548.116372 mapread 0x007b3e thru 0x011b16 (0x9fd9 bytes) 000446 1787575548.123046 MIRROR_RESYNC 3 mirrors 000447 1787575549.353205 write 0x0082e9 thru 0x0168df (0xe5f7 bytes) 000451 1787575549.523170 mapwrite 0x0185f3 thru 0x021452 (0x8e60 bytes) 000452 1787575549.682486 MIRROR_RESYNC 3 mirrors 000453 1787575550.651988 MIRROR_EXTEND to 4 mirrors 000454 1787575551.636258 write 0x03410c thru 0x03ffff (0xbef4 bytes) 000455 1787575551.805073 trunc from 0x040000 to 0x03583a (0xa7c5 bytes) 000456 1787575551.856894 mapread 0x025c1f thru 0x02a1f8 (0x45da bytes) 000457 1787575551.894882 write 0x025f21 thru 0x034a41 (0xeb21 bytes) 000458 1787575551.896987 write 0x003256 thru 0x00858e (0x5339 bytes) 000459 1787575551.910128 read 0x012c5f thru 0x01c997 (0x9d39 bytes) 000460 1787575551.925412 read 0x015a06 thru 0x01a230 (0x482b bytes) 000462 1787575551.929887 trunc from 0x03583b to 0x01e22b (0x1760f bytes) 000463 1787575551.957176 read 0x016091 thru 0x01ab7e (0x4aee bytes) 000464 1787575551.961360 read 0x001874 thru 0x00f056 (0xd7e3 bytes) 000465 1787575551.981862 mapwrite 0x01e575 thru 0x02a1cd (0xbc59 bytes) 000466 1787575552.080911 read 0x01441c thru 0x01d3b3 (0x8f98 bytes) 000467 1787575552.082318 write 0x0328d6 thru 0x038b7d (0x62a8 bytes) 000468 1787575552.087631 write 0x00d283 thru 0x00e6ca (0x1448 bytes) 000469 1787575552.094414 write 0x02547c thru 0x02a21e (0x4da3 bytes) 000470 1787575552.102494 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38b7e): Mirror 2: 0xaab49a33 (stale) Mirror 3: 0xaab49a33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b7e): Mirror 2: 0xaab49a33 (stale) Mirror 3: 0xaab49a33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b7e: skipped 000471 1787575552.546657 trunc from 0x038b7e to 0x0351b6 (0x39c7 bytes) 000472 1787575552.568069 write 0x02841f thru 0x030e79 (0x8a5b bytes) 000473 1787575552.577929 trunc from 0x0351b7 to 0x01f27c (0x15f3a bytes) 000474 1787575552.594782 write 0x012f83 thru 0x022245 (0xf2c3 bytes) 000479 1787575552.610005 mapread 0x01440c thru 0x019657 (0x524c bytes) 000481 1787575552.614039 mapread 0x00e372 thru 0x01ac24 (0xc8b3 bytes) 000482 1787575552.631920 mapread 0x0012f5 thru 0x009e15 (0x8b21 bytes) 000483 1787575552.641890 mapread 0x013d5a thru 0x01fb25 (0xbdcc bytes) 000484 1787575552.643023 write 0x009307 thru 0x00b6b4 (0x23ae bytes) 000486 1787575552.645037 write 0x0211c4 thru 0x02d2f7 (0xc134 bytes) 000488 1787575552.649072 read 0x014253 thru 0x018a93 (0x4841 bytes) 000489 1787575552.650171 trunc from 0x02d2f8 to 0x01daf1 (0xf806 bytes) 000490 1787575552.674244 read 0x00f1a3 thru 0x01764f (0x84ad bytes) 000491 1787575552.674983 write 0x0029e0 thru 0x006364 (0x3985 bytes) 000492 1787575552.676044 mapwrite 0x01fae8 thru 0x0272d6 (0x77ef bytes) 000494 1787575552.755352 mapwrite 0x002818 thru 0x00a8c1 (0x80aa bytes) 000495 1787575552.790026 mapwrite 0x014552 thru 0x021bc9 (0xd678 bytes) 000497 1787575552.889038 mapwrite 0x01a143 thru 0x0265b1 (0xc46f bytes) 000498 1787575553.018057 read 0x02002a thru 0x0272d6 (0x72ad bytes) 000499 1787575553.026839 MIRROR_RESYNC 3 mirrors 000500 1787575554.366901 MIRROR_EXTEND to 4 mirrors 000502 1787575555.177178 MIRROR_RESYNC 4 mirrors 000509 1787575555.885099 write 0x01d8ed thru 0x023329 (0x5a3d bytes) 000510 1787575555.999196 mapread 0x01e66b thru 0x021fa7 (0x393d bytes) 000511 1787575556.003078 mapread 0x00cf54 thru 0x019f71 (0xd01e bytes) 000512 1787575556.041091 mapwrite 0x00d29d thru 0x01cecc (0xfc30 bytes) 000513 1787575556.096840 MIRROR_RESYNC 4 mirrors 000516 1787575557.187854 write 0x01b6a1 thru 0x0270b5 (0xba15 bytes) 000518 1787575557.296855 write 0x0085b9 thru 0x00d972 (0x53ba bytes) 000519 1787575557.311777 read 0x01bd8b thru 0x0272d6 (0xb54c bytes) 000520 1787575557.312531 read 0x010383 thru 0x012411 (0x208f bytes) 000521 1787575557.339685 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x272d7): Mirror 3: 0xc10fc23b (stale) Mirror 4: 0xc10fc23b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x272d7): Mirror 3: 0xc10fc23b (stale) Mirror 4: 0xc10fc23b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x272d7: skipped 000522 1787575557.860098 mapread 0x02475d thru 0x0272d6 (0x2b7a bytes) 000523 1787575557.903994 MIRROR_RESYNC 3 mirrors 000525 1787575559.062030 mapread 0x010dcc thru 0x0114ce (0x703 bytes) 000527 1787575559.100273 trunc from 0x0272d7 to 0x00e9f8 (0x188de bytes) 000528 1787575559.234199 mapwrite 0x034dec thru 0x0364f3 (0x1708 bytes) 000529 1787575559.318236 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x364f4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364f4): Mirror 3: 0xbf653b3a (stale) Mirror 4: 0xbf653b3a (stale) CRC-32 checksum value for chunk [0, 0x364f4): Mirror 1: 0xbfc82bf Mirror 5: 0xbfc82bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364f4: skipped 000530 1787575560.752890 mapread 0x017875 thru 0x0260dc (0xe868 bytes) 000531 1787575560.799011 trunc from 0x0364f4 to 0x03d1c7 (0x6cd4 bytes) 000532 1787575560.912840 read 0x02573b thru 0x029261 (0x3b27 bytes) 000533 1787575560.935860 mapwrite 0x00a3a6 thru 0x0112c9 (0x6f24 bytes) 000534 1787575561.112195 read 0x01fcc6 thru 0x02ae98 (0xb1d3 bytes) 000535 1787575561.128584 mapread 0x0340cf thru 0x0340f4 (0x26 bytes) 000536 1787575561.129404 read 0x02d791 thru 0x03a8ae (0xd11e bytes) 000537 1787575561.129986 write 0x000b8e thru 0x00c919 (0xbd8c bytes) 000540 1787575561.133912 write 0x027a9b thru 0x036a5a (0xefc0 bytes) 000541 1787575561.151186 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d1c8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1c8): Mirror 3: 0x4b2c444b (stale) Mirror 4: 0x4b2c444b (stale) Mirror 5: 0xe08f350c (stale) CRC-32 checksum value for chunk [0, 0x3d1c8): Mirror 1: 0xfeba4ef1 Mirror 6: 0xfeba4ef1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1c8: skipped 000542 1787575562.235879 mapwrite 0x01c2ca thru 0x029082 (0xcdb9 bytes) 000543 1787575562.628912 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d1c8): Mirror 4: 0x4b2c444b (stale) Mirror 5: 0xe08f350c (stale) Mirror 6: 0xfeba4ef1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1c8): Mirror 4: 0x4b2c444b (stale) Mirror 5: 0xe08f350c (stale) Mirror 6: 0xfeba4ef1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1c8: skipped 000546 1787575563.141181 write 0x01b676 thru 0x027b2d (0xc4b8 bytes) 000549 1787575563.156263 mapread 0x0283cc thru 0x029c5f (0x1894 bytes) 000550 1787575563.175442 mapwrite 0x0065dc thru 0x0095d0 (0x2ff5 bytes) 000551 1787575563.284067 write 0x030d25 thru 0x0384c3 (0x779f bytes) 000553 1787575563.321261 write 0x0160eb thru 0x0239d4 (0xd8ea bytes) 000554 1787575563.332944 mapwrite 0x00a84b thru 0x01204b (0x7801 bytes) 000555 1787575563.463896 mapread 0x023b91 thru 0x02cf1c (0x938c bytes) 000558 1787575563.477479 read 0x00d969 thru 0x01b94b (0xdfe3 bytes) 000559 1787575563.478172 mapwrite 0x037849 thru 0x03bfd3 (0x478b bytes) 000560 1787575563.549622 read 0x0241dd thru 0x02e3c3 (0xa1e7 bytes) 000561 1787575563.550254 mapread 0x01f15f thru 0x022ecf (0x3d71 bytes) 000562 1787575563.551059 write 0x023e22 thru 0x031e6e (0xe04d bytes) 000563 1787575563.559316 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d1c8): Mirror 4: 0x4b2c444b (stale) Mirror 5: 0xe08f350c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1c8): Mirror 4: 0x4b2c444b (stale) Mirror 5: 0xe08f350c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1c8: skipped 000565 1787575564.174173 mapwrite 0x0214e0 thru 0x02c814 (0xb335 bytes) 000567 1787575564.334681 write 0x02aaed thru 0x032732 (0x7c46 bytes) 000568 1787575564.343688 MIRROR_RESYNC 3 mirrors 000572 1787575565.971896 read 0x01788e thru 0x026fda (0xf74d bytes) 000574 1787575565.995619 mapread 0x025c46 thru 0x02b042 (0x53fd bytes) 000576 1787575566.046596 mapwrite 0x031709 thru 0x034468 (0x2d60 bytes) 000577 1787575566.315243 write 0x03fc33 thru 0x03ffff (0x3cd bytes) 000579 1787575566.321014 mapread 0x00ee15 thru 0x011853 (0x2a3f bytes) 000580 1787575566.372003 mapread 0x010c59 thru 0x010eaa (0x252 bytes) 000582 1787575566.380912 read 0x02cc06 thru 0x033a06 (0x6e01 bytes) 000583 1787575566.383164 trunc from 0x040000 to 0x0248ad (0x1b752 bytes) 000584 1787575566.416022 trunc from 0x0248ae to 0x00f248 (0x15665 bytes) 000589 1787575566.455685 mapread 0x00c2dd thru 0x00f248 (0x2f6c bytes) 000590 1787575566.456703 write 0x025577 thru 0x0335ac (0xe036 bytes) 000594 1787575566.472402 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x335ad): Mirror 4: 0x3050e843 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x335ad): Mirror 4: 0x3050e843 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x335ad: skipped 000595 1787575566.909986 read 0x028b9d thru 0x030bd8 (0x803c bytes) 000596 1787575566.939522 write 0x005902 thru 0x00c833 (0x6f32 bytes) 000597 1787575566.962143 trunc from 0x0335ad to 0x01a199 (0x19413 bytes) 000599 1787575566.991352 mapread 0x004cf6 thru 0x008eef (0x41fa bytes) 000600 1787575567.000062 write 0x008054 thru 0x011370 (0x931d bytes) 000602 1787575567.008061 read 0x009561 thru 0x011d18 (0x87b8 bytes) 000606 1787575567.016125 MIRROR_RESYNC 2 mirrors 000607 1787575568.099197 write 0x00021d thru 0x0012f3 (0x10d7 bytes) 000608 1787575568.218293 mapwrite 0x0037fe thru 0x01260b (0xee0e bytes) 000612 1787575568.425204 write 0x0052fa thru 0x014cbc (0xf9c3 bytes) 000614 1787575568.433709 trunc from 0x01a19a to 0x00885f (0x1193a bytes) 000617 1787575568.517608 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x8860): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8860): Mirror 4: 0x3031d55f (stale) CRC-32 checksum value for chunk [0, 0x8860): Mirror 1: 0x20d5f20b Mirror 5: 0x20d5f20b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8860: skipped 000618 1787575569.892591 mapwrite 0x03d97a thru 0x03ffff (0x2686 bytes) 000620 1787575570.168045 mapread 0x00da95 thru 0x01ac31 (0xd19d bytes) 000621 1787575570.195913 trunc from 0x040000 to 0x005a04 (0x3a5fb bytes) 000622 1787575570.233274 read 0x003ae8 thru 0x005a04 (0x1f1d bytes) 000623 1787575570.244393 read 0x0044aa thru 0x005a04 (0x155b bytes) 000625 1787575570.254891 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x5a05): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5a05): Mirror 4: 0x2512547 (stale) Mirror 5: 0xe75ec82b (stale) CRC-32 checksum value for chunk [0, 0x5a05): Mirror 1: 0xe75ec82b Mirror 6: 0xe75ec82b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5a05: skipped 000626 1787575571.416159 write 0x03bc43 thru 0x03cdef (0x11ad bytes) 000628 1787575571.585734 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cdf0): Mirror 4: 0x6d91ad98 (stale) Mirror 5: 0xf5f852ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cdf0): Mirror 4: 0x6d91ad98 (stale) Mirror 5: 0xf5f852ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cdf0: skipped 000630 1787575572.034860 write 0x038f22 thru 0x03ffff (0x70de bytes) 000631 1787575572.048541 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x859da718 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x859da718 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000632 1787575572.355108 mapwrite 0x02af12 thru 0x039863 (0xe952 bytes) 000634 1787575572.582468 trunc from 0x040000 to 0x021dce (0x1e231 bytes) 000635 1787575572.661110 mapread 0x00f813 thru 0x0158d3 (0x60c1 bytes) 000636 1787575572.673914 mapread 0x01ee01 thru 0x02176d (0x296d bytes) 000638 1787575572.685750 write 0x00dbe8 thru 0x01db50 (0xff69 bytes) 000639 1787575572.688405 mapread 0x00cecc thru 0x016104 (0x9239 bytes) 000640 1787575572.691074 mapread 0x017484 thru 0x021dce (0xa94b bytes) 000641 1787575572.695612 write 0x0319be thru 0x038a30 (0x7073 bytes) 000642 1787575572.698788 trunc from 0x038a31 to 0x0140c1 (0x2496f bytes) 000646 1787575572.721213 write 0x028371 thru 0x0380bd (0xfd4d bytes) 000652 1787575572.740481 read 0x02e0d6 thru 0x0330dc (0x5007 bytes) 000654 1787575572.741558 read 0x02e818 thru 0x030deb (0x25d4 bytes) 000656 1787575572.744383 mapread 0x01f547 thru 0x022061 (0x2b1b bytes) 000657 1787575572.756641 mapread 0x02b8a1 thru 0x0380bd (0xc81d bytes) 000658 1787575572.758910 read 0x003bfe thru 0x00ddbd (0xa1c0 bytes) 000660 1787575572.768122 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x380be): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380be): Mirror 5: 0xfd735d3b (stale) CRC-32 checksum value for chunk [0, 0x380be): Mirror 1: 0x86e40d6d Mirror 6: 0x86e40d6d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380be: skipped 000662 1787575573.863688 MIRROR_RESYNC 3 mirrors 000664 1787575574.550813 read 0x0193ab thru 0x02809e (0xecf4 bytes) 000665 1787575574.569330 mapwrite 0x02b092 thru 0x02fd6d (0x4cdc bytes) 000667 1787575574.917399 write 0x02140d thru 0x0273b7 (0x5fab bytes) 000668 1787575574.949212 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x380be): Mirror 6: 0x86e40d6d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380be): Mirror 6: 0x86e40d6d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380be: skipped 000670 1787575575.363920 mapread 0x002062 thru 0x010df1 (0xed90 bytes) 000671 1787575575.403829 trunc from 0x0380be to 0x000b99 (0x37524 bytes) 000672 1787575575.421102 read 0x0002ba thru 0x000b99 (0x8e0 bytes) 000674 1787575575.429218 trunc from 0x000b9a to 0x005c51 (0x50b8 bytes) 000675 1787575575.447373 trunc from 0x005c52 to 0x00924c (0x35fb bytes) 000679 1787575575.480593 mapread 0x004a86 thru 0x00924c (0x47c7 bytes) 000680 1787575575.494637 mapwrite 0x00a425 thru 0x010a54 (0x6630 bytes) 000682 1787575575.594753 trunc from 0x010a55 to 0x014a3d (0x3fe9 bytes) 000684 1787575575.625783 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000686 1787575575.822825 mapread 0x0032a3 thru 0x009245 (0x5fa3 bytes) 000687 1787575575.877589 trunc from 0x014a3e to 0x03f1ae (0x2a771 bytes) 000688 1787575575.898859 read 0x018292 thru 0x021a60 (0x97cf bytes) 000689 1787575575.931599 write 0x005a66 thru 0x00b478 (0x5a13 bytes) 000696 1787575575.943036 trunc from 0x03f1af to 0x017973 (0x2783b bytes) 000698 1787575576.001881 mapread 0x005378 thru 0x013460 (0xe0e9 bytes) 000699 1787575576.046023 mapwrite 0x02d9cc thru 0x02e8ad (0xee2 bytes) 000702 1787575576.277314 write 0x011e5d thru 0x01f85a (0xd9fe bytes) 000703 1787575576.308729 trunc from 0x02e8ae to 0x01e675 (0x10238 bytes) 000704 1787575576.390642 mapwrite 0x031c47 thru 0x03d980 (0xbd3a bytes) 000705 1787575576.604297 mapread 0x02ae5f thru 0x031df7 (0x6f99 bytes) 000710 1787575576.609905 mapread 0x02c291 thru 0x03add4 (0xeb44 bytes) 000711 1787575576.611228 trunc from 0x03d981 to 0x035e93 (0x7aed bytes) 000713 1787575576.653877 trunc from 0x035e94 to 0x002826 (0x3366d bytes) 000714 1787575576.698033 mapread 0x0004ab thru 0x002826 (0x237c bytes) 000715 1787575576.708799 read 0x001a47 thru 0x002826 (0xde0 bytes) 000718 1787575576.710260 mapread 0x00199f thru 0x002826 (0xe88 bytes) 000720 1787575576.711521 mapwrite 0x02f240 thru 0x031498 (0x2259 bytes) 000721 1787575576.943496 write 0x016133 thru 0x0222e2 (0xc1b0 bytes) 000725 1787575577.011853 MIRROR_EXTEND to 2 mirrors 000726 1787575577.901924 mapwrite 0x0296d1 thru 0x02e818 (0x5148 bytes) 000729 1787575578.262820 mapread 0x009db2 thru 0x00bcd3 (0x1f22 bytes) 000730 1787575578.287639 read 0x0231e9 thru 0x027067 (0x3e7f bytes) 000731 1787575578.288197 write 0x002f25 thru 0x01003e (0xd11a bytes) 000732 1787575578.300013 MIRROR_RESYNC 2 mirrors 000734 1787575579.361794 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000735 1787575579.552277 read 0x01eaa2 thru 0x021f44 (0x34a3 bytes) 000739 1787575579.558735 read 0x002be1 thru 0x00ad0d (0x812d bytes) 000744 1787575579.576705 mapread 0x01fa76 thru 0x02c3fe (0xc989 bytes) 000745 1787575579.580415 read 0x01be9f thru 0x0272e7 (0xb449 bytes) 000746 1787575579.581199 mapread 0x015a9e thru 0x021356 (0xb8b9 bytes) 000747 1787575579.582072 write 0x011fde thru 0x0177b2 (0x57d5 bytes) 000748 1787575579.601056 mapread 0x01b214 thru 0x026292 (0xb07f bytes) 000752 1787575579.604751 write 0x0197c7 thru 0x01c30f (0x2b49 bytes) 000753 1787575579.605856 trunc from 0x031499 to 0x022358 (0xf140 bytes) 000756 1787575579.644914 mapread 0x0059d9 thru 0x007649 (0x1c71 bytes) 000757 1787575579.646110 mapread 0x003383 thru 0x007753 (0x43d1 bytes) 000758 1787575579.647922 trunc from 0x022359 to 0x02bc23 (0x98cb bytes) 000759 1787575579.660366 mapread 0x018544 thru 0x025ca2 (0xd75f bytes) 000761 1787575579.678682 MIRROR_EXTEND to 2 mirrors 000762 1787575580.302136 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000763 1787575580.518815 write 0x0039d1 thru 0x011496 (0xdac6 bytes) 000766 1787575580.570719 MIRROR_EXTEND to 2 mirrors 000767 1787575581.930205 read 0x005e24 thru 0x0100cf (0xa2ac bytes) 000768 1787575581.957881 trunc from 0x02bc24 to 0x02b840 (0x3e3 bytes) 000770 1787575582.049760 trunc from 0x02b841 to 0x012752 (0x190ee bytes) 000771 1787575582.078773 trunc from 0x012753 to 0x00aec2 (0x7890 bytes) 000772 1787575582.101918 trunc from 0x00aec3 to 0x010eb9 (0x5ff7 bytes) 000773 1787575582.125835 write 0x02011b thru 0x02ca0a (0xc8f0 bytes) 000774 1787575582.129368 mapwrite 0x026116 thru 0x03231e (0xc209 bytes) 000776 1787575582.227261 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3231f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3231f): Mirror 2: 0x89f9515 (stale) CRC-32 checksum value for chunk [0, 0x3231f): Mirror 1: 0xe1e7f4f7 Mirror 3: 0xe1e7f4f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3231f: skipped 000777 1787575583.923313 read 0x0280a4 thru 0x029695 (0x15f2 bytes) 000778 1787575583.960966 write 0x001f1a thru 0x00f9ab (0xda92 bytes) 000779 1787575584.257215 trunc from 0x03231f to 0x0144e2 (0x1de3c bytes) 000780 1787575584.351549 mapread 0x00e0ed thru 0x010250 (0x2164 bytes) 000781 1787575584.366650 mapwrite 0x000269 thru 0x000d1e (0xab6 bytes) 000783 1787575584.538057 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x144e3): Mirror 3: 0xd29539a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x144e3): Mirror 3: 0xd29539a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x144e3: skipped 000784 1787575584.997001 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000785 1787575585.153900 mapwrite 0x02f027 thru 0x02ff17 (0xef1 bytes) 000786 1787575585.234109 mapread 0x005136 thru 0x00d289 (0x8154 bytes) 000788 1787575585.256595 read 0x00901c thru 0x01695f (0xd944 bytes) 000789 1787575585.280124 write 0x01b505 thru 0x02285b (0x7357 bytes) 000791 1787575585.295480 mapread 0x002aee thru 0x0129be (0xfed1 bytes) 000793 1787575585.302958 MIRROR_EXTEND to 2 mirrors 000796 1787575586.724989 mapwrite 0x013616 thru 0x0179b8 (0x43a3 bytes) 000797 1787575587.055416 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000799 1787575587.280867 mapwrite 0x0107e2 thru 0x012921 (0x2140 bytes) 000801 1787575587.412872 read 0x024790 thru 0x0269c7 (0x2238 bytes) 000802 1787575587.422244 trunc from 0x02ff18 to 0x0338c6 (0x39af bytes) 000803 1787575587.445324 trunc from 0x0338c7 to 0x0303b7 (0x350f bytes) 000804 1787575587.463877 mapwrite 0x02b7b2 thru 0x030541 (0x4d90 bytes) 000805 1787575587.543315 mapwrite 0x03eaad thru 0x03ffff (0x1553 bytes) 000810 1787575587.678493 mapread 0x028068 thru 0x02a539 (0x24d2 bytes) 000813 1787575587.689403 mapread 0x016eb1 thru 0x017d42 (0xe92 bytes) 000814 1787575587.693404 mapwrite 0x002f31 thru 0x0049b8 (0x1a88 bytes) 000816 1787575587.804663 mapread 0x0062cc thru 0x006bcc (0x901 bytes) 000817 1787575587.805967 mapread 0x03b2dd thru 0x03eca9 (0x39cd bytes) 000820 1787575587.809885 trunc from 0x040000 to 0x007052 (0x38fad bytes) 000822 1787575587.834005 mapwrite 0x021d02 thru 0x02afa4 (0x92a3 bytes) 000823 1787575588.114552 mapread 0x0090ce thru 0x00bf8d (0x2ec0 bytes) 000824 1787575588.156827 mapread 0x022621 thru 0x029e33 (0x7813 bytes) 000825 1787575588.162703 write 0x033605 thru 0x034f26 (0x1922 bytes) 000826 1787575588.166471 mapread 0x019453 thru 0x01c70f (0x32bd bytes) 000827 1787575588.194526 read 0x030dcd thru 0x034f26 (0x415a bytes) 000828 1787575588.237966 MIRROR_EXTEND to 2 mirrors 000830 1787575590.243545 MIRROR_EXTEND to 3 mirrors 000832 1787575591.947693 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000834 1787575592.330158 write 0x0210e0 thru 0x026d28 (0x5c49 bytes) 000835 1787575592.393227 read 0x0044e9 thru 0x00b27d (0x6d95 bytes) 000836 1787575592.401673 write 0x01749c thru 0x02394c (0xc4b1 bytes) 000837 1787575592.412550 mapread 0x0264a4 thru 0x033679 (0xd1d6 bytes) 000839 1787575592.436408 mapwrite 0x0028d7 thru 0x00e3cf (0xbaf9 bytes) 000840 1787575592.698334 read 0x00cf61 thru 0x0118f9 (0x4999 bytes) 000841 1787575592.731995 read 0x0087b3 thru 0x01549a (0xcce8 bytes) 000845 1787575592.740361 write 0x0315ff thru 0x03e36b (0xcd6d bytes) 000848 1787575592.751003 write 0x01237b thru 0x017a65 (0x56eb bytes) 000849 1787575592.752848 trunc from 0x03e36c to 0x035f32 (0x8439 bytes) 000850 1787575592.813905 mapwrite 0x0393b0 thru 0x03ef56 (0x5ba7 bytes) 000851 1787575592.971514 trunc from 0x03ef57 to 0x00e3d8 (0x30b7e bytes) 000856 1787575593.011842 trunc from 0x00e3d9 to 0x014cfc (0x6924 bytes) 000858 1787575593.048485 mapwrite 0x01018f thru 0x015af5 (0x5967 bytes) 000859 1787575593.266534 write 0x016182 thru 0x01ef6e (0x8ded bytes) 000862 1787575593.273978 read 0x019e4b thru 0x01b723 (0x18d9 bytes) 000863 1787575593.275174 trunc from 0x01ef6f to 0x00ddb5 (0x111b9 bytes) 000864 1787575593.310846 read 0x003a7c thru 0x004d99 (0x131e bytes) 000865 1787575593.311905 write 0x03b8e2 thru 0x03ba06 (0x125 bytes) 000866 1787575593.320999 mapread 0x00723a thru 0x00b532 (0x42f9 bytes) 000868 1787575593.324172 write 0x0072e1 thru 0x00ce8e (0x5bae bytes) 000873 1787575593.364863 read 0x02d69a thru 0x02dc40 (0x5a7 bytes) 000874 1787575593.384752 read 0x021fb6 thru 0x024517 (0x2562 bytes) 000875 1787575593.385720 mapwrite 0x007347 thru 0x00a7e2 (0x349c bytes) 000876 1787575593.435623 mapread 0x017a39 thru 0x01f354 (0x791c bytes) 000877 1787575593.449412 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ba07): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba07): Mirror 3: 0x82038ba4 (stale) CRC-32 checksum value for chunk [0, 0x3ba07): Mirror 1: 0xf68dcbaf Mirror 4: 0xf68dcbaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba07: skipped 000882 1787575595.371241 mapread 0x02c2ae thru 0x02df14 (0x1c67 bytes) 000885 1787575595.419309 mapwrite 0x03b522 thru 0x03ffff (0x4ade bytes) 000886 1787575595.653671 read 0x01ee97 thru 0x029259 (0xa3c3 bytes) 000887 1787575595.661793 trunc from 0x040000 to 0x00f38d (0x30c72 bytes) 000889 1787575595.690747 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xf38e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf38e): Mirror 3: 0xf1f57650 (stale) Mirror 4: 0x869723e7 (stale) CRC-32 checksum value for chunk [0, 0xf38e): Mirror 1: 0x869723e7 Mirror 5: 0x869723e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf38e: skipped 000890 1787575597.789148 mapwrite 0x0021c6 thru 0x0068d9 (0x4714 bytes) 000893 1787575598.313658 write 0x014650 thru 0x01995e (0x530f bytes) 000894 1787575598.318206 trunc from 0x01995f to 0x00ba18 (0xdf46 bytes) 000895 1787575598.370845 mapread 0x00276c thru 0x0030c8 (0x95d bytes) 000896 1787575598.373314 trunc from 0x00ba19 to 0x0295a1 (0x1db89 bytes) 000897 1787575598.398701 write 0x0032a7 thru 0x00a0eb (0x6e45 bytes) 000899 1787575598.405789 trunc from 0x0295a2 to 0x006516 (0x2308b bytes) 000900 1787575598.432800 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6517): Mirror 3: 0x16214e8f (stale) Mirror 5: 0xc37c095a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6517): Mirror 3: 0x16214e8f (stale) Mirror 5: 0xc37c095a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6517: skipped 000901 1787575599.013653 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6517): Mirror 5: 0xc37c095a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6517): Mirror 5: 0xc37c095a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6517: skipped 000902 1787575599.464113 mapread 0x003191 thru 0x006516 (0x3386 bytes) 000905 1787575599.536232 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000908 1787575599.814956 mapwrite 0x032fd5 thru 0x03a1ce (0x71fa bytes) 000911 1787575599.921070 write 0x0320fd thru 0x03dfa2 (0xbea6 bytes) 000912 1787575599.922840 write 0x0361f3 thru 0x03ffff (0x9e0d bytes) 000913 1787575599.925323 trunc from 0x040000 to 0x0278f6 (0x18709 bytes) 000914 1787575599.966382 mapread 0x0032d5 thru 0x0060a0 (0x2dcc bytes) 000916 1787575599.992922 trunc from 0x0278f7 to 0x00d741 (0x1a1b5 bytes) 000917 1787575600.023152 mapwrite 0x03aa5b thru 0x03ffff (0x55a5 bytes) 000918 1787575600.235702 mapread 0x00e114 thru 0x01a662 (0xc54f bytes) 000920 1787575600.268094 read 0x035791 thru 0x03ad71 (0x55e1 bytes) 000921 1787575600.271558 write 0x0160ec thru 0x01aa30 (0x4945 bytes) 000922 1787575600.273587 mapwrite 0x023141 thru 0x0319f3 (0xe8b3 bytes) 000924 1787575600.341994 write 0x02474c thru 0x02a17f (0x5a34 bytes) 000925 1787575600.343113 trunc from 0x040000 to 0x02f767 (0x10898 bytes) 000926 1787575600.368651 write 0x03368b thru 0x033c01 (0x577 bytes) 000930 1787575600.373133 read 0x01bd3b thru 0x020f06 (0x51cc bytes) 000931 1787575600.374529 trunc from 0x033c02 to 0x02d039 (0x6bc8 bytes) 000932 1787575600.395037 mapwrite 0x024e39 thru 0x02b362 (0x652a bytes) 000935 1787575600.458842 trunc from 0x02d03a to 0x01553e (0x17afb bytes) 000936 1787575600.482988 mapread 0x008e62 thru 0x009922 (0xac1 bytes) 000940 1787575600.486004 mapwrite 0x0077de thru 0x00b778 (0x3f9b bytes) 000941 1787575600.565665 MIRROR_EXTEND to 2 mirrors 000944 1787575601.356758 mapread 0x00e09c thru 0x0114b7 (0x341c bytes) 000945 1787575601.433178 write 0x003880 thru 0x007b5b (0x42dc bytes) 000946 1787575601.770624 mapwrite 0x01c335 thru 0x02b723 (0xf3ef bytes) 000947 1787575602.090749 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000949 1787575602.286804 mapread 0x01cf03 thru 0x02b723 (0xe821 bytes) 000950 1787575602.317300 write 0x01d4d6 thru 0x0239c4 (0x64ef bytes) 000951 1787575602.318529 read 0x01d3a3 thru 0x028920 (0xb57e bytes) 000952 1787575602.319143 read 0x02230a thru 0x02b723 (0x941a bytes) 000954 1787575602.323842 read 0x0206c8 thru 0x021bbe (0x14f7 bytes) 000957 1787575602.326373 trunc from 0x02b724 to 0x00a9da (0x20d49 bytes) 000960 1787575602.350706 mapwrite 0x003d0d thru 0x007e13 (0x4107 bytes) 000962 1787575602.436391 MIRROR_EXTEND to 2 mirrors 000963 1787575603.161205 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000964 1787575603.450725 mapread 0x000a54 thru 0x00a9da (0x9f87 bytes) 000965 1787575603.493991 trunc from 0x00a9db to 0x0327e7 (0x27e0d bytes) 000967 1787575603.524770 trunc from 0x0327e8 to 0x024fc6 (0xd821 bytes) 000971 1787575603.542001 mapwrite 0x035ea5 thru 0x03d780 (0x78dc bytes) 000972 1787575603.753614 write 0x01b818 thru 0x024772 (0x8f5b bytes) 000973 1787575603.784740 read 0x034e8b thru 0x03cb6f (0x7ce5 bytes) 000974 1787575603.787528 write 0x039495 thru 0x03efcc (0x5b38 bytes) 000975 1787575603.796083 write 0x038d55 thru 0x03ffff (0x72ab bytes) 000980 1787575603.822009 trunc from 0x040000 to 0x00f031 (0x30fce bytes) 000983 1787575603.871788 trunc from 0x00f032 to 0x000729 (0xe908 bytes) 000984 1787575603.907718 mapread 0x00064c thru 0x000729 (0xde bytes) 000985 1787575603.925568 mapread 0x000022 thru 0x000729 (0x708 bytes) 000986 1787575603.929765 write 0x024ac1 thru 0x0300ff (0xb63f bytes) 000990 1787575603.949821 write 0x030823 thru 0x032156 (0x1934 bytes) 000991 1787575603.956798 read 0x02e1d4 thru 0x032156 (0x3f83 bytes) 000993 1787575603.958531 mapread 0x00a806 thru 0x00aa00 (0x1fb bytes) 000994 1787575603.976264 write 0x0226cf thru 0x02f9e3 (0xd315 bytes) 000997 1787575603.979399 mapread 0x01a27c thru 0x026dfc (0xcb81 bytes) 000998 1787575603.991765 trunc from 0x032157 to 0x01294e (0x1f808 bytes) 001000 1787575604.037414 mapread 0x00f1b2 thru 0x011648 (0x2497 bytes) All operations completed A-OK! PASS 70a (99s) == sanity-flr test 71: check mirror extend parent fid ==== 08:46:53 (1787575613) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 08:46:57 (1787575617) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.318051 s, 6.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447974 s, 2.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 08:47:08 (1787575628) Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1091232 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3075136 .. failed Extending file size to 2248352 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4808864 .. Extending file size to 4897536 .. Extending file size to 7724064 .. Extending file size to 6069280 .. Extending file size to 6418336 .. Extending file size to 7144160 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8913472 .. Extending file size to 8449792 .. Extending file size to 8638656 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8639200 .. Extending file size to 8763584 .. Extending file size to 9138496 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57269 57271 57272 57273 57274 done 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) 192.168.203.118@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:48:34 (1787575714) Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6320512 .. Extending file size to 7660768 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7397056 .. Extending file size to 8900416 .. Extending file size to 8165696 .. Extending file size to 9118016 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8763808 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9383648 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58695 58696 58699 58701 58702 done 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) 192.168.203.118@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:49:56 (1787575796) Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13818 s, 15.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.406158 s, 5.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x63ec4854 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:50:15 (1787575815) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142229 s, 14.7 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 08:50:23 (1787575823) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:50:31 (1787575831) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.388549 s, 5.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:50:40 (1787575840) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.286029 s, 7.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:50:50 (1787575850) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.437439 s, 4.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:51:27 (1787575887) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg318-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126861 s, 8.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:52:32 (1787575952) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg318-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.40902 s, 5.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (69s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:53:41 (1787576021) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg318-server mds-ost sync done. Creating new pool test_204f oleg318-server: Pool lustre.test_204f created Adding targets to pool oleg318-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg318-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.494988 s, 4.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg318-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg318-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg318-server: Pool lustre.test_204f destroyed PASS 204f (85s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:55:06 (1787576106) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:55:15 (1787576115) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:55:22 (1787576122) oleg318-server: Pool lustre.test_206 created Adding targets to pool oleg318-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg318-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg318-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg318-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg318-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:55:48 (1787576148) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597195 s, 17.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666873 s, 15.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe4f5e7d4 Mirror 3: 0xe4f5e7d4 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:56:00 (1787576160) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:56:03 (1787576163) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:56:08 (1787576168) Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.143784 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ba0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x12e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x12e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x12e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x18c0000 Read this (4 bytes): BBBB 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:56:47 (1787576207) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0344366 s, 238 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:56:54 (1787576214) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582165 s, 18.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1da lmm_fid: [0x200000404:0x1da:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1da lmm_fid: [0x200000404:0x1da:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834394 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 08:56:56.000000000 -0400 Modify: 2026-08-24 08:56:56.000000000 -0400 Change: 2026-08-24 08:56:58.000000000 -0400 Birth: 2026-08-24 08:56:58.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:57:03 (1787576223) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:57:06 (1787576226) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.508542 s, 80.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0804265 s, 50.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 08:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1dc:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 08:57 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:57:16 (1787576236) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:57:24 (1787576244) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1864 sec =========== 08:57:34 (1787576254) === sanity-flr: start cleanup 08:57:36 (1787576256) === === sanity-flr: finish cleanup 08:57:42 (1787576262) ===