-----============= acceptance-small: sanity-flr ============----- Tue Sep 29 00:48:45 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg247-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 00:49:10 (1790657350) === oleg247-client.virtnet: executing check_config_client /mnt/lustre oleg247-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg247-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e40d8c84000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e40d8c84000.idle_timeout=debug disable quota as required oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 00:49:55 (1790657395) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7624] == sanity-flr test 0a: lfs mirror create with -N option == 00:49:58 (1790657398) PASS 0a (23s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 00:50:21 (1790657421) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 00:50:24 (1790657424) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 00:50:28 (1790657428) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 00:50:55 (1790657455) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 00:50:59 (1790657459) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 00:51:03 (1790657463) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8e40d8c84000.stats=clear osc.lustre-OST0001-osc-ffff8e40d8c84000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 00:51:11 (1790657471) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (15s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 00:51:26 (1790657486) PASS 0j (12s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 00:51:38 (1790657498) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 00:51:42 (1790657502) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 00:51:52 (1790657512) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (14s) == sanity-flr test 2: create components from existing files ========================================================== 00:52:07 (1790657527) PASS 2 (10s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 00:52:16 (1790657536) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 00:52:21 (1790657541) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 00:52:31 (1790657551) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 00:52:45 (1790657565) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 00:52:53 (1790657573) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.83431 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13029 s, 8.0 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 00:53:08 (1790657588) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.247474 s, 38.1 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 00:53:19 (1790657599) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (16s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 00:53:35 (1790657615) Stopping /mnt/lustre-ost1 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e40d8c84000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e40d8c84000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg247-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e40d8c84000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e40d8c84000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg247-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (101s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 00:55:16 (1790657716) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.391644 s, 18.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e40d8c84000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e40d8c84000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg247-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (55s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 00:56:11 (1790657771) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e40d8c84000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e40d8c84000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg247-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e40d8c84000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e40d8c84000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg247-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (100s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:57:51 (1790657871) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.86569 s, 10.8 MB/s -rw-r--r-- 1 root root 52428800 Sep 29 00:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.632s user 0m0.001s sys 0m0.180s Stopping /mnt/lustre-ost1 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e40d8c84000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e40d8c84000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.476s user 0m0.000s sys 0m0.207s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg247-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg247-server oleg247-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e40d8c84000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e40d8c84000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.740s user 0m0.001s sys 0m0.200s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg247-server: oleg247-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg247-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg247-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e40d8c84000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e40d8c84000.ost_server_uuid in FULL state after 0 sec PASS 33b (111s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:59:42 (1790657982) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:59:46 (1790657986) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:59:50 (1790657990) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 00:59:54 (1790657994) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 01:00:01 (1790658001) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.567583 s, 22.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:00:15 (1790658015) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512476 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368511 s, 28.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:00:24 (1790658024) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:00:32 (1790658032) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000777372 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114594 s, 894 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000725663 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00120361 s, 851 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 01:00:45 (1790658045) create test files with size 15728983 .. 1b17ff3e6fa2568443d68d2e3bfd1a21 - 85cd91ff561b7cfdf3c1000cfe0e4747 - 43a1271c4d436c57d7db79a0e87e8fb3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg247-client.virtnet /mnt/lustre (opts:) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 01:01:12 (1790658072) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:01:27 (1790658087) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:01:33 (1790658093) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232333 s, 4.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057749 s, 18.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 01:01:45 (1790658105) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161934 s, 13.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.34634 s, 12.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.222057 s, 9.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 01:02:04 (1790658124) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 01:02:08 (1790658128) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:02:11 (1790658131) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375869 s, 27.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 01:02:19 (1790658139) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:02:22 (1790658142) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:02:35 (1790658155) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.338363 s, 31.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg247-client.virtnet /mnt/lustre (opts:) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:02:46 (1790658166) llite.lustre-ffff8e40c92f1800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.393177 s, 26.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg247-client.virtnet /mnt/lustre (opts:) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8e40d8804000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 01:03:03 (1790658183) error lfs split: create victim file failed: File exists PASS 44f (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:03:23 (1790658203) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:04:03 (1790658243) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:04:41 (1790658281) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 01:04:44 (1790658284) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.202328 s, 15.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.148575 s, 21.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x19d8146a (nosync,stale) Mirror 2: 0x19d8146a (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6ddff58ea0a87a234a0dbaef1d1d2d94 - mirror 1 checksum: 6585dd42d2c59bc4ce2a0061b583c52d - mirror 2 checksum: 6ddff58ea0a87a234a0dbaef1d1d2d94 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-29T01:04:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x102:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-29T01:04:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x103:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6ddff58ea0a87a234a0dbaef1d1d2d94 - mirror 1 checksum: 6585dd42d2c59bc4ce2a0061b583c52d - mirror 2 checksum: 6585dd42d2c59bc4ce2a0061b583c52d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-29T01:04:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x102:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-29T01:04:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x103:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:04:57 (1790658297) Failing mds1 on oleg247-server Stopping /mnt/lustre-mds1 (opts:) on oleg247-server 01:05:12 (1790658312) shut down facet: mds1 facet_host: oleg247-server facet_failover_host: oleg247-server Failover mds1 to oleg247-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 01:05:44 (1790658344) targets are mounted 01:05:44 (1790658344) facet_failover done oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619805 s, 16.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x69 lmm_fid: [0x200000404:0x69:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x104:0x0] } PASS 50A (77s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:06:15 (1790658375) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:06:19 (1790658379) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:06:23 (1790658383) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00700611 s, 2.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:06:31 (1790658391) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:06:34 (1790658394) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0106062 s, 386 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0728902 s, 56.2 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:06:43 (1790658403) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:07:06 (1790658426) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:07:37 (1790658457) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa7:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:08:02 (1790658482) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:08:29 (1790658509) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 01:08:37 (1790658517) mirror create pid 55028 mirror split pid 55029 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 55028 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 55029 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 01:09:46 (1790658586) Seed set to 8589 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1790658589.520205 write 0x023e1c thru 0x0253df (0x15c4 bytes) 000006 1790658589.524205 mapread 0x017f59 thru 0x01820a (0x2b2 bytes) 000007 1790658589.540609 punch from 0x01d607 to 0x024085 (0x6a7f bytes) 000009 1790658589.554511 mapwrite 0x02efa2 thru 0x037196 (0x81f5 bytes) 000010 1790658589.702112 punch from 0x01db56 to 0x020c03 (0x30ae bytes) 000012 1790658589.715165 mapwrite 0x00eaec thru 0x01ca6e (0xdf83 bytes) 000013 1790658589.827321 read 0x019b1f thru 0x02704d (0xd52f bytes) 000014 1790658589.843245 read 0x000744 thru 0x00e329 (0xdbe6 bytes) 000015 1790658589.853140 read 0x0175ad thru 0x01b0a5 (0x3af9 bytes) 000016 1790658589.853756 punch from 0x002b3a to 0x00ad8c (0x8253 bytes) 000017 1790658589.866264 punch from 0x00691f to 0x009455 (0x2b37 bytes) 000020 1790658589.886835 punch from 0x021029 to 0x02536e (0x4346 bytes) 000022 1790658589.894201 mapwrite 0x02693b thru 0x02dacd (0x7193 bytes) 000030 1790658589.978049 mapread 0x034b6f thru 0x037196 (0x2628 bytes) 000031 1790658589.979696 mapwrite 0x0050b9 thru 0x00ba64 (0x69ac bytes) truncating to largest ever: 0x2208a 000033 1790658590.061265 trunc from 0x037197 to 0x022089 (0x1510d bytes) 000035 1790658590.091133 mapwrite 0x038ab9 thru 0x03add8 (0x2320 bytes) 000036 1790658590.235270 mapwrite 0x03a1f0 thru 0x03ffff (0x5e10 bytes) 000037 1790658590.313549 write 0x0063d9 thru 0x012664 (0xc28c bytes) 000038 1790658590.320527 MIRROR_EXTEND to 2 mirrors 000039 1790658591.567416 read 0x0030d8 thru 0x0031ba (0xe3 bytes) 000041 1790658591.577131 mapread 0x017f41 thru 0x0206a3 (0x8763 bytes) 000042 1790658591.603784 punch from 0x03b56d to 0x03ffff (0x4a93 bytes) 000043 1790658591.673156 MIRROR_RESYNC 2 mirrors 000044 1790658592.682547 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000045 1790658592.867877 punch from 0x005a46 to 0x008dbc (0x3377 bytes) 000046 1790658592.893798 mapwrite 0x035347 thru 0x03a404 (0x50be bytes) 000047 1790658593.019974 read 0x01f59b thru 0x023b5f (0x45c5 bytes) truncating to largest ever: 0x3aa59 000048 1790658593.035828 trunc from 0x040000 to 0x03aa58 (0x55a7 bytes) 000049 1790658593.077842 read 0x02c908 thru 0x02dcef (0x13e8 bytes) 000051 1790658593.094558 mapread 0x032f36 thru 0x03aa58 (0x7b23 bytes) 000052 1790658593.095558 mapread 0x01fec6 thru 0x028ae4 (0x8c1f bytes) 000053 1790658593.096402 punch from 0x002156 to 0x011569 (0xf414 bytes) 000055 1790658593.103811 punch from 0x003366 to 0x008a62 (0x56fd bytes) 000056 1790658593.108845 mapwrite 0x01f8d0 thru 0x02ed25 (0xf456 bytes) 000057 1790658593.218717 read 0x01bb25 thru 0x0291e2 (0xd6be bytes) 000058 1790658593.220232 write 0x028771 thru 0x02c748 (0x3fd8 bytes) 000059 1790658593.222147 write 0x026436 thru 0x032c9f (0xc86a bytes) 000061 1790658593.226221 read 0x006b93 thru 0x01443e (0xd8ac bytes) 000063 1790658593.239237 write 0x011fb8 thru 0x012fd7 (0x1020 bytes) 000064 1790658593.240176 mapwrite 0x0333a5 thru 0x03c157 (0x8db3 bytes) 000067 1790658593.336342 trunc from 0x03c158 to 0x0368b1 (0x58a6 bytes) 000073 1790658593.358483 trunc from 0x0368b2 to 0x03454c (0x2365 bytes) 000077 1790658593.381802 write 0x007845 thru 0x0095bf (0x1d7b bytes) 000079 1790658593.384504 trunc from 0x03454d to 0x004d35 (0x2f817 bytes) 000082 1790658593.410281 trunc from 0x004d36 to 0x0383c3 (0x3368e bytes) 000086 1790658593.433571 mapwrite 0x0039fc thru 0x0070b7 (0x36bc bytes) 000088 1790658593.516243 write 0x036a8d thru 0x039ba1 (0x3115 bytes) 000090 1790658593.522745 read 0x0371c0 thru 0x038044 (0xe85 bytes) 000091 1790658593.523312 punch from 0x005213 to 0x00ff3b (0xad29 bytes) 000093 1790658593.541433 trunc from 0x039ba2 to 0x0223c3 (0x177de bytes) 000095 1790658593.560540 write 0x035ff2 thru 0x03a523 (0x4532 bytes) 000097 1790658593.563299 read 0x030ee3 thru 0x0383a7 (0x74c5 bytes) 000099 1790658593.577061 read 0x00b1fe thru 0x00dc30 (0x2a33 bytes) 000100 1790658593.577617 read 0x02443f thru 0x034119 (0xfcdb bytes) 000103 1790658593.580551 write 0x037aeb thru 0x03fad1 (0x7fe7 bytes) 000104 1790658593.582297 mapwrite 0x028c8e thru 0x02b824 (0x2b97 bytes) 000106 1790658593.661009 mapread 0x0245f7 thru 0x0295d0 (0x4fda bytes) 000107 1790658593.661877 mapwrite 0x0248b1 thru 0x031d14 (0xd464 bytes) 000108 1790658593.722525 mapread 0x00cebf thru 0x01a356 (0xd498 bytes) 000110 1790658593.725278 read 0x01f282 thru 0x02a345 (0xb0c4 bytes) 000111 1790658593.726189 write 0x03381f thru 0x03ffff (0xc7e1 bytes) 000112 1790658593.727649 mapwrite 0x018c2b thru 0x023b6e (0xaf44 bytes) 000114 1790658593.784082 mapread 0x00bdf0 thru 0x018617 (0xc828 bytes) 000115 1790658593.786688 mapread 0x03c2cf thru 0x03ffff (0x3d31 bytes) 000116 1790658593.788803 read 0x0072ad thru 0x0160f0 (0xee44 bytes) 000117 1790658593.789456 mapread 0x00dbb8 thru 0x012504 (0x494d bytes) 000120 1790658593.794313 read 0x00d1c6 thru 0x00f1d0 (0x200b bytes) 000121 1790658593.795012 punch from 0x018e4a to 0x027d8e (0xef45 bytes) 000122 1790658593.804513 mapread 0x029c00 thru 0x037f40 (0xe341 bytes) 000123 1790658593.806001 mapread 0x00d2f6 thru 0x00f2e9 (0x1ff4 bytes) 000124 1790658593.807370 trunc from 0x040000 to 0x00bfb9 (0x34046 bytes) 000127 1790658593.837321 mapread 0x004b47 thru 0x00bfb9 (0x7473 bytes) 000129 1790658593.839279 punch from 0x00141f to 0x007b28 (0x670a bytes) 000130 1790658593.846742 write 0x01c6ff thru 0x024f1a (0x881c bytes) 000131 1790658593.848069 punch from 0x0207ee to 0x024f1a (0x472d bytes) 000132 1790658593.865104 punch from 0x01abdc to 0x021223 (0x6648 bytes) 000133 1790658593.870386 mapwrite 0x024faf thru 0x030852 (0xb8a4 bytes) 000134 1790658593.960378 read 0x028976 thru 0x030852 (0x7edd bytes) 000135 1790658593.966322 MIRROR_EXTEND to 2 mirrors 000136 1790658594.984273 punch from 0x01c4d5 to 0x02b35c (0xee88 bytes) 000138 1790658595.079881 MIRROR_RESYNC 2 mirrors 000139 1790658596.192558 punch from 0x01c5f2 to 0x01cafc (0x50b bytes) 000140 1790658596.287075 read 0x0249ea thru 0x02b7a7 (0x6dbe bytes) 000142 1790658596.296784 mapread 0x01c8f4 thru 0x02631e (0x9a2b bytes) 000143 1790658596.319067 read 0x0212a4 thru 0x0272f4 (0x6051 bytes) 000144 1790658596.320222 read 0x02b6b3 thru 0x030852 (0x51a0 bytes) 000145 1790658596.333447 write 0x02d2d8 thru 0x03227b (0x4fa4 bytes) 000149 1790658596.342881 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000154 1790658596.558435 punch from 0x018115 to 0x019632 (0x151e bytes) 000156 1790658596.572377 read 0x002e06 thru 0x009208 (0x6403 bytes) 000157 1790658596.581869 mapwrite 0x033484 thru 0x0381cc (0x4d49 bytes) 000161 1790658596.688748 punch from 0x01309e to 0x019c78 (0x6bdb bytes) 000163 1790658596.695488 punch from 0x0248cb to 0x031a24 (0xd15a bytes) 000164 1790658596.701499 write 0x02e8be thru 0x02fa96 (0x11d9 bytes) truncating to largest ever: 0x3b286 000165 1790658596.711850 trunc from 0x0381cd to 0x03b285 (0x30b9 bytes) 000166 1790658596.730560 punch from 0x026f6b to 0x033512 (0xc5a8 bytes) 000168 1790658596.741513 punch from 0x00101f to 0x00645c (0x543e bytes) 000169 1790658596.748371 mapread 0x0365c7 thru 0x03b285 (0x4cbf bytes) 000171 1790658596.759726 mapread 0x014919 thru 0x0165ec (0x1cd4 bytes) 000172 1790658596.778186 punch from 0x02db99 to 0x03b285 (0xd6ed bytes) 000174 1790658596.791274 write 0x0305a1 thru 0x035d6e (0x57ce bytes) 000175 1790658596.807547 MIRROR_EXTEND to 2 mirrors 000177 1790658598.129588 read 0x01341f thru 0x01dd3e (0xa920 bytes) 000178 1790658598.143350 punch from 0x00cd7b to 0x00fb79 (0x2dff bytes) 000180 1790658598.230344 punch from 0x03b1d6 to 0x03b285 (0xb0 bytes) 000181 1790658598.236270 punch from 0x00b27f to 0x0123f7 (0x7179 bytes) 000182 1790658598.242213 write 0x00ee2e thru 0x015873 (0x6a46 bytes) 000185 1790658598.253976 punch from 0x000fb9 to 0x0048f5 (0x393d bytes) 000186 1790658598.279299 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b286): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b286): Mirror 2: 0x3303da76 (stale) CRC-32 checksum value for chunk [0, 0x3b286): Mirror 1: 0xdcb3fba1 Mirror 3: 0xdcb3fba1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b286: skipped 000187 1790658599.772211 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000189 1790658600.169453 trunc from 0x03b286 to 0x001298 (0x39fed bytes) 000190 1790658600.274045 write 0x00527e thru 0x00dc98 (0x8a1b bytes) 000193 1790658600.278090 punch from 0x00bbcc to 0x00dc98 (0x20cd bytes) 000194 1790658600.295122 write 0x02a17e thru 0x02a83b (0x6be bytes) 000195 1790658600.296374 trunc from 0x02a83c to 0x01b4ed (0xf34e bytes) 000196 1790658600.322629 mapwrite 0x01bfdf thru 0x02785e (0xb880 bytes) 000197 1790658600.437748 mapread 0x021b90 thru 0x02785e (0x5ccf bytes) 000199 1790658600.439175 read 0x017fe6 thru 0x026a82 (0xea9d bytes) 000200 1790658600.439766 mapwrite 0x00717b thru 0x00c724 (0x55aa bytes) 000201 1790658600.516828 mapread 0x026cd8 thru 0x02785e (0xb87 bytes) 000202 1790658600.518131 mapread 0x024b36 thru 0x02667b (0x1b46 bytes) 000203 1790658600.519452 trunc from 0x02785f to 0x034ddd (0xd57f bytes) 000204 1790658600.540503 punch from 0x00b869 to 0x01a8e0 (0xf078 bytes) 000205 1790658600.546130 trunc from 0x034dde to 0x0219a2 (0x1343b bytes) 000208 1790658600.565163 mapwrite 0x00cefc thru 0x013b7b (0x6c80 bytes) 000209 1790658600.616972 trunc from 0x0219a3 to 0x0336fe (0x11d5c bytes) 000210 1790658600.633495 write 0x0376a3 thru 0x03ffff (0x895d bytes) 000211 1790658600.639899 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6bca01cc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8b4cdfc3 Mirror 4: 0x8b4cdfc3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000214 1790658601.852705 write 0x015243 thru 0x01842d (0x31eb bytes) 000215 1790658601.971390 mapwrite 0x004c33 thru 0x0107ac (0xbb7a bytes) 000216 1790658602.046884 mapwrite 0x0175c4 thru 0x023b31 (0xc56e bytes) 000217 1790658602.117045 mapwrite 0x03f7f6 thru 0x03ffff (0x80a bytes) 000218 1790658602.167432 mapwrite 0x01e03f thru 0x021cb9 (0x3c7b bytes) 000219 1790658602.207460 punch from 0x01061a to 0x01ebc4 (0xe5ab bytes) 000221 1790658602.214255 trunc from 0x040000 to 0x0048d5 (0x3b72a bytes) 000222 1790658602.246127 punch from 0x00262c to 0x0048d5 (0x22aa bytes) 000223 1790658602.251470 read 0x00064c thru 0x0048d5 (0x428a bytes) 000224 1790658602.258503 read 0x00429b thru 0x0048d5 (0x63b bytes) 000225 1790658602.259091 read 0x0037a2 thru 0x0048d5 (0x1134 bytes) 000227 1790658602.260069 mapread 0x000e56 thru 0x00137d (0x528 bytes) 000228 1790658602.262289 read 0x002079 thru 0x00330a (0x1292 bytes) 000230 1790658602.263422 trunc from 0x0048d6 to 0x00300c (0x18c9 bytes) 000231 1790658602.285318 read 0x002896 thru 0x00300c (0x777 bytes) 000232 1790658602.285890 punch from 0x0013bc to 0x00300c (0x1c51 bytes) 000234 1790658602.292656 mapwrite 0x008fd2 thru 0x0126a2 (0x96d1 bytes) 000238 1790658602.406677 mapread 0x0032fe thru 0x0125c7 (0xf2ca bytes) 000239 1790658602.408613 read 0x00c245 thru 0x0126a2 (0x645e bytes) 000240 1790658602.410656 read 0x00746f thru 0x00ec5e (0x77f0 bytes) 000241 1790658602.419171 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x126a3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x126a3): Mirror 3: 0x5ca5645d (stale) Mirror 4: 0x67bff141 (stale) CRC-32 checksum value for chunk [0, 0x126a3): Mirror 1: 0xed293ae3 Mirror 5: 0xed293ae3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x126a3: skipped 000242 1790658603.301124 mapwrite 0x039129 thru 0x039647 (0x51f bytes) 000243 1790658603.508076 read 0x002e04 thru 0x00ead3 (0xbcd0 bytes) 000244 1790658603.519031 write 0x008abe thru 0x01189f (0x8de2 bytes) 000245 1790658603.527639 mapread 0x02dc8c thru 0x03603a (0x83af bytes) 000248 1790658603.544331 write 0x025e2a thru 0x033678 (0xd84f bytes) truncating to largest ever: 0x3c930 000249 1790658603.552260 trunc from 0x039648 to 0x03c92f (0x32e8 bytes) 000251 1790658603.578267 punch from 0x02084c to 0x02e40d (0xdbc2 bytes) 000252 1790658603.598456 MIRROR_RESYNC 4 mirrors 000253 1790658606.036583 read 0x00352b thru 0x00a80a (0x72e0 bytes) 000254 1790658606.045376 mapwrite 0x03668e thru 0x03ffff (0x9972 bytes) 000256 1790658606.285217 mapwrite 0x006eb6 thru 0x00d7d2 (0x691d bytes) 000257 1790658606.397664 trunc from 0x040000 to 0x028151 (0x17eae bytes) 000258 1790658606.437715 mapread 0x00182a thru 0x006fba (0x5791 bytes) 000259 1790658606.438649 punch from 0x00b119 to 0x00cb91 (0x1a79 bytes) 000260 1790658606.447708 mapwrite 0x033d0e thru 0x038678 (0x496b bytes) 000261 1790658606.598965 mapwrite 0x025e06 thru 0x026565 (0x760 bytes) 000262 1790658606.718333 MIRROR_RESYNC 4 mirrors 000263 1790658608.879653 punch from 0x0087c7 to 0x010871 (0x80ab bytes) 000264 1790658608.972100 punch from 0x015b42 to 0x01dbfe (0x80bd bytes) 000265 1790658608.978745 mapwrite 0x028eee thru 0x02b29b (0x23ae bytes) 000267 1790658609.079317 mapread 0x010bf9 thru 0x01e3c8 (0xd7d0 bytes) 000268 1790658609.109743 mapread 0x02191d thru 0x02e655 (0xcd39 bytes) 000269 1790658609.122904 mapwrite 0x015c41 thru 0x020215 (0xa5d5 bytes) 000270 1790658609.214003 read 0x02dfcd thru 0x03054b (0x257f bytes) 000271 1790658609.223970 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38679): Mirror 4: 0x634843af (stale) Mirror 5: 0x634843af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38679): Mirror 4: 0x634843af (stale) Mirror 5: 0x634843af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38679: skipped 000273 1790658609.794255 read 0x0088f1 thru 0x013478 (0xab88 bytes) 000274 1790658609.804435 trunc from 0x038679 to 0x01a1f2 (0x1e486 bytes) 000275 1790658609.827172 mapwrite 0x034e42 thru 0x03e142 (0x9301 bytes) 000276 1790658609.968112 mapwrite 0x01cb67 thru 0x020ab7 (0x3f51 bytes) 000278 1790658610.079442 mapread 0x0031f9 thru 0x00fdd7 (0xcbdf bytes) 000279 1790658610.093586 trunc from 0x03e143 to 0x0001e8 (0x3df5a bytes) 000280 1790658610.122640 write 0x032fda thru 0x035768 (0x278f bytes) 000281 1790658610.130488 punch from 0x01ab53 to 0x024e95 (0xa343 bytes) 000282 1790658610.175465 mapread 0x003b9a thru 0x01051a (0xc981 bytes) 000283 1790658610.207298 punch from 0x02f26f to 0x035768 (0x64fa bytes) 000284 1790658610.221458 punch from 0x00277c to 0x0107d3 (0xe058 bytes) 000285 1790658610.247869 MIRROR_RESYNC 3 mirrors 000287 1790658612.432225 read 0x000dbf thru 0x00cb8d (0xbdcf bytes) 000288 1790658612.443906 read 0x005bdf thru 0x00d55b (0x797d bytes) 000289 1790658612.453544 read 0x006803 thru 0x010248 (0x9a46 bytes) 000291 1790658612.454899 mapread 0x002769 thru 0x0113f7 (0xec8f bytes) 000293 1790658612.457283 write 0x004462 thru 0x0136eb (0xf28a bytes) 000294 1790658612.635545 mapread 0x017e3f thru 0x01db70 (0x5d32 bytes) 000296 1790658612.637163 mapread 0x0355e8 thru 0x035768 (0x181 bytes) 000297 1790658612.649741 write 0x034f6d thru 0x039411 (0x44a5 bytes) 000299 1790658612.654580 trunc from 0x039412 to 0x02ea01 (0xaa10 bytes) 000300 1790658612.702804 mapwrite 0x0179fe thru 0x02118a (0x978d bytes) 000301 1790658612.795187 mapwrite 0x010167 thru 0x01cbe3 (0xca7d bytes) 000302 1790658612.864340 trunc from 0x02ea02 to 0x0391d0 (0xa7cf bytes) 000303 1790658612.880674 write 0x006134 thru 0x00e834 (0x8701 bytes) 000304 1790658612.885352 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x391d1): Mirror 5: 0x7f8f01c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x391d1): Mirror 5: 0x7f8f01c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x391d1: skipped 000306 1790658613.687393 trunc from 0x0391d1 to 0x025ca5 (0x1352b bytes) 000307 1790658613.713587 write 0x0388bb thru 0x03ffff (0x7745 bytes) 000308 1790658613.721269 punch from 0x01b1b7 to 0x01ca60 (0x18aa bytes) 000309 1790658613.744147 trunc from 0x040000 to 0x01186f (0x2e790 bytes) 000310 1790658613.773375 trunc from 0x011870 to 0x00d369 (0x4506 bytes) 000311 1790658613.820140 trunc from 0x00d36a to 0x02636c (0x19003 bytes) 000312 1790658613.864076 write 0x028665 thru 0x03770b (0xf0a7 bytes) 000313 1790658613.869034 mapwrite 0x03ed58 thru 0x03ffff (0x12a8 bytes) 000314 1790658614.088722 punch from 0x005a76 to 0x00813c (0x26c7 bytes) 000315 1790658614.116415 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000320 1790658614.499108 write 0x023f65 thru 0x02adb6 (0x6e52 bytes) 000321 1790658614.526289 write 0x024679 thru 0x029833 (0x51bb bytes) 000322 1790658614.528317 read 0x02ee43 thru 0x033c1c (0x4dda bytes) 000323 1790658614.546066 punch from 0x0029b8 to 0x0104bf (0xdb08 bytes) 000324 1790658614.567793 mapwrite 0x020502 thru 0x02308f (0x2b8e bytes) 000326 1790658614.786741 mapwrite 0x004a28 thru 0x01152b (0xcb04 bytes) 000328 1790658614.958467 trunc from 0x040000 to 0x0362c2 (0x9d3d bytes) 000329 1790658615.007083 write 0x014cab thru 0x01c9fe (0x7d54 bytes) 000330 1790658615.013496 read 0x00f699 thru 0x010be6 (0x154e bytes) 000332 1790658615.027117 write 0x0073a6 thru 0x0168d0 (0xf52b bytes) 000333 1790658615.029804 write 0x01af55 thru 0x029122 (0xe1ce bytes) 000335 1790658615.044383 MIRROR_EXTEND to 2 mirrors 000336 1790658616.254619 write 0x013ae9 thru 0x023675 (0xfb8d bytes) 000337 1790658616.361087 mapread 0x0073dd thru 0x015393 (0xdfb7 bytes) 000341 1790658616.397119 punch from 0x012d46 to 0x0225a1 (0xf85c bytes) 000342 1790658616.423604 mapread 0x00a15a thru 0x019779 (0xf620 bytes) 000343 1790658616.434286 mapread 0x02b8db thru 0x0362c2 (0xa9e8 bytes) 000345 1790658616.466367 MIRROR_RESYNC 2 mirrors 000346 1790658617.908753 trunc from 0x0362c3 to 0x0087a5 (0x2db1d bytes) 000347 1790658617.995235 write 0x028ae7 thru 0x0336f3 (0xac0d bytes) 000348 1790658618.002013 read 0x018312 thru 0x024c24 (0xc913 bytes) 000349 1790658618.011350 mapwrite 0x010671 thru 0x01b40c (0xad9c bytes) 000350 1790658618.095020 punch from 0x031f6d to 0x0336f3 (0x1787 bytes) 000351 1790658618.105230 write 0x01d5c1 thru 0x0260d0 (0x8b10 bytes) 000352 1790658618.119727 mapwrite 0x0281f4 thru 0x02e02b (0x5e38 bytes) 000356 1790658618.181242 mapread 0x017793 thru 0x01bd9a (0x4608 bytes) 000357 1790658618.182870 trunc from 0x0336f4 to 0x027fa6 (0xb74d bytes) 000358 1790658618.218239 trunc from 0x027fa7 to 0x0334fb (0xb555 bytes) 000359 1790658618.237785 read 0x01d03f thru 0x02b513 (0xe4d5 bytes) 000361 1790658618.249536 read 0x0060d4 thru 0x00bb95 (0x5ac2 bytes) 000363 1790658618.255392 punch from 0x006ee7 to 0x00f2e8 (0x8402 bytes) 000365 1790658618.264745 trunc from 0x0334fc to 0x0330c8 (0x433 bytes) 000366 1790658618.301198 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x330c9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x330c9): Mirror 2: 0x24850af8 (stale) CRC-32 checksum value for chunk [0, 0x330c9): Mirror 1: 0x9048f2ad Mirror 3: 0x9048f2ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x330c9: skipped 000368 1790658619.336780 trunc from 0x0330c9 to 0x02e183 (0x4f45 bytes) 000369 1790658619.565113 trunc from 0x02e184 to 0x02618f (0x7ff4 bytes) 000371 1790658619.636971 read 0x01bb36 thru 0x0205ee (0x4ab9 bytes) 000373 1790658619.697256 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26190): Mirror 2: 0xe84a45f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26190): Mirror 2: 0xe84a45f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26190: skipped 000377 1790658620.373279 mapread 0x00dc31 thru 0x0154bc (0x788c bytes) 000378 1790658620.407010 write 0x00f38a thru 0x01b52c (0xc1a3 bytes) 000379 1790658620.416887 read 0x0112fb thru 0x01cdeb (0xbaf1 bytes) 000380 1790658620.427803 write 0x03776e thru 0x03a81d (0x30b0 bytes) 000381 1790658620.429697 punch from 0x0165bf to 0x0226bc (0xc0fe bytes) 000382 1790658620.454189 punch from 0x01bad9 to 0x022c14 (0x713c bytes) 000383 1790658620.460617 punch from 0x02c91e to 0x02d6b5 (0xd98 bytes) 000384 1790658620.469850 write 0x035423 thru 0x0357c6 (0x3a4 bytes) 000385 1790658620.476032 trunc from 0x03a81e to 0x01feb7 (0x1a966 bytes) 000386 1790658620.506064 write 0x03e09e thru 0x03ffff (0x1f62 bytes) 000387 1790658620.511155 write 0x03bbf7 thru 0x03ffff (0x4409 bytes) 000388 1790658620.523335 mapwrite 0x021127 thru 0x026c90 (0x5b6a bytes) 000389 1790658620.609189 trunc from 0x040000 to 0x008ead (0x37152 bytes) 000390 1790658620.634817 trunc from 0x008eae to 0x03987b (0x309ce bytes) 000391 1790658620.671366 punch from 0x0074bb to 0x012db0 (0xb8f6 bytes) 000394 1790658620.687484 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000395 1790658620.901098 mapread 0x017de7 thru 0x0248b1 (0xcacb bytes) 000396 1790658620.969557 trunc from 0x03987c to 0x003de9 (0x35a92 bytes) 000399 1790658621.006712 trunc from 0x003dea to 0x028c84 (0x24e9b bytes) 000400 1790658621.038846 punch from 0x00e5de to 0x01d1f1 (0xec14 bytes) 000401 1790658621.047448 mapread 0x00dcfa thru 0x013514 (0x581b bytes) 000402 1790658621.062630 mapread 0x023de7 thru 0x0243c3 (0x5dd bytes) truncating to largest ever: 0x3cf99 000404 1790658621.074875 trunc from 0x028c85 to 0x03cf98 (0x14314 bytes) 000409 1790658621.109824 trunc from 0x03cf99 to 0x02b829 (0x1176f bytes) 000416 1790658621.155570 mapread 0x023810 thru 0x02b829 (0x801a bytes) 000417 1790658621.166523 mapread 0x003589 thru 0x00bbfd (0x8675 bytes) 000418 1790658621.179293 mapwrite 0x012cde thru 0x020338 (0xd65b bytes) 000419 1790658621.294133 mapwrite 0x02405a thru 0x02d0f1 (0x9098 bytes) 000422 1790658621.415623 punch from 0x019397 to 0x01d26c (0x3ed6 bytes) 000423 1790658621.421459 trunc from 0x02d0f2 to 0x039cf3 (0xcc02 bytes) 000424 1790658621.462570 punch from 0x00a7a8 to 0x014a6b (0xa2c4 bytes) 000426 1790658621.473146 trunc from 0x039cf4 to 0x020bcf (0x19124 bytes) 000427 1790658621.509981 mapwrite 0x03bbff thru 0x03ffff (0x4401 bytes) 000428 1790658621.663886 punch from 0x023206 to 0x02edd8 (0xbbd3 bytes) 000429 1790658621.678766 mapwrite 0x01a0c1 thru 0x01e453 (0x4393 bytes) 000430 1790658621.803348 trunc from 0x040000 to 0x024845 (0x1b7ba bytes) 000431 1790658621.816456 punch from 0x004ca8 to 0x008460 (0x37b9 bytes) 000434 1790658621.824434 mapwrite 0x02e6aa thru 0x0348a8 (0x61ff bytes) 000435 1790658621.983859 read 0x0093b8 thru 0x013e68 (0xaab1 bytes) 000436 1790658621.997816 punch from 0x019d9d to 0x022727 (0x898b bytes) 000437 1790658622.004531 trunc from 0x0348a9 to 0x03c3eb (0x7b43 bytes) 000439 1790658622.032538 write 0x03ac1b thru 0x03ffff (0x53e5 bytes) 000442 1790658622.046806 mapwrite 0x0380ab thru 0x038c6b (0xbc1 bytes) 000443 1790658622.158408 write 0x036fc1 thru 0x03ffff (0x903f bytes) 000448 1790658622.170582 mapread 0x01f063 thru 0x022980 (0x391e bytes) 000452 1790658622.186408 punch from 0x032243 to 0x0337d1 (0x158f bytes) 000453 1790658622.240583 punch from 0x010223 to 0x017b49 (0x7927 bytes) 000454 1790658622.249731 write 0x0027d8 thru 0x0091c5 (0x69ee bytes) 000455 1790658622.253090 trunc from 0x040000 to 0x01fef7 (0x20108 bytes) 000456 1790658622.324498 mapwrite 0x033daf thru 0x034927 (0xb79 bytes) 000457 1790658622.575810 mapwrite 0x00301b thru 0x00e5e7 (0xb5cd bytes) 000460 1790658622.693007 mapread 0x020a7e thru 0x02b7b9 (0xad3c bytes) 000461 1790658622.708812 mapread 0x031470 thru 0x034927 (0x34b8 bytes) 000462 1790658622.709900 punch from 0x027af7 to 0x034927 (0xce31 bytes) 000464 1790658622.717240 punch from 0x0042a1 to 0x00b3f2 (0x7152 bytes) 000466 1790658622.724132 write 0x004903 thru 0x00d708 (0x8e06 bytes) 000468 1790658622.731079 write 0x029ca9 thru 0x033c2c (0x9f84 bytes) 000469 1790658622.757402 trunc from 0x034928 to 0x036bd7 (0x22b0 bytes) 000471 1790658622.806262 MIRROR_EXTEND to 2 mirrors 000472 1790658624.428346 MIRROR_EXTEND to 3 mirrors 000473 1790658625.966188 write 0x021f0d thru 0x0294e1 (0x75d5 bytes) 000477 1790658626.101779 read 0x01601b thru 0x016d90 (0xd76 bytes) 000478 1790658626.113361 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36bd8): Mirror 3: 0x3af8ef8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36bd8): Mirror 3: 0x3af8ef8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36bd8: skipped 000480 1790658626.456850 MIRROR_RESYNC 2 mirrors 000483 1790658627.397405 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000485 1790658627.563898 write 0x029c90 thru 0x039176 (0xf4e7 bytes) 000486 1790658627.582742 trunc from 0x039177 to 0x03bba3 (0x2a2d bytes) 000487 1790658627.613894 trunc from 0x03bba4 to 0x0361a8 (0x59fb bytes) 000490 1790658627.637857 read 0x007168 thru 0x013a52 (0xc8eb bytes) 000491 1790658627.647824 write 0x034336 thru 0x03720b (0x2ed6 bytes) 000493 1790658627.649309 trunc from 0x03720c to 0x014a7c (0x2278f bytes) 000494 1790658627.674756 mapread 0x009e4a thru 0x00d0c3 (0x327a bytes) 000496 1790658627.677091 punch from 0x00ea97 to 0x014a7c (0x5fe6 bytes) 000498 1790658627.687229 write 0x00f41c thru 0x01a836 (0xb41b bytes) 000499 1790658627.698143 mapread 0x000702 thru 0x006745 (0x6044 bytes) 000500 1790658627.728270 trunc from 0x01a837 to 0x03c2fc (0x21ac6 bytes) 000501 1790658627.750964 mapwrite 0x01f0a2 thru 0x028687 (0x95e6 bytes) 000503 1790658627.820032 read 0x023252 thru 0x02451d (0x12cc bytes) 000504 1790658627.820876 mapwrite 0x03ba99 thru 0x03ffff (0x4567 bytes) 000506 1790658627.909002 trunc from 0x040000 to 0x028e1b (0x171e4 bytes) 000507 1790658627.926956 mapwrite 0x02fc1c thru 0x03c6a8 (0xca8d bytes) 000510 1790658628.089911 mapread 0x0352d0 thru 0x037b00 (0x2831 bytes) 000512 1790658628.092904 mapwrite 0x02a47e thru 0x035190 (0xad13 bytes) 000513 1790658628.177062 read 0x03b198 thru 0x03c6a8 (0x1511 bytes) 000516 1790658628.180743 trunc from 0x03c6a9 to 0x02928d (0x1341b bytes) 000517 1790658628.206517 punch from 0x001a2b to 0x00c6a5 (0xac7b bytes) 000519 1790658628.214044 mapwrite 0x01ba05 thru 0x01baad (0xa9 bytes) 000520 1790658628.269445 mapread 0x028a14 thru 0x02928d (0x87a bytes) 000521 1790658628.282970 mapwrite 0x00ea69 thru 0x0145d4 (0x5b6c bytes) 000524 1790658628.356736 write 0x023d8c thru 0x02ae29 (0x709e bytes) 000528 1790658628.361193 write 0x0012df thru 0x003264 (0x1f86 bytes) 000529 1790658628.362964 trunc from 0x02ae2a to 0x00173f (0x296ea bytes) 000531 1790658628.392323 trunc from 0x001740 to 0x01a22a (0x18aeb bytes) 000532 1790658628.428161 punch from 0x013b37 to 0x01a22a (0x66f4 bytes) 000533 1790658628.464087 MIRROR_EXTEND to 2 mirrors 000535 1790658629.443905 trunc from 0x01a22b to 0x01a148 (0xe2 bytes) 000536 1790658629.544036 mapwrite 0x00db41 thru 0x01bff5 (0xe4b5 bytes) 000537 1790658629.661107 punch from 0x00127d to 0x00bda4 (0xab28 bytes) 000539 1790658629.668029 trunc from 0x01bff6 to 0x01d2ab (0x12b6 bytes) 000540 1790658629.694697 trunc from 0x01d2ac to 0x039839 (0x1c58e bytes) 000541 1790658629.716653 punch from 0x024647 to 0x02a284 (0x5c3e bytes) 000543 1790658629.728678 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3983a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3983a): Mirror 2: 0x980f99e9 (stale) CRC-32 checksum value for chunk [0, 0x3983a): Mirror 1: 0x58567d54 Mirror 3: 0x58567d54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3983a: skipped 000545 1790658630.881353 trunc from 0x03983a to 0x00d119 (0x2c720 bytes) 000547 1790658630.971734 mapread 0x00cd2e thru 0x00d119 (0x3ec bytes) 000551 1790658630.983841 mapread 0x006274 thru 0x00d119 (0x6ea6 bytes) 000552 1790658630.992734 mapread 0x000a46 thru 0x005cde (0x5299 bytes) 000553 1790658631.003751 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xd11a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd11a): Mirror 2: 0xb8c9583c (stale) Mirror 3: 0x7ad6612d (stale) CRC-32 checksum value for chunk [0, 0xd11a): Mirror 1: 0x7ad6612d Mirror 4: 0x7ad6612d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd11a: skipped 000554 1790658631.979135 mapread 0x00c0b7 thru 0x00d119 (0x1063 bytes) 000555 1790658632.013976 punch from 0x00cc61 to 0x00d119 (0x4b9 bytes) 000556 1790658632.102095 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd11a): Mirror 2: 0xb8c9583c (stale) Mirror 3: 0x7ad6612d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd11a): Mirror 2: 0xb8c9583c (stale) Mirror 3: 0x7ad6612d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd11a: skipped 000557 1790658632.432353 write 0x0167ee thru 0x01e8fc (0x810f bytes) 000558 1790658632.434797 mapread 0x01034c thru 0x014e26 (0x4adb bytes) 000559 1790658632.456764 write 0x006fa9 thru 0x0094ee (0x2546 bytes) 000560 1790658632.473734 mapwrite 0x03c9dd thru 0x03ffff (0x3623 bytes) 000562 1790658632.587759 write 0x004606 thru 0x00f5ca (0xafc5 bytes) 000563 1790658632.604678 mapwrite 0x01cdc6 thru 0x026305 (0x9540 bytes) 000564 1790658632.731802 write 0x00776c thru 0x013fc3 (0xc858 bytes) 000565 1790658632.732829 mapwrite 0x03f912 thru 0x03ffff (0x6ee bytes) 000566 1790658632.802636 read 0x005adc thru 0x013dff (0xe324 bytes) 000568 1790658632.805108 mapread 0x02449f thru 0x02f84b (0xb3ad bytes) 000569 1790658632.817634 write 0x03ba9c thru 0x03ffff (0x4564 bytes) 000570 1790658632.819077 write 0x0265f8 thru 0x026a22 (0x42b bytes) 000571 1790658632.821169 punch from 0x006480 to 0x0094fe (0x307f bytes) 000573 1790658632.839878 read 0x030c34 thru 0x031c4f (0x101c bytes) 000575 1790658632.851957 write 0x03aeb4 thru 0x03ffff (0x514c bytes) 000576 1790658632.854097 punch from 0x019a13 to 0x01fe02 (0x63f0 bytes) 000577 1790658632.877080 MIRROR_RESYNC 3 mirrors 000578 1790658633.870028 punch from 0x001311 to 0x007bb4 (0x68a4 bytes) 000579 1790658633.936025 read 0x00ea97 thru 0x010f35 (0x249f bytes) 000581 1790658633.944139 punch from 0x001f5a to 0x0045e3 (0x268a bytes) 000582 1790658633.949141 trunc from 0x040000 to 0x02a37a (0x15c85 bytes) 000585 1790658633.977653 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a37b): Mirror 3: 0x21ce6b85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a37b): Mirror 3: 0x21ce6b85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a37b: skipped 000586 1790658634.246081 MIRROR_RESYNC 2 mirrors 000587 1790658634.971836 write 0x02d271 thru 0x03722c (0x9fbc bytes) 000588 1790658635.064909 punch from 0x02cc05 to 0x030f45 (0x4341 bytes) 000590 1790658635.079604 mapwrite 0x01eac2 thru 0x01fb61 (0x10a0 bytes) 000591 1790658635.139390 read 0x015a58 thru 0x01df7e (0x8527 bytes) 000592 1790658635.150988 punch from 0x0361b5 to 0x03722c (0x1078 bytes) 000593 1790658635.156759 trunc from 0x03722d to 0x001291 (0x35f9b bytes) 000594 1790658635.176511 read 0x000704 thru 0x001291 (0xb8e bytes) 000596 1790658635.184833 punch from 0x000e71 to 0x001291 (0x421 bytes) 000597 1790658635.195253 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1292): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1292): Mirror 3: 0xdc052caa (stale) CRC-32 checksum value for chunk [0, 0x1292): Mirror 1: 0xdc052caa Mirror 4: 0xdc052caa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1292: skipped 000598 1790658635.822729 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1292): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1292): Mirror 3: 0xdc052caa (stale) CRC-32 checksum value for chunk [0, 0x1292): Mirror 1: 0xdc052caa Mirror 4: 0xdc052caa Mirror 5: 0xdc052caa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1292: skipped 000603 1790658636.590837 punch from 0x0004e9 to 0x001291 (0xda9 bytes) 000604 1790658636.677285 write 0x00703a thru 0x010363 (0x932a bytes) 000605 1790658636.678652 punch from 0x0010fa to 0x00c136 (0xb03d bytes) 000606 1790658636.694335 mapread 0x00aed8 thru 0x00b082 (0x1ab bytes) 000608 1790658636.710995 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10364): Mirror 4: 0xcbb4d662 (stale) Mirror 5: 0xcbb4d662 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10364): Mirror 4: 0xcbb4d662 (stale) Mirror 5: 0xcbb4d662 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10364: skipped 000609 1790658637.066095 read 0x0047dc thru 0x0063ff (0x1c24 bytes) 000610 1790658637.072397 trunc from 0x010364 to 0x02a682 (0x1a31f bytes) 000612 1790658637.121662 trunc from 0x02a683 to 0x021833 (0x8e4f bytes) 000613 1790658637.144878 read 0x005eec thru 0x00d4ca (0x75df bytes) 000614 1790658637.165093 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21834): Mirror 4: 0xb335cc50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21834): Mirror 4: 0xb335cc50 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21834: skipped 000618 1790658637.666825 mapwrite 0x00e6f6 thru 0x01df2e (0xf839 bytes) 000620 1790658637.859650 punch from 0x01db3c to 0x021833 (0x3cf8 bytes) 000623 1790658637.870460 punch from 0x013b3b to 0x02075c (0xcc22 bytes) 000624 1790658637.878168 mapwrite 0x02d22b thru 0x0340a7 (0x6e7d bytes) 000625 1790658638.004831 mapread 0x00f504 thru 0x01c334 (0xce31 bytes) 000626 1790658638.023697 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000628 1790658638.168119 punch from 0x010e0c to 0x0125e4 (0x17d9 bytes) 000629 1790658638.176350 punch from 0x0022c5 to 0x0069a6 (0x46e2 bytes) 000630 1790658638.181891 trunc from 0x0340a8 to 0x021698 (0x12a0f bytes) 000631 1790658638.197656 mapread 0x007634 thru 0x007ffc (0x9c9 bytes) 000634 1790658638.204516 mapwrite 0x02e5fe thru 0x039905 (0xb308 bytes) 000635 1790658638.292963 punch from 0x026098 to 0x02def8 (0x7e61 bytes) 000636 1790658638.298756 mapread 0x027279 thru 0x029f60 (0x2ce8 bytes) 000639 1790658638.308535 read 0x032251 thru 0x039905 (0x76b5 bytes) 000640 1790658638.309128 write 0x032343 thru 0x03da7a (0xb738 bytes) 000642 1790658638.311656 trunc from 0x03da7b to 0x007f8c (0x35aee bytes) 000643 1790658638.333022 read 0x001b26 thru 0x007f8c (0x6467 bytes) 000644 1790658638.341785 read 0x005da1 thru 0x007f8c (0x21ec bytes) 000645 1790658638.342888 read 0x00284e thru 0x007f8c (0x573f bytes) 000647 1790658638.345620 write 0x0006a4 thru 0x00fbd4 (0xf531 bytes) 000650 1790658638.349596 read 0x009327 thru 0x0099c0 (0x69a bytes) 000652 1790658638.350808 mapread 0x00ed6c thru 0x00fbd4 (0xe69 bytes) 000653 1790658638.352218 mapwrite 0x0105f9 thru 0x019019 (0x8a21 bytes) 000655 1790658638.453448 write 0x01aefe thru 0x01e0bd (0x31c0 bytes) 000658 1790658638.457397 read 0x00c999 thru 0x01bfdd (0xf645 bytes) 000659 1790658638.462732 MIRROR_EXTEND to 2 mirrors 000660 1790658639.158046 mapread 0x018f91 thru 0x01e0bd (0x512d bytes) 000661 1790658639.179545 trunc from 0x01e0be to 0x02f30b (0x1124e bytes) 000663 1790658639.298111 mapwrite 0x039b03 thru 0x03cf62 (0x3460 bytes) 000664 1790658639.370413 write 0x014db0 thru 0x022f0a (0xe15b bytes) 000665 1790658639.382044 read 0x01036e thru 0x01e090 (0xdd23 bytes) 000666 1790658639.390146 read 0x016aa4 thru 0x01a368 (0x38c5 bytes) 000667 1790658639.390977 punch from 0x03834a to 0x03cf62 (0x4c19 bytes) 000668 1790658639.412551 MIRROR_RESYNC 2 mirrors 000669 1790658640.834453 mapread 0x035dd4 thru 0x03cf62 (0x718f bytes) 000670 1790658640.860362 MIRROR_RESYNC 2 mirrors 000671 1790658641.239139 write 0x0160d0 thru 0x018a6b (0x299c bytes) 000672 1790658641.400398 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000673 1790658641.616379 read 0x0320a3 thru 0x03cf62 (0xaec0 bytes) 000674 1790658641.628580 mapwrite 0x00270b thru 0x005811 (0x3107 bytes) 000675 1790658641.699435 mapread 0x02a52b thru 0x02ae07 (0x8dd bytes) 000677 1790658641.710842 read 0x01ce3f thru 0x02cb58 (0xfd1a bytes) 000679 1790658641.722947 mapread 0x012d56 thru 0x0145f8 (0x18a3 bytes) 000680 1790658641.724676 trunc from 0x03cf63 to 0x02c462 (0x10b00 bytes) 000681 1790658641.741880 mapread 0x001533 thru 0x006414 (0x4ee2 bytes) 000682 1790658641.743057 punch from 0x01ffec to 0x028fb7 (0x8fcc bytes) 000683 1790658641.750016 read 0x010c1a thru 0x010e08 (0x1ef bytes) 000685 1790658641.751648 mapread 0x01ac8e thru 0x0269cd (0xbd40 bytes) 000686 1790658641.764582 punch from 0x02b784 to 0x02c462 (0xcdf bytes) 000688 1790658641.773821 MIRROR_EXTEND to 2 mirrors 000689 1790658643.100050 trunc from 0x02c463 to 0x02db41 (0x16df bytes) 000690 1790658643.292067 write 0x02fd93 thru 0x035d4a (0x5fb8 bytes) 000691 1790658643.294285 read 0x031b6d thru 0x035d4a (0x41de bytes) 000692 1790658643.295660 punch from 0x0339d7 to 0x035d4a (0x2374 bytes) 000697 1790658643.326443 punch from 0x02fd06 to 0x035d4a (0x6045 bytes) 000698 1790658643.335943 trunc from 0x035d4b to 0x029072 (0xccd8 bytes) 000699 1790658643.370144 read 0x0256d2 thru 0x029072 (0x39a1 bytes) 000702 1790658643.399850 punch from 0x001c4c to 0x010435 (0xe7ea bytes) 000703 1790658643.409840 mapwrite 0x00ab10 thru 0x010cd6 (0x61c7 bytes) 000704 1790658643.616507 read 0x0120d6 thru 0x01292b (0x856 bytes) 000707 1790658643.617833 mapread 0x00e844 thru 0x0109ce (0x218b bytes) 000710 1790658643.620038 mapread 0x0187c9 thru 0x0194f3 (0xd2b bytes) 000711 1790658643.621089 punch from 0x015a1d to 0x01f6aa (0x9c8e bytes) 000715 1790658643.639438 punch from 0x025d8c to 0x029072 (0x32e7 bytes) 000716 1790658643.652413 mapwrite 0x00042b thru 0x002eeb (0x2ac1 bytes) 000717 1790658643.806548 MIRROR_RESYNC 2 mirrors 000718 1790658645.146250 punch from 0x004342 to 0x0064df (0x219e bytes) 000722 1790658645.266067 mapread 0x004276 thru 0x00f58a (0xb315 bytes) 000723 1790658645.288087 write 0x034fae thru 0x03ffff (0xb052 bytes) 000724 1790658645.290755 punch from 0x0388e0 to 0x03a68f (0x1db0 bytes) 000725 1790658645.313269 trunc from 0x040000 to 0x026005 (0x19ffa bytes) 000727 1790658645.345894 mapread 0x014764 thru 0x020812 (0xc0af bytes) 000728 1790658645.374787 write 0x02e59c thru 0x031794 (0x31f9 bytes) 000729 1790658645.377239 punch from 0x0092d6 to 0x012aac (0x97d7 bytes) 000730 1790658645.393949 write 0x01d509 thru 0x02122a (0x3d22 bytes) 000731 1790658645.405007 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3d507 000733 1790658646.803889 trunc from 0x031795 to 0x03d506 (0xbd72 bytes) 000734 1790658646.960382 mapread 0x022eaa thru 0x025027 (0x217e bytes) 000736 1790658647.010429 mapread 0x015ce3 thru 0x01c947 (0x6c65 bytes) 000737 1790658647.037563 trunc from 0x03d507 to 0x032d3b (0xa7cb bytes) 000738 1790658647.096468 mapread 0x01e604 thru 0x026275 (0x7c72 bytes) 000739 1790658647.098900 punch from 0x025c82 to 0x032d3b (0xd0ba bytes) 000740 1790658647.123980 MIRROR_RESYNC 2 mirrors 000743 1790658648.140835 mapread 0x006bb6 thru 0x00f9a8 (0x8df3 bytes) 000745 1790658648.163533 punch from 0x01dc72 to 0x0238f8 (0x5c87 bytes) 000746 1790658648.228238 write 0x001212 thru 0x00fcc3 (0xeab2 bytes) 000748 1790658648.239916 mapread 0x01df67 thru 0x024b8a (0x6c24 bytes) 000749 1790658648.254496 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32d3c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32d3c): Mirror 2: 0x54522e05 (stale) CRC-32 checksum value for chunk [0, 0x32d3c): Mirror 1: 0x5a1bfda3 Mirror 3: 0x5a1bfda3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32d3c: skipped 000750 1790658649.644520 mapread 0x000d98 thru 0x008dbf (0x8028 bytes) 000751 1790658649.709326 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000752 1790658650.208297 punch from 0x012c1e to 0x018201 (0x55e4 bytes) 000753 1790658650.337022 read 0x020833 thru 0x02c7db (0xbfa9 bytes) 000759 1790658650.356899 punch from 0x01747b to 0x01e80a (0x7390 bytes) 000762 1790658650.377201 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32d3c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32d3c): Mirror 3: 0x5a1bfda3 (stale) CRC-32 checksum value for chunk [0, 0x32d3c): Mirror 1: 0xfe25b772 Mirror 4: 0xfe25b772 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32d3c: skipped 000764 1790658651.576168 punch from 0x016b30 to 0x01ad96 (0x4267 bytes) 000765 1790658651.694839 MIRROR_RESYNC 3 mirrors 000766 1790658653.288206 write 0x002248 thru 0x008875 (0x662e bytes) 000768 1790658653.492604 trunc from 0x032d3c to 0x0246a9 (0xe692 bytes) 000769 1790658653.529451 read 0x0117dd thru 0x01bd91 (0xa5b5 bytes) 000770 1790658653.560224 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x246aa): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x246aa): Mirror 3: 0xae7605d0 (stale) Mirror 4: 0xae7605d0 (stale) CRC-32 checksum value for chunk [0, 0x246aa): Mirror 1: 0x6bc81cf8 Mirror 5: 0x6bc81cf8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x246aa: skipped 000771 1790658655.094289 read 0x009da2 thru 0x00f4e7 (0x5746 bytes) 000772 1790658655.109477 mapread 0x014843 thru 0x0190d6 (0x4894 bytes) 000773 1790658655.148863 mapwrite 0x01a9ba thru 0x026ab1 (0xc0f8 bytes) 000777 1790658655.306564 read 0x00ff4a thru 0x01ac41 (0xacf8 bytes) 000779 1790658655.318208 trunc from 0x026ab2 to 0x021fc5 (0x4aec bytes) 000780 1790658655.341992 write 0x02b845 thru 0x02d541 (0x1cfd bytes) 000781 1790658655.343753 read 0x022677 thru 0x0268a4 (0x422e bytes) 000782 1790658655.355220 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d542): Mirror 3: 0xb923001b (stale) Mirror 4: 0xb923001b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d542): Mirror 3: 0xb923001b (stale) Mirror 4: 0xb923001b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d542: skipped 000784 1790658655.670862 read 0x02ba99 thru 0x02d541 (0x1aa9 bytes) 000785 1790658655.684131 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d542): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d542): Mirror 3: 0xb923001b (stale) Mirror 4: 0xb923001b (stale) CRC-32 checksum value for chunk [0, 0x2d542): Mirror 1: 0x8ba2f487 Mirror 5: 0x8ba2f487 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d542: skipped 000786 1790658657.170979 mapwrite 0x00a204 thru 0x00d356 (0x3153 bytes) 000787 1790658657.574002 write 0x0186c2 thru 0x01a499 (0x1dd8 bytes) 000790 1790658657.593665 mapread 0x020057 thru 0x02be9b (0xbe45 bytes) 000791 1790658657.613490 write 0x00ea97 thru 0x012499 (0x3a03 bytes) 000792 1790658657.614500 mapwrite 0x01f405 thru 0x029a20 (0xa61c bytes) 000793 1790658657.679447 mapread 0x019707 thru 0x021fca (0x88c4 bytes) 000795 1790658657.680822 read 0x027c4e thru 0x02d541 (0x58f4 bytes) 000797 1790658657.681865 mapwrite 0x00ab05 thru 0x011390 (0x688c bytes) 000798 1790658657.735595 punch from 0x021751 to 0x02d541 (0xbdf1 bytes) 000799 1790658657.741226 mapwrite 0x03314d thru 0x037daf (0x4c63 bytes) 000800 1790658657.825770 mapwrite 0x0228c1 thru 0x0310f9 (0xe839 bytes) 000803 1790658657.901451 mapwrite 0x0164ec thru 0x020b50 (0xa665 bytes) 000804 1790658657.969917 mapread 0x01a8e7 thru 0x026015 (0xb72f bytes) 000807 1790658657.973749 read 0x010f25 thru 0x0168e3 (0x59bf bytes) 000808 1790658657.974651 mapwrite 0x033bff thru 0x03a666 (0x6a68 bytes) 000809 1790658658.073620 mapwrite 0x023c60 thru 0x02a84f (0x6bf0 bytes) 000810 1790658658.124665 read 0x00d777 thru 0x014976 (0x7200 bytes) 000811 1790658658.125340 punch from 0x01ebd6 to 0x02d86e (0xec99 bytes) 000812 1790658658.131745 write 0x034d7a thru 0x03b5e3 (0x686a bytes) 000813 1790658658.133528 trunc from 0x03b5e4 to 0x0073e3 (0x34200 bytes) 000814 1790658658.162395 MIRROR_RESYNC 4 mirrors 000815 1790658659.184184 read 0x006f92 thru 0x0073e3 (0x452 bytes) 000816 1790658659.195806 MIRROR_RESYNC 4 mirrors 000817 1790658659.726033 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000818 1790658660.095887 trunc from 0x0073e4 to 0x013c17 (0xc834 bytes) 000819 1790658660.209618 trunc from 0x013c18 to 0x0387dd (0x24bc6 bytes) 000820 1790658660.258673 MIRROR_RESYNC 3 mirrors 000823 1790658661.332391 MIRROR_RESYNC 3 mirrors 000824 1790658661.655455 write 0x033005 thru 0x035aec (0x2ae8 bytes) 000825 1790658661.721513 write 0x01e76a thru 0x01eec0 (0x757 bytes) 000827 1790658661.729125 write 0x012aa7 thru 0x01c79c (0x9cf6 bytes) 000829 1790658661.740814 mapread 0x01f473 thru 0x02e3f3 (0xef81 bytes) 000832 1790658661.777977 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x387de): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x387de): Mirror 3: 0x5e3852df (stale) Mirror 5: 0x5e3852df (stale) CRC-32 checksum value for chunk [0, 0x387de): Mirror 1: 0xb0a5b249 Mirror 6: 0xb0a5b249 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x387de: skipped 000833 1790658663.179328 write 0x02be0d thru 0x03658a (0xa77e bytes) 000834 1790658663.331775 trunc from 0x0387de to 0x00faac (0x28d31 bytes) 000837 1790658663.368905 mapread 0x00a8b7 thru 0x00e34f (0x3a99 bytes) 000838 1790658663.400177 read 0x001967 thru 0x00faac (0xe146 bytes) 000841 1790658663.411695 mapread 0x00a731 thru 0x00faac (0x537c bytes) 000843 1790658663.414728 trunc from 0x00faad to 0x0178cb (0x7e1f bytes) 000844 1790658663.440699 mapread 0x0037b9 thru 0x0131dc (0xfa24 bytes) 000847 1790658663.456229 trunc from 0x0178cc to 0x02cdca (0x154ff bytes) 000848 1790658663.482544 mapwrite 0x03919d thru 0x03ffff (0x6e63 bytes) 000849 1790658663.593631 read 0x03771d thru 0x03ffff (0x88e3 bytes) 000850 1790658663.595519 mapwrite 0x007353 thru 0x008623 (0x12d1 bytes) 000851 1790658663.656700 trunc from 0x040000 to 0x0239c1 (0x1c63e bytes) 000852 1790658663.694193 mapwrite 0x00c5c3 thru 0x01100d (0x4a4b bytes) 000853 1790658663.767253 write 0x014947 thru 0x02249b (0xdb55 bytes) 000854 1790658663.777606 read 0x018c3f thru 0x0239c1 (0xad83 bytes) 000855 1790658663.785489 punch from 0x0232a8 to 0x0239c1 (0x71a bytes) 000856 1790658663.807503 MIRROR_RESYNC 4 mirrors 000857 1790658664.941050 trunc from 0x0239c2 to 0x020378 (0x3649 bytes) 000859 1790658665.052259 trunc from 0x020379 to 0x01380d (0xcb6b bytes) 000860 1790658665.086396 write 0x020543 thru 0x0218ae (0x136c bytes) 000861 1790658665.091833 punch from 0x01e148 to 0x02050d (0x23c6 bytes) 000862 1790658665.106198 punch from 0x01a7a1 to 0x01b369 (0xbc9 bytes) 000865 1790658665.112783 punch from 0x002176 to 0x008744 (0x65cf bytes) 000866 1790658665.123095 mapwrite 0x02fb07 thru 0x03a9d8 (0xaed2 bytes) 000868 1790658665.326103 trunc from 0x03a9d9 to 0x00c279 (0x2e75f bytes) 000869 1790658665.356950 mapwrite 0x01ac4f thru 0x01b43a (0x7ec bytes) 000872 1790658665.491396 punch from 0x0113ea to 0x01b43a (0xa051 bytes) 000873 1790658665.503188 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1b43b): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b43b): Mirror 3: 0xe2bc4400 (stale) Mirror 5: 0xe2bc4400 (stale) Mirror 6: 0xe2bc4400 (stale) CRC-32 checksum value for chunk [0, 0x1b43b): Mirror 1: 0xe41e7e78 Mirror 7: 0xe41e7e78 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b43b: skipped 000874 1790658666.688732 punch from 0x019e3d to 0x01b43a (0x15fe bytes) 000875 1790658666.784367 punch from 0x012daa to 0x01344b (0x6a2 bytes) 000876 1790658666.790815 trunc from 0x01b43b to 0x01dfc7 (0x2b8d bytes) 000877 1790658666.819089 MIRROR_RESYNC 5 mirrors 000878 1790658668.613052 trunc from 0x01dfc8 to 0x0033af (0x1ac18 bytes) 000879 1790658668.730799 mapread 0x00308f thru 0x0033af (0x321 bytes) 000881 1790658668.739894 write 0x00ca81 thru 0x019587 (0xcb07 bytes) 000882 1790658668.750565 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19588): Mirror 3: 0x7d52546d (stale) Mirror 5: 0x7d52546d (stale) Mirror 6: 0x7d52546d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19588): Mirror 3: 0x7d52546d (stale) Mirror 5: 0x7d52546d (stale) Mirror 6: 0x7d52546d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19588: skipped 000883 1790658669.090995 punch from 0x00da06 to 0x0175d1 (0x9bcc bytes) 000885 1790658669.105125 MIRROR_RESYNC 4 mirrors 000886 1790658671.446797 MIRROR_EXTEND to 5 mirrors truncating to largest ever: 0x3f1b6 000887 1790658672.375597 trunc from 0x019588 to 0x03f1b5 (0x25c2e bytes) 000888 1790658672.537294 write 0x019229 thru 0x01fa46 (0x681e bytes) 000893 1790658672.565400 write 0x02b5a4 thru 0x033a9e (0x84fb bytes) 000894 1790658672.576731 trunc from 0x03f1b6 to 0x003ecb (0x3b2ea bytes) 000895 1790658672.602064 mapread 0x000890 thru 0x003ecb (0x363c bytes) 000896 1790658672.623159 mapwrite 0x033ab2 thru 0x038e42 (0x5391 bytes) 000897 1790658672.699783 trunc from 0x038e43 to 0x02bca7 (0xd19b bytes) 000899 1790658672.718466 mapwrite 0x014067 thru 0x01a178 (0x6112 bytes) 000901 1790658672.776624 punch from 0x00890a to 0x009e8b (0x1582 bytes) 000902 1790658672.785553 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2bca8): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bca8): Mirror 3: 0x13cafd20 (stale) Mirror 5: 0x13cafd20 (stale) Mirror 6: 0x13cafd20 (stale) Mirror 7: 0x13cafd20 (stale) CRC-32 checksum value for chunk [0, 0x2bca8): Mirror 1: 0xf5390428 Mirror 8: 0xf5390428 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bca8: skipped 000903 1790658674.129158 read 0x003b8f thru 0x008a8c (0x4efe bytes) 000904 1790658674.182455 MIRROR_RESYNC 6 mirrors 000905 1790658676.172619 mapwrite 0x033a1f thru 0x035093 (0x1675 bytes) 000906 1790658676.336065 punch from 0x01f23c to 0x0285f0 (0x93b5 bytes) 000907 1790658676.343101 punch from 0x00df9d to 0x017a99 (0x9afd bytes) 000908 1790658676.348436 mapwrite 0x03f30d thru 0x03ffff (0xcf3 bytes) 000909 1790658676.419249 mapread 0x000349 thru 0x0060f7 (0x5daf bytes) 000910 1790658676.433767 MIRROR_RESYNC 6 mirrors 000912 1790658679.124141 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000913 1790658679.684318 mapwrite 0x01ad4d thru 0x025348 (0xa5fc bytes) 000914 1790658679.931978 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5af2c3e8 (stale) Mirror 5: 0x5af2c3e8 (stale) Mirror 6: 0x5af2c3e8 (stale) Mirror 8: 0x5af2c3e8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x17082b42 Mirror 9: 0x17082b42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000916 1790658681.759165 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5af2c3e8 (stale) Mirror 6: 0x5af2c3e8 (stale) Mirror 8: 0x5af2c3e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5af2c3e8 (stale) Mirror 6: 0x5af2c3e8 (stale) Mirror 8: 0x5af2c3e8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x17082b42 Mirror 9: 0x17082b42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000917 1790658682.640700 read 0x020d5e thru 0x030964 (0xfc07 bytes) 000918 1790658682.653596 write 0x014af2 thru 0x018f62 (0x4471 bytes) 000919 1790658682.813456 trunc from 0x040000 to 0x034554 (0xbaab bytes) 000921 1790658682.834200 mapwrite 0x00f8b9 thru 0x015a06 (0x614e bytes) 000923 1790658682.893532 write 0x02604b thru 0x027e33 (0x1de9 bytes) 000925 1790658682.910135 mapwrite 0x035905 thru 0x03b666 (0x5d62 bytes) 000928 1790658683.024061 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b667): Mirror 5: 0xe250beda (stale) Mirror 6: 0xe250beda (stale) Mirror 9: 0xf652d38e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b667): Mirror 5: 0xe250beda (stale) Mirror 6: 0xe250beda (stale) Mirror 9: 0xf652d38e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b667: skipped 000929 1790658683.477115 trunc from 0x03b667 to 0x02f6fb (0xbf6b bytes) 000930 1790658683.501455 write 0x030a29 thru 0x03b27a (0xa852 bytes) 000932 1790658683.504890 write 0x01432c thru 0x017c43 (0x3918 bytes) 000933 1790658683.517141 write 0x0346b9 thru 0x03ffff (0xb947 bytes) truncating to largest ever: 0x3f722 000934 1790658683.523651 trunc from 0x040000 to 0x03f721 (0x8de bytes) 000939 1790658683.555717 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f722): Mirror 6: 0xdbca3e5e (stale) Mirror 9: 0x7810e459 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f722): Mirror 6: 0xdbca3e5e (stale) Mirror 9: 0x7810e459 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f722: skipped 000940 1790658683.899448 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f722): Mirror 9: 0x7810e459 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f722): Mirror 9: 0x7810e459 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f722: skipped 000942 1790658684.175264 mapread 0x013a9c thru 0x01d2cf (0x9834 bytes) 000944 1790658684.198708 MIRROR_RESYNC 2 mirrors 000945 1790658685.217968 write 0x00074c thru 0x004771 (0x4026 bytes) 000947 1790658685.321574 mapread 0x01c9f7 thru 0x027f45 (0xb54f bytes) 000951 1790658685.358539 write 0x010295 thru 0x0102fc (0x68 bytes) 000952 1790658685.376091 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) 000955 1790658685.654734 mapwrite 0x01f9cb thru 0x0260a7 (0x66dd bytes) 000956 1790658685.799461 read 0x01ae39 thru 0x021b5c (0x6d24 bytes) 000957 1790658685.801634 punch from 0x02363f to 0x02c000 (0x89c2 bytes) 000959 1790658685.821951 write 0x02ef49 thru 0x03761b (0x86d3 bytes) 000960 1790658685.854522 trunc from 0x03f722 to 0x01b457 (0x242ca bytes) 000961 1790658685.889690 mapwrite 0x01a272 thru 0x01e7b7 (0x4546 bytes) 000966 1790658686.023320 write 0x02cefb thru 0x0382f8 (0xb3fe bytes) 000969 1790658686.028736 write 0x003422 thru 0x0115da (0xe1b9 bytes) 000970 1790658686.041325 mapwrite 0x00126b thru 0x006f5b (0x5cf1 bytes) 000972 1790658686.182681 read 0x01b2c2 thru 0x026cdd (0xba1c bytes) 000976 1790658686.197703 punch from 0x01861f to 0x0273de (0xedc0 bytes) 000978 1790658686.204073 mapwrite 0x02cb3f thru 0x03a312 (0xd7d4 bytes) 000981 1790658686.282443 mapwrite 0x02fd9b thru 0x033f07 (0x416d bytes) 000987 1790658686.347612 trunc from 0x03a313 to 0x015d88 (0x2458a bytes) 000989 1790658686.372378 trunc from 0x015d89 to 0x0072bd (0xeacb bytes) 000990 1790658686.390954 mapwrite 0x0056fd thru 0x006be1 (0x14e5 bytes) 000991 1790658686.475461 MIRROR_EXTEND to 2 mirrors 000993 1790658687.066545 punch from 0x003443 to 0x0072bd (0x3e7b bytes) 000994 1790658687.171122 mapread 0x002d85 thru 0x0072bd (0x4539 bytes) 000995 1790658687.202230 read 0x002063 thru 0x0072bd (0x525b bytes) 000996 1790658687.215422 MIRROR_RESYNC 2 mirrors 000997 1790658688.368505 write 0x026a74 thru 0x029455 (0x29e2 bytes) 000998 1790658688.457895 trunc from 0x029456 to 0x006368 (0x230ed bytes) 000999 1790658688.488402 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a8:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (108s) == sanity-flr test 71: check mirror extend parent fid ==== 01:11:34 (1790658694) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 01:11:39 (1790658699) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.344034 s, 6.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149313 s, 7.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x288:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x288:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 01:11:49 (1790658709) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre2 Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7561024 .. Extending file size to 1438784 .. failed Extending file size to 6976576 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8808832 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8560800 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9152032 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9078976 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57944 57945 57947 57948 57951 done 192.168.202.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre2 (opts:) 192.168.202.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:13:17 (1790658797) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre2 Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3731008 .. Extending file size to 6584224 .. done Extending file size to 8976928 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8545952 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8809152 .. Extending file size to 9203456 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59474 59475 59477 59479 59480 done 192.168.202.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre2 (opts:) 192.168.202.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:14:36 (1790658876) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0589402 s, 35.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.478215 s, 4.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x49e1911f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:14:58 (1790658898) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0568692 s, 36.9 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 01:15:05 (1790658905) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:15:14 (1790658914) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242846 s, 8.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:15:23 (1790658923) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.258173 s, 8.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:15:34 (1790658934) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.4468 s, 4.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:16:10 (1790658970) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg247-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104174 s, 10.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a5:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:17:13 (1790659033) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg247-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22549 s, 9.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:18:15 (1790659095) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg247-server mds-ost sync done. Creating new pool test_204f oleg247-server: Pool lustre.test_204f created Adding targets to pool oleg247-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg247-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.465447 s, 4.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg247-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg247-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg247-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:19:35 (1790659175) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:19:43 (1790659183) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:19:52 (1790659192) oleg247-server: Pool lustre.test_206 created Adding targets to pool oleg247-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg247-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg247-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg247-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg247-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:20:21 (1790659221) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138743 s, 7.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829573 s, 12.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8c1ab6b8 Mirror 3: 0x8c1ab6b8 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:20:33 (1790659233) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:20:37 (1790659237) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:20:41 (1790659241) Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.104232 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1890000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2340000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2340000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2340000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1a20000 Read this (4 bytes): BBBB 192.168.202.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg247-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:21:23 (1790659283) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0260363 s, 315 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:21:32 (1790659292) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890985 s, 11.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x214 lmm_fid: [0x200000404:0x214:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x214 lmm_fid: [0x200000404:0x214:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834452 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 01:21:34.000000000 -0400 Modify: 2026-09-29 01:21:34.000000000 -0400 Change: 2026-09-29 01:21:36.000000000 -0400 Birth: 2026-09-29 01:21:34.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:21:41 (1790659301) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:21:45 (1790659305) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.859302 s, 47.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0446565 s, 91.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 29 01:21 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x216:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 29 01:21 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:21:54 (1790659314) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:22:01 (1790659321) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 2003 sec =========== 01:22:10 (1790659330) === sanity-flr: start cleanup 01:22:12 (1790659332) === === sanity-flr: finish cleanup 01:22:18 (1790659338) ===